Commit Graph

2667 Commits

Author SHA1 Message Date
Teon Banek
4c6eb0746e query/procedure: Implement mgp_value C API
Reviewers: mferencevic, ipaljak, llugovic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2498
2019-10-28 15:24:14 +01:00
Lovro Lugovic
3750fbc093 Merge Interpreter and TransactionEngine
Summary:
Depends on D2471

- Add pointer to storage to `InterpreterContext`
- Rename `operator()` to `Prepare`
- Use `Interpret` instead of `operator()` (`Interpret` will be removed soon)
- Remove the `in_explicit_transaction` parameter
- Remove the memory resource parameter from `Interpret`
- Remove the storage accessor parameter from `Interpret`
- Fix up tests (remove the `Interpreter` from `database_transaction_timeout`)

Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2482
2019-10-25 16:11:10 +02:00
Matej Ferencevic
98855889d5 Implement WalFile for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2510
2019-10-25 16:08:17 +02:00
Lovro Lugovic
d2fac02b74 Pull out InterpreterContext
Summary: Make `InterpreterContext` a top level instead of a nested struct

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2512
2019-10-25 15:58:47 +02:00
Lovro Lugovic
4c25719c45 Extract state shared between interpreters into InterpreterContext
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D2471
2019-10-25 14:12:14 +02:00
Teon Banek
35180f11d3 Update .clang-tidy to not warn trailing return type
Summary:
This is primarily an update which disables the
`modernize-use-trailing-return-type` suggestion introduced in
clang-tidy-9. Old clang-tidy versions should ignore removed checks which
don't exist in the first place, so this should change should be
compatible with any version.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2511
2019-10-25 11:16:50 +02:00
Matej Ferencevic
bafc0a9d16 Reduce memory blowup on each Pull
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2509
2019-10-23 10:01:17 +02:00
Matej Ferencevic
3bed1cc3ea Correctly pass config into AllVerticesIterable
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2500
2019-10-21 14:15:32 +02:00
Teon Banek
e566126832 query/procedure: Implement memory management C API
Reviewers: mferencevic, ipaljak, llugovic

Reviewed By: mferencevic, llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2489
2019-10-18 14:35:04 +02:00
Teon Banek
469102aa5e Pass mgp_memory to custom Module.main_fn
Summary: Depends on D2495

Reviewers: mferencevic, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2496
2019-10-18 13:38:01 +02:00
Teon Banek
2e1063bbdb Introduce mgp_memory to mg_procedure API
Summary:
This makes the concrete memory resource an explicit argument in custom
procedures. Unfortunately, the API is a bit uglier but at least we don't
have to worry about threading, global state or similar. This is the
primary reason for this change as we would probably like to allow
procedures to spawn threads to compute stuff in a parallel fashion.

Reviewers: mferencevic, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2495
2019-10-18 13:37:54 +02:00
Matej Ferencevic
842f0f3bf6 Add missing check to snapshot recovery
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2490
2019-10-16 13:18:52 +02:00
Matej Ferencevic
e5a4a84de2 Fix MonotonicBufferResource buffer growth
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2483
2019-10-15 15:04:35 +02:00
Teon Banek
7afa4fd7b3 Add C API for working with graph constructs in procedures
Reviewers: mferencevic, ipaljak, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2469
2019-10-15 15:06:35 +02:00
Teon Banek
103f51d3ec Add C API for creating a procedure result
Reviewers: mferencevic, ipaljak, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2464
2019-10-04 10:27:39 +02:00
Lovro Lugovic
5c5c926515 An aggregate element's input expression can be NULL
Summary: For example, the aggregate element produced for `COUNT(*)` has its `value` set to `NULL`.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2463
2019-10-03 10:07:24 +02:00
Teon Banek
d385ebfa15 Add C API for raw memory allocations in procedures
Reviewers: ipaljak, dsantl, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2461
2019-10-02 10:28:49 +02:00
Matej Ferencevic
3adedc0679 Implement snapshots for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2400
2019-10-01 13:42:37 +02:00
Matej Ferencevic
3756534490 Don't crash the database when using utils::InputFile
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2460
2019-10-01 10:15:42 +02:00
Matej Ferencevic
8c7c593680 Clean-up of storage v2 delta
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2451
2019-09-27 10:47:28 +02:00
Teon Banek
f63a5e9de5 Rename PluginRegistry to ModuleRegistry
Summary:
The registry is now in the `query::procedure` namespace, this makes the
naming more consistent. I.e. we are dealing with custom procedure which
are contained in modules. This naming convention is similar to Python
source code where each file represents a module and each module provides
multiple functions (or procedures in our case). At the moment we only
support exactly 1 procedure per module, but the openCypher syntax allows
for more.

Reviewers: mferencevic, ipaljak, dsantl

Differential Revision: https://phabricator.memgraph.io/D2454
2019-09-26 13:29:13 +02:00
Teon Banek
6f25e43b52 Stub out C API for working with mg_value in custom procedures
Summary:
The `mg_value` is the main data type used in upcoming custom procedures. Each
custom procedure will receive arguments as an array of `mg_value` instances.
The results of custom procedures will also be built as `mg_value` instances.

We should strive to have as simple and straightforward C API for custom
procedures, because we want to limit the possibility of errors and bugs on the
user facing side. Custom procedures will run in Memgraph and as such can cause
crashes and potentially a lot of nasty stuff. The burden is on the writer of
custom procedures not to mess things up, so we should make messing things up
from the C API hard if not impossible to do.

Reviewers: mferencevic, ipaljak, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2443
2019-09-25 15:46:25 +02:00
Matej Ferencevic
9ad49698e9 Add support for disabling properties on edges
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2447
2019-09-25 10:17:19 +02:00
Matej Ferencevic
8b6ae08682 Create/drop label indices explicitly in storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2432
2019-09-24 13:12:50 +02:00
Matej Ferencevic
3079522f47 Make storage v2 config hierarchical
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2434
2019-09-23 16:03:53 +02:00
Matej Ferencevic
9d39ecab7b Implement encoder/decoder for durability v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2393
2019-09-18 16:32:32 +02:00
Teon Banek
3a7bd6236e Set -Werror=return-stack-address in CMakeLists
Summary:
This is a clang only flag, it promotes warnings to errors when returning
an address to something on the stack

Reviewers: mferencevic, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2394
2019-09-18 10:04:34 +02:00
Teon Banek
04c037e0fd Add PluginRegistry for openCypher custom procedures
Summary:
This diff implements a mechanism for registering plugins which provide
custom procedures for openCypher. Although the `Plugin` struct already
stores some function pointers, these are not set in stone w.r.t. to
requirements and signatures.

For example, in the future, we may want to allow a single plugin to
register multiple custom procedures instead of just one.

Reviewers: ipaljak, dsantl, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2386
2019-09-17 12:46:45 +02:00
Matej Ferencevic
fe2196b403 Add more file utils
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2384
2019-09-13 15:24:52 +02:00
Matej Ferencevic
9652f8e265 Add clear method to SkipList
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2385
2019-09-13 15:24:14 +02:00
Matej Ferencevic
55d8b98aeb Prepare storage v2 config for more options
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2382
2019-09-13 12:42:23 +02:00
Matej Ferencevic
51fc4df87a Add utils::InputFile wrapper
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2377
2019-09-12 16:30:11 +02:00
Matej Ferencevic
b74141aafe Fix accessor locking in storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2376
2019-09-12 13:11:12 +02:00
Teon Banek
7bd45f8714 Make query execution work with storage_v2
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2203
2019-09-12 10:22:00 +02:00
Matej Ferencevic
68f19df305 Migrate LDAP integration to auth module
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2368
2019-09-11 17:15:08 +02:00
Matej Ferencevic
73379bc57e Add functions for listing indices and constraints
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2371
2019-09-11 15:45:14 +02:00
Teon Banek
6d9933373e Add missing include to scoped_profile.hpp
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic, ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2370
2019-09-11 12:46:37 +02:00
Ivan Paljak
deee3b8ab7 Remove Raft's dependency on transaction id
Reviewers: buda, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2338
2019-09-10 18:02:05 +02:00
Teon Banek
85b01f8497 Add query/db_accessor.hpp which wraps storage implementation
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2361
2019-09-10 15:43:07 +02:00
Matej Ferencevic
13daf1efd6 Implement auth module
Summary:
This diff adds support for an auth module. The module is used to provide
authentication and authorization (only user to role mappings). The module can
be written in any language and uses a simple protocol to communicate with
Memgraph.

Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2359
2019-09-10 14:46:06 +02:00
Teon Banek
1000a5e544 Use BasicResult instead of optional for v2 Commit
Summary: Depends on D2365

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2366
2019-09-10 13:32:54 +02:00
Teon Banek
51d8d16114 Add utils::BasicResult specialization for void
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2365
2019-09-10 13:32:54 +02:00
Teon Banek
f02617f2bd Replace query::GraphView with storage::View
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2356
2019-09-06 14:58:27 +02:00
Teon Banek
ec22026fed Add glue::ToBolt(Vertex|Edge|Path|Value) for V2 storage
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2355
2019-09-06 10:51:10 +02:00
Teon Banek
2e51ef9f66 Use storage/v2/view in glue/communication
Summary:
Switching to Storage V2 API will require passing storage::View when
serializing VertexAccessor and EdgeAccessor, so this is just the first
step in adapting the code.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2352
2019-09-05 10:04:44 +02:00
Teon Banek
9a94205a07 Document exceptions and other details in storage v2 API
Summary:
Most instances of `@throw std::bad_alloc` are left unexplained as these
functions perform general heap allocations are it's obvious from the
function name that it will do so. Basically anything with `Create`, `Make` or
`Build` implies allocations. Additionally, which parts exactly perform
allocations are an implementation detail. Functions which do unexpected
heap allocations have the reason stated in the documentation, these
functions typically have exactly one spot which could raise such an
exception.

Some functions are marked as `noexcept`, these are usually "special
functions" such as constructors and operators. This could potentially
improve performance because STL may use API overloads that work faster
with `noexcept` stuff. Remaining non-throwing functions aren't marked as
`noexcept` as that wasn't our practice nor is common in our codebase. On
the other hand, if we continue enforcing the documentation of thrown
exceptions, perhaps we should start using `noexcept`.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2350
2019-09-05 10:04:40 +02:00
Teon Banek
02722a1ed5 Add default constructor and std::hash to v2/id_types
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2349
2019-09-03 16:13:52 +02:00
Teon Banek
7213bec886 Make Gid a full on class wrapper around uint64_t
Summary:
This makes Gid the same as the one in storage/v2. Before they can be
merge into one implementation, we probably want to have a similar
transition for remaining ID types.

Depends on D2346

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2347
2019-09-03 15:14:21 +02:00
Teon Banek
cdfaf0d4a6 Move gid::Gid to storage/common/types
Summary:
It never made sense that a global ID is its own namespace in the storage
directory tree.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2346
2019-09-03 14:57:44 +02:00
Ivan Paljak
58be850d5c Fix HA log compaction test
Summary: The test now checks if the cluster is alive at the end of the test.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2343
2019-09-03 08:45:53 +02:00