Commit Graph

2585 Commits

Author SHA1 Message Date
Matej Ferencevic
9eb1f1d5cd Implement ClearProperties for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2537
2019-11-07 13:07:26 +01:00
Teon Banek
df43ccea95 Add CallProcedure LogicalOperator
Summary: Depends on D2534

Reviewers: mferencevic, ipaljak, llugovic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2535
2019-11-06 15:46:00 +01:00
Teon Banek
5567e6ec96 Add utils::LimitedMemoryResource
Reviewers: llugovic, mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2534
2019-11-06 15:45:56 +01:00
Matej Ferencevic
a835939a6e Implement buffering for utils file wrappers
Summary:
This change increases the throughput of the storage v2 durability 20x. With
this change, the storage v2 durability is 3x faster than the storage v1
durability in both recovery and snapshotting (before the change v2 durability
is slower than v1 durability).

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2529
2019-11-05 12:30:09 +01:00
Matej Ferencevic
47d6196b32 Improve binary coverage script
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2530
2019-11-04 17:02:03 +01:00
Matej Ferencevic
73134ab6b7 Integrate all storage v2 flags into main binary
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2528
2019-11-04 17:01:56 +01:00
Teon Banek
32069c77a0 Add CallProcedure clause to Cypher
Summary:
This adds support for basic invocation to CALL clause of openCypher. The
accepted CIP has a lot more features that are avaiable with CALL clause.

https://github.com/opencypher/openCypher/blob/master/cip/1.accepted/CIP2015-06-24-call-procedures.adoc#appendix-procedure-naming-conventions

Reviewers: mferencevic, ipaljak, llugovic

Reviewed By: mferencevic, llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2523
2019-11-04 16:21:29 +01:00
Matej Ferencevic
4248b140d4 Backup existing durability files
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2526
2019-11-04 10:38:13 +01:00
Matej Ferencevic
42516afce8 Remove Kafka integration implementation and tests
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2525
2019-11-04 09:56:07 +01:00
Matej Ferencevic
875a4a8629 Implement WAL loading for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2521
2019-10-31 17:12:37 +01:00
Teon Banek
b5482a8169 Add .clang-tidy for tests and update the config
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2524
2019-10-31 16:07:09 +01:00
Teon Banek
043ae87a83 query/procedure: Implement graph related API
Reviewers: mferencevic, llugovic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2522
2019-10-30 15:11:53 +01:00
Lovro Lugovic
bd998dc618 Refactor query parsing in the Interpreter
Summary: Depends on D2482

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2499
2019-10-30 10:32:22 +01:00
Lovro Lugovic
b0cc564f8d Make ResultStreamFaker a normal class
Summary:
Store accumulated results as `communication::bolt::Value`s instead of
`TypedValue`s.

Add additional overloads for `Result` and `Summary` which accept `TypedValue`s
but internally perform conversions.

Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2514
2019-10-30 10:22:21 +01:00
Matej Ferencevic
13ba9cc23e Implement WAL writing for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2515
2019-10-29 16:18:12 +01:00
Teon Banek
4879686ef4 query/procedure: Implement mgp_list & mgp_map API
Reviewers: mferencevic, llugovic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2518
2019-10-29 15:52:05 +01:00
Teon Banek
a72743757c query/procedure: Implement mgp_result API
Reviewers: mferencevic, ipaljak, llugovic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2516
2019-10-29 11:02:30 +01:00
Matej Ferencevic
2e0586e182 Implement single function for WAL loading
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2513
2019-10-28 15:42:49 +01:00
Teon Banek
4c6eb0746e query/procedure: Implement mgp_value C API
Reviewers: mferencevic, ipaljak, llugovic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2498
2019-10-28 15:24:14 +01:00
Lovro Lugovic
3750fbc093 Merge Interpreter and TransactionEngine
Summary:
Depends on D2471

- Add pointer to storage to `InterpreterContext`
- Rename `operator()` to `Prepare`
- Use `Interpret` instead of `operator()` (`Interpret` will be removed soon)
- Remove the `in_explicit_transaction` parameter
- Remove the memory resource parameter from `Interpret`
- Remove the storage accessor parameter from `Interpret`
- Fix up tests (remove the `Interpreter` from `database_transaction_timeout`)

Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2482
2019-10-25 16:11:10 +02:00
Matej Ferencevic
98855889d5 Implement WalFile for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2510
2019-10-25 16:08:17 +02:00
Lovro Lugovic
d2fac02b74 Pull out InterpreterContext
Summary: Make `InterpreterContext` a top level instead of a nested struct

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2512
2019-10-25 15:58:47 +02:00
Lovro Lugovic
4c25719c45 Extract state shared between interpreters into InterpreterContext
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D2471
2019-10-25 14:12:14 +02:00
Teon Banek
35180f11d3 Update .clang-tidy to not warn trailing return type
Summary:
This is primarily an update which disables the
`modernize-use-trailing-return-type` suggestion introduced in
clang-tidy-9. Old clang-tidy versions should ignore removed checks which
don't exist in the first place, so this should change should be
compatible with any version.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2511
2019-10-25 11:16:50 +02:00
Matej Ferencevic
bafc0a9d16 Reduce memory blowup on each Pull
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2509
2019-10-23 10:01:17 +02:00
Matej Ferencevic
3bed1cc3ea Correctly pass config into AllVerticesIterable
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2500
2019-10-21 14:15:32 +02:00
Teon Banek
e566126832 query/procedure: Implement memory management C API
Reviewers: mferencevic, ipaljak, llugovic

Reviewed By: mferencevic, llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2489
2019-10-18 14:35:04 +02:00
Teon Banek
469102aa5e Pass mgp_memory to custom Module.main_fn
Summary: Depends on D2495

Reviewers: mferencevic, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2496
2019-10-18 13:38:01 +02:00
Teon Banek
2e1063bbdb Introduce mgp_memory to mg_procedure API
Summary:
This makes the concrete memory resource an explicit argument in custom
procedures. Unfortunately, the API is a bit uglier but at least we don't
have to worry about threading, global state or similar. This is the
primary reason for this change as we would probably like to allow
procedures to spawn threads to compute stuff in a parallel fashion.

Reviewers: mferencevic, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2495
2019-10-18 13:37:54 +02:00
Matej Ferencevic
842f0f3bf6 Add missing check to snapshot recovery
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2490
2019-10-16 13:18:52 +02:00
Matej Ferencevic
e5a4a84de2 Fix MonotonicBufferResource buffer growth
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2483
2019-10-15 15:04:35 +02:00
Teon Banek
7afa4fd7b3 Add C API for working with graph constructs in procedures
Reviewers: mferencevic, ipaljak, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2469
2019-10-15 15:06:35 +02:00
Teon Banek
103f51d3ec Add C API for creating a procedure result
Reviewers: mferencevic, ipaljak, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2464
2019-10-04 10:27:39 +02:00
Lovro Lugovic
5c5c926515 An aggregate element's input expression can be NULL
Summary: For example, the aggregate element produced for `COUNT(*)` has its `value` set to `NULL`.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2463
2019-10-03 10:07:24 +02:00
Teon Banek
d385ebfa15 Add C API for raw memory allocations in procedures
Reviewers: ipaljak, dsantl, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2461
2019-10-02 10:28:49 +02:00
Matej Ferencevic
3adedc0679 Implement snapshots for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2400
2019-10-01 13:42:37 +02:00
Matej Ferencevic
3756534490 Don't crash the database when using utils::InputFile
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2460
2019-10-01 10:15:42 +02:00
Matej Ferencevic
8c7c593680 Clean-up of storage v2 delta
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2451
2019-09-27 10:47:28 +02:00
Teon Banek
f63a5e9de5 Rename PluginRegistry to ModuleRegistry
Summary:
The registry is now in the `query::procedure` namespace, this makes the
naming more consistent. I.e. we are dealing with custom procedure which
are contained in modules. This naming convention is similar to Python
source code where each file represents a module and each module provides
multiple functions (or procedures in our case). At the moment we only
support exactly 1 procedure per module, but the openCypher syntax allows
for more.

Reviewers: mferencevic, ipaljak, dsantl

Differential Revision: https://phabricator.memgraph.io/D2454
2019-09-26 13:29:13 +02:00
Teon Banek
6f25e43b52 Stub out C API for working with mg_value in custom procedures
Summary:
The `mg_value` is the main data type used in upcoming custom procedures. Each
custom procedure will receive arguments as an array of `mg_value` instances.
The results of custom procedures will also be built as `mg_value` instances.

We should strive to have as simple and straightforward C API for custom
procedures, because we want to limit the possibility of errors and bugs on the
user facing side. Custom procedures will run in Memgraph and as such can cause
crashes and potentially a lot of nasty stuff. The burden is on the writer of
custom procedures not to mess things up, so we should make messing things up
from the C API hard if not impossible to do.

Reviewers: mferencevic, ipaljak, dsantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2443
2019-09-25 15:46:25 +02:00
Matej Ferencevic
9ad49698e9 Add support for disabling properties on edges
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2447
2019-09-25 10:17:19 +02:00
Matej Ferencevic
8b6ae08682 Create/drop label indices explicitly in storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2432
2019-09-24 13:12:50 +02:00
Matej Ferencevic
3079522f47 Make storage v2 config hierarchical
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2434
2019-09-23 16:03:53 +02:00
Matej Ferencevic
9d39ecab7b Implement encoder/decoder for durability v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2393
2019-09-18 16:32:32 +02:00
Teon Banek
3a7bd6236e Set -Werror=return-stack-address in CMakeLists
Summary:
This is a clang only flag, it promotes warnings to errors when returning
an address to something on the stack

Reviewers: mferencevic, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2394
2019-09-18 10:04:34 +02:00
Teon Banek
04c037e0fd Add PluginRegistry for openCypher custom procedures
Summary:
This diff implements a mechanism for registering plugins which provide
custom procedures for openCypher. Although the `Plugin` struct already
stores some function pointers, these are not set in stone w.r.t. to
requirements and signatures.

For example, in the future, we may want to allow a single plugin to
register multiple custom procedures instead of just one.

Reviewers: ipaljak, dsantl, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2386
2019-09-17 12:46:45 +02:00
Matej Ferencevic
fe2196b403 Add more file utils
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2384
2019-09-13 15:24:52 +02:00
Matej Ferencevic
9652f8e265 Add clear method to SkipList
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2385
2019-09-13 15:24:14 +02:00
Matej Ferencevic
55d8b98aeb Prepare storage v2 config for more options
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2382
2019-09-13 12:42:23 +02:00
Matej Ferencevic
51fc4df87a Add utils::InputFile wrapper
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2377
2019-09-12 16:30:11 +02:00