Summary:
This diff fixes the variable expand operator to work correctly then the start
and destination nodes use the same symbol or when the destination symbol is an
existing symbol.
Previously, the variable expand operator produced paths that were both
completely wrong (they shouldn't have been produced) and nonexistent (they
didn't even exist in the storage). Invalid data was produced because of a
wrong equality check that was introduced in D1703.
This issue was reported externally and the supplied test case was:
```
CREATE (p1:Person {id: 1})-[:KNOWS]->(:Person {id: 2})-[:KNOWS]->(:Person {id: 3})-[:KNOWS]->(:Person {id: 4})-[:KNOWS]->(p1);
MATCH path = (pers:Person {id: 3})-[:KNOWS*2]->(pers) RETURN path;
```
Also, tests have been added so the behavior remains correct.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2793
Summary:
Semantics of 'all' and 'single' were updated to be
consistent with that of 'any' and 'none'
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2789
Summary: Change any function's handling of Null elements in a list
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2787
Summary:
`DUMP DATABASE` used a separate transaction to read database data. That
wouldn't be an issue if the query was correctly disallowed in multicommand
transactions. Because it was allowed the output wasn't transactionally correct.
Instead of disabling `DUMP DATABASE` in multicommand transactions this change
fixes it so that it works properly in multicommand transactions.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2781
Summary:
Before this change properties were joined by ", " and returned as a single string,
which was ambiguous for properties that contain ", ". This diff solves this
problem by returning properties as a list type.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2778
Summary:
This diff fixes the issue for label name (and edge type/property)
with spaces and special characters to avoid possible OpenCypher injections.
Consider an example where label name is 'hello :world'. `DUMP DATABASE`
used to return query which creates a node (u:hello :world) - i.e. node
that contains two labels 'hello' and 'world'. This fix escapes names to
create the following node with exactly one label as expected:
```
(u:`hello :world`)
```
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2774
Summary:
This diff improves the performance of `PropertyStore` with two main
techniques:
First:
`PropertyValue` has a very expensive constructor and destructor. The
`PropertyValue` was previously passed as a return value from many functions
wrapped in a `std::optional`. That caused the `PropertyValue`
constructor/destructor to be called for each intermediary value that was passed
between functions. This diff changes the functions to return a `bool` value
that imitates the `std::optional` "emptyness" flag and the `PropertyValue` is
modified using a pointer to it so that its constructor/destructor is called
only once.
Second:
The `PropertyStore` buffer was previously iterated through at least twice.
First to determine the exact position of the encoded property and then to
actually decode the property. This diff combines the two passes into a single
pass so that the property is immediately loaded if it is found.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2766
Summary:
Use Unwind + ScanAllByLabelPropertyValue logical operator to
accelerate the execution of queries like the following one:
`MATCH (n:Label) WHERE n.property IN [] ...`
Reviewers: llugovic, mferencevic
Reviewed By: llugovic, mferencevic
Subscribers: llugovic, pullbot
Differential Revision: https://phabricator.memgraph.io/D2758
Summary: All integers are signed in openCypher.
Reviewers: buda, llugovic
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2746
Summary:
- Add the `mgp_must_abort(const mgp_graph *graph)` C API.
- Add the `ProcCtx.must_abort()` Python API.
The usage is very simple -- the function returns a boolean indicating whether
the procedure should abort.
Reviewers: mferencevic, dsantl
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2742
Summary:
The Python `ExceptionInfo` data has the potential to extend lifetime of Python
objects that were on the Python frame while the execution was thrown. This
lifetime extension is very dangerous when executing Python procedures because
we use a custom internal memory allocator that is destroyed immediately when
the procedure is done with its execution.
This diff only keeps the formatted Python traceback (as a string) to avoid any
lifetime extension for the Python objects so that the memory allocator can
safely be destroyed after the procedure has finished its execution.
Reviewers: llugovic
Reviewed By: llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2739
Summary:
`PyObject_Init` shouldn't be used in conjunction with `PyObject_New`.
The official Python documentation doesn't say this explicitly and many examples
on the web suggest that they must be used together. When they are used together
everything works when Memgraph is built against the release Python binary, but
doesn't work when built against the debug Python binary.
An implicit example that `PyObject_Init` shouldn't be used with `PyObject_New`
can be found in the Python source, in the file `Include/objimpl.h`:
```
This example code implements an object constructor with a custom
allocator, where PyObject_New is inlined, and shows the important
distinction between two steps (at least):
1) the actual allocation of the object storage;
2) the initialization of the Python specific fields
in this storage with PyObject_{Init, InitVar}.
PyObject *
YourObject_New(...)
{
PyObject *op;
op = (PyObject *) Your_Allocator(_PyObject_SIZE(YourTypeStruct));
if (op == NULL)
return PyErr_NoMemory();
PyObject_Init(op, &YourTypeStruct);
op->ob_field = value;
...
return op;
}
Note that in C++, the use of the new operator usually implies that
the 1st step is performed automatically for you, so in a C++ class
constructor you would start directly with PyObject_Init/InitVar
```
It explains that `PyObject_New` is actually equal to `malloc` +
`PyObject_Init`.
Reviewers: llugovic
Reviewed By: llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2737
Summary:
This fixes an issue in Py(Vertex|Edge)GetProperty and prevents any
further issues of that type at the cost of additional typing effort.
Reviewers: ipaljak, llugovic
Reviewed By: ipaljak, llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2735
Summary:
In order for this race condition to cause damage, an index/constraint must be
created/dropped at the exact moment that the GC is cleaning
indices/constraints.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2733
Summary:
Currently, when starting Memgraph with the production package (DEB/RPM),
Memgraph always outputs an error for not being able to replace an existing
query module (`example.so` with `example.c`). This diff introduces a precheck
so that the error message is correct - so that Memgraph doesn't try to replace
an `.so` file with a `.c` file before verifying that the `.c` file is a valid
query module (which it obviously isn't). Also, I have moved the source of the
example into a subdirectory so that it isn't even considered while loading
modules.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2724
Summary:
The storage now uses a file in the data directory (`.lock`) to determine
whether there is another instance of the storage running with the same data
directory. That helps notify the user/administrator that the system is running
in an unsupported configuration.
Reviewers: teon.banek, ipaljak
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2719
Summary:
The importer now supports all of the flags that the modern Neo4j CSV importer
supports.
Reviewers: teon.banek, ipaljak
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2709
Summary:
This diff contains simple tests for unique constraints which tries to
change property values or labels in multiple threads at the same time.
During testing, a bug has been encountered in unique constraints, i.e.
one guard lock on vertices was missing.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: mferencevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D2711
Summary:
You should now be able to invoke query procedures written in Python. To
test the example you can run memgraph with PYTHONPATH set to `include`.
For example, assuming you are in the root of the repo, run this command.
PYTHONPATH=$PWD/include ./build/memgraph --query-modules-directory=./query_modules/
Alternatively, you can set a symlink inside the ./query_modules to point
to `include/mgp.py`, so there's no need to set PYTHONPATH. For example,
assuming you are in the root of the repo, run the following.
cd ./query_modules
ln -s ../include/mgp.py
cd ..
./build/memgraph --query-modules-directory=./query_modules/
Depends on D207
Reviewers: mferencevic, ipaljak, dsantl
Reviewed By: ipaljak
Subscribers: buda, tlastre, pullbot
Differential Revision: https://phabricator.memgraph.io/D2708
Summary:
This diff contains a necessary functionality to save and restore unique
constraint operations. The previous snapshot/WAL version is backward
compatible. Integration tests for migration from older snapshot and WAL
versions are also included.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2680
Summary:
The new CSV parser in `mg_import_csv` behaves the same when importing a CSV
file as the standard Python CSV importer. Tests are added for all CSV field
edge-cases.
Reviewers: teon.banek, ipaljak
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2702
Summary:
When invoking a Python registered procedures we want to convert
`mgp_value` types to user facing 'mgp' types.
Depends on D2706
Reviewers: mferencevic, ipaljak
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2707
Summary:
This issue was already fixed in D2119 for the iterator, but I missed to fix the
const iterator in that diff...
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2703
Summary:
This diff restores (and fixes) the old mg_import_csv implementation. The
importer now supports the new storage engine.
Reviewers: teon.banek, ipaljak
Reviewed By: teon.banek, ipaljak
Subscribers: buda, pullbot
Differential Revision: https://phabricator.memgraph.io/D2690
Summary:
Now that unique constraint feature is added to Memgraph database,
we should update `DUMP DATABASE` with list of existing unique constraints.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2698
Summary:
With this diff you should now be able to register `example.py` and read
procedures found there. The procedures will be listed through `CALL
mg.procedures() YIELD *` query, but invoking them will raise
`NotYetImplemented`.
If you wish to test this, you will need to run the Memgraph executable
with PYTHONPATH set to the `include` directory where `mgp.py` is found.
Additionally, you need to pass `--query-modules-directory` flag to
Memgraph, such that it points to where it will find the `example.py`.
For example, when running from the root directory of Memgraph repo, the
shell invocation below should do the trick (assuming `./build/memgraph`
is where is the executable). Make sure that `./query_modules/` does not
have `example.so` built, as that may interfere with loading
`example.py`.
PYTHONPATH=$PWD/include ./build/memgraph --query-modules-directory=./query_modules/
Reviewers: mferencevic, ipaljak, llugovic
Reviewed By: mferencevic, ipaljak, llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2678
Summary:
Before this change, unique constraints supported only pairs of label
and a single property. With this change, unique constraints can be
created for label and set of properties.
Better tests for unique constraints in general are also included in
this diff.
Reviewers: mferencevic, teon.banek
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2653
Summary:
With this diff, each build of Memgraph has a version that uniquely identifies
it. The given version uniquely identifies both official release builds and
development builds. Enterprise/community builds are also differentiated in the
version. Also, support for custom suffixes is added to support custom builds
for customers.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2662
Summary:
`query::procedure::Module` is now an interface class through which
users interact with loaded modules. At the moment, the interface is implemented
by three concrete classes: `query::procedure::BuiltinModule`,
`query::procedure::SharedLibraryModule` and `query::procedure::PythonModule`.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2664
Summary:
This diff contains a basic implementation of unique constraints consistent with
the MVCC storage.
Stale records in the unique constraints are collected by the garbage collector.
Tests for checking correctness of unique constraints and violations are included.
Note: currently we only support a pair of label and a single property. Support for
multiple properties will be added later.
Reviewers: mferencevic, teon.banek
Reviewed By: mferencevic, teon.banek
Subscribers: buda, ipaljak, pullbot
Differential Revision: https://phabricator.memgraph.io/D2608
Summary:
This only imports the `.py` files into the global interpreter. These
modules are not exposed to query execution. A later diff will add that
support.
Reviewers: mferencevic, ipaljak
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2660
Summary:
The antlr openCypher parser generation is moved to query. Also, header files
have been added to the list of generated files so that if any header file is
deleted CMake will know that it has to regenerate it.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2652
Summary:
The property store stores a map of `PropertyId` to `PropertyValue` mappings. It
compresses all of the values in order to use as little memory as possible.
Reviewers: teon.banek, ipaljak
Reviewed By: teon.banek
Subscribers: buda, pullbot
Differential Revision: https://phabricator.memgraph.io/D2604
Summary:
Most of the examples require properties to be enabled on edges so this change
specifies that flag explicitly. Also, errors are handled better by replacing
`bolt_client` with `mg_client`. Missing indices are created to improve import
speed.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2594
Summary:
The configuration file that was used for the Debian/CentOS package was manually
written. This diff adds a configuration file generator that extracts all of the
necessary information about the flags directly from the built binary and uses
that information to generate the configuration file for the packages.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2589
Summary:
The function will also be used during AST construction in order to
support `CALL ... YIELD *` syntax.
Reviewers: mferencevic, ipaljak
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2586
Summary:
The garbage collector had a race condition when it would delete deltas that
were in the middle of an object's delta chain. In the process of deleting
(unlinking) the delta, the garbage collector previously wouldn't acquire any
locks. That operation was then racing with the standard MVCC
`CreateAndLinkDelta` function that adds a new delta into the chain.
Fortunately, `CreateAndLinkDelta` always does its modifications while holding a
lock to the owner of the chain (either a vertex or an edge) so this change just
adds the lock acquiring to the garbage collector.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2582
Summary:
The functions that previously had locks in them are always called while the
vertex lock is already being held. Also, the lock guards were implemented
incorrectly.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2580
Summary:
Edge filters (edge type and destination vertex) are now handled natively in the
storage API. The API is implemented to be the fastest possible when using the
filters with the assumption that the number of edge types will be small.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2576
Summary:
This diff renames `__reload__` procedure to be `mg.reload` accepting a
module name. The main CallCustomProcedure function is now split into
multiple parts, so that there's more control over finding a procedure,
type checking its arguments and finally checking the returned result
set.
Depends on D2572
Reviewers: mferencevic, ipaljak
Reviewed By: ipaljak
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2573
Summary:
The query execution now has a timeout for each Cypher query it executes. The
timeout is implemented using TSC and will work only when TSC is available (same
as PROFILE). TSC is used to mitigate the performance impact of reading the
current time constantly.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2562
Summary:
Now when iterating over a label+property index the index verifies that the
bounds meet the criteria imposed by openCypher.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2552
Summary:
This simplifies the C API and reduces total allocations done when
constructing a type.
Reviewers: mferencevic, ipaljak
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2550
Summary:
Previously, when accessing the labels/properties/edges of a vertex/edge that
was just created the NEW view would correctly display the change, but the OLD
view would be invalid and would crash the database. With this change the OLD
view of a freshly created vertex/edge won't cause a crash, but will instead
report an error.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2549
Summary:
The type system is modelled after "CIP2015-09-16"
https://github.com/opencypher/openCypher/blob/master/cip/1.accepted/CIP2015-09-16-public-type-system-type-annotation.adoc
This is needed for registering procedures and their signatures. The
users will be able to specify what a custom procedure accepts and
returns. All of this needs to be available for inspection during
runtime. Therefore, this diff implements printing types as a user
presentable string. In the future, we will probably want to add type
checking through these types, because openCypher requires type checking
on values passed in and returned from custom procedures.
Reviewers: mferencevic, ipaljak, dsantl
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2544
Summary: Also test Explain and Profile through Intepreter.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2546
Summary:
The dumper is now a function and doesn't have to worry about any state. The
function streams the Cypher queries directly to the client. This diff also
makes the dumper work with storage v2.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2545
Summary:
The atomic memory order should be `acquire` for `load` operations, `release`
for `store` operations and `acq_rel` for any RMW (read-modify-write) operations
(like `fetch_add`).
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2540
Summary:
- Add the `AnyStream` wrapper
- Remove the `Results` struct and store a function (handler) instead
Reviewers: teon.banek, mferencevic
Reviewed By: teon.banek, mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2497
Summary:
All mgp_* symbols are exported from Memgraph executable, no other
symbols should be visible.
The primary C API header, mg_procedure.h, is now part of the
installation. Also, added a shippable query module example.
Directory `query_modules` is meant to contain sources of modules we
write and ship as part of the installation. Currently, there's only an
example module, but there may be potentially more. Some modules could
only be installed as part of the enterprise release.
For Memgraph to load custom procedures, it needs to be started with a
flag pointing to a directory with compiled shared libraries implementing
those procedures.
Reviewers: mferencevic, ipaljak, llugovic, dsantl, buda
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2538
Summary:
This change increases the throughput of the storage v2 durability 20x. With
this change, the storage v2 durability is 3x faster than the storage v1
durability in both recovery and snapshotting (before the change v2 durability
is slower than v1 durability).
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2529
Summary:
Store accumulated results as `communication::bolt::Value`s instead of
`TypedValue`s.
Add additional overloads for `Result` and `Summary` which accept `TypedValue`s
but internally perform conversions.
Reviewers: teon.banek, mferencevic
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2514
Summary:
Depends on D2471
- Add pointer to storage to `InterpreterContext`
- Rename `operator()` to `Prepare`
- Use `Interpret` instead of `operator()` (`Interpret` will be removed soon)
- Remove the `in_explicit_transaction` parameter
- Remove the memory resource parameter from `Interpret`
- Remove the storage accessor parameter from `Interpret`
- Fix up tests (remove the `Interpreter` from `database_transaction_timeout`)
Reviewers: teon.banek, mferencevic
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2482
Summary: Make `InterpreterContext` a top level instead of a nested struct
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2512
