* Add base of e2e tests
* Add python dependencies
* Explicitly close customer in destructor
* Parametrize tests and add test for CHECK STREAM
* Add tests for SHOW STREAMS
* Add test for concurrent start/stop during check
* Add test for calling check with an already started stream
* Run streams e2e tests on CI servers
Co-authored-by: antonio2368 <antonio2368@users.noreply.github.com>
Co-authored-by: Jure Bajic <jbajic@users.noreply.github.com>
* Use the correct transformation result type
* Execute the result queries in streams
* Change the result type of parameters to nullable map
* Serialize transformation name
* Fix order of transformation parameters
* Use actual transformation in Streams
* Clear the Python transformations under GIL
* Add CHECK STREAM query
* Handle missing record fields properly
* Add CREATE, START, STOP and DROP queries
* Fix definition of port in replica query
* Explicitly stop the consumer before removing
* Fix offset committing in Consumer
* Add tests for basic stream queries
* Remove unnecessary WITH keywords from CREATE query
* Add tests
* Add STREAM privilege
* Disable not working test
The functionality is tested manually, but I couldn't make it work with
the mock kafka cluster.
* Add support for multiple topic names
* Replace skiplist by synchronized map
* Make Consumer::Test const and improve error handling
The improvement in the error handling is mostly done regarding to the
Test function. Instead of trying to revert the assignments, Test just
stores the last commited assignment. When Start or Test is called, they
check for the last commited assignments, and if it is saved, then they
try to restore it. This way:
1. All the failures are returned to the user (failed to save/restore)
2. Failed assignment cannot terminate Memgraph
* Make Test do not block creating/droping other streams
* Added public interface for registering mgp_trans and extended modules accordingly
* Added test for mgp_trans
* Added mg.transformations() to the module registry
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
* Stop the Consumer grafefully when it is destroyed
* Add Streams
* Add Streams to InterpreterContext
* Remove options to limit processed batches in Consumer
* Add Streams unit tests
* Stop waiting for a full batch if the Consumer stopped
* Add ReadLock functionality to Synchronized
* Use per Consumer-based locking
* Replace shared_mutex with RWLock
This PR introduces READ COMMITTED and READ UNCOMMITTED isolation levels.
The isolation level can be set with a config or with a query for different scopes.
Sending the py::Object by value caused UB because multiple threads could copy the same object at the same time without the GIL.
By sending the object by reference we eliminate UB and avoid UB.
* Throw OOMException while creating vertices and edges
* Throw on indices creation
* Throw on setting a property
* Throw oom exception while recovering
* Throw exception when query engine asks for extra memory
* Block out of memor exception during skip list GC
* Add LOAD CSV clause infrastructure
* Add LoadCsv operator
* Update csv::Reader class
* Support csv files with and without header
Co-authored-by: jseljan <josip.seljan@memgraph.io>
* Added handshake support
* Add support for v4 hello and goodbye
* Add support for pulling n results
* Add support for transactions
* Add pull n for the dump
* Add support for NOOP
* Add support for multiple queries
* Update bolt session to support qid
* Update drivers test with multiple versions and go
* Extract failure handling into a function
* Use unique ptr instead of optional for query execution
* Destroy stream before query execution
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
Summary:
This diff fixes the variable expand operator to work correctly then the start
and destination nodes use the same symbol or when the destination symbol is an
existing symbol.
Previously, the variable expand operator produced paths that were both
completely wrong (they shouldn't have been produced) and nonexistent (they
didn't even exist in the storage). Invalid data was produced because of a
wrong equality check that was introduced in D1703.
This issue was reported externally and the supplied test case was:
```
CREATE (p1:Person {id: 1})-[:KNOWS]->(:Person {id: 2})-[:KNOWS]->(:Person {id: 3})-[:KNOWS]->(:Person {id: 4})-[:KNOWS]->(p1);
MATCH path = (pers:Person {id: 3})-[:KNOWS*2]->(pers) RETURN path;
```
Also, tests have been added so the behavior remains correct.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2793
Summary:
Semantics of 'all' and 'single' were updated to be
consistent with that of 'any' and 'none'
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2789
Summary: Change any function's handling of Null elements in a list
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2787
Summary:
`DUMP DATABASE` used a separate transaction to read database data. That
wouldn't be an issue if the query was correctly disallowed in multicommand
transactions. Because it was allowed the output wasn't transactionally correct.
Instead of disabling `DUMP DATABASE` in multicommand transactions this change
fixes it so that it works properly in multicommand transactions.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2781
Summary:
Before this change properties were joined by ", " and returned as a single string,
which was ambiguous for properties that contain ", ". This diff solves this
problem by returning properties as a list type.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2778
Summary:
This diff fixes the issue for label name (and edge type/property)
with spaces and special characters to avoid possible OpenCypher injections.
Consider an example where label name is 'hello :world'. `DUMP DATABASE`
used to return query which creates a node (u:hello :world) - i.e. node
that contains two labels 'hello' and 'world'. This fix escapes names to
create the following node with exactly one label as expected:
```
(u:`hello :world`)
```
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2774
Summary:
Use Unwind + ScanAllByLabelPropertyValue logical operator to
accelerate the execution of queries like the following one:
`MATCH (n:Label) WHERE n.property IN [] ...`
Reviewers: llugovic, mferencevic
Reviewed By: llugovic, mferencevic
Subscribers: llugovic, pullbot
Differential Revision: https://phabricator.memgraph.io/D2758
Summary: All integers are signed in openCypher.
Reviewers: buda, llugovic
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2746
Summary:
- Add the `mgp_must_abort(const mgp_graph *graph)` C API.
- Add the `ProcCtx.must_abort()` Python API.
The usage is very simple -- the function returns a boolean indicating whether
the procedure should abort.
Reviewers: mferencevic, dsantl
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2742
Summary:
The Python `ExceptionInfo` data has the potential to extend lifetime of Python
objects that were on the Python frame while the execution was thrown. This
lifetime extension is very dangerous when executing Python procedures because
we use a custom internal memory allocator that is destroyed immediately when
the procedure is done with its execution.
This diff only keeps the formatted Python traceback (as a string) to avoid any
lifetime extension for the Python objects so that the memory allocator can
safely be destroyed after the procedure has finished its execution.
Reviewers: llugovic
Reviewed By: llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2739
Summary:
`PyObject_Init` shouldn't be used in conjunction with `PyObject_New`.
The official Python documentation doesn't say this explicitly and many examples
on the web suggest that they must be used together. When they are used together
everything works when Memgraph is built against the release Python binary, but
doesn't work when built against the debug Python binary.
An implicit example that `PyObject_Init` shouldn't be used with `PyObject_New`
can be found in the Python source, in the file `Include/objimpl.h`:
```
This example code implements an object constructor with a custom
allocator, where PyObject_New is inlined, and shows the important
distinction between two steps (at least):
1) the actual allocation of the object storage;
2) the initialization of the Python specific fields
in this storage with PyObject_{Init, InitVar}.
PyObject *
YourObject_New(...)
{
PyObject *op;
op = (PyObject *) Your_Allocator(_PyObject_SIZE(YourTypeStruct));
if (op == NULL)
return PyErr_NoMemory();
PyObject_Init(op, &YourTypeStruct);
op->ob_field = value;
...
return op;
}
Note that in C++, the use of the new operator usually implies that
the 1st step is performed automatically for you, so in a C++ class
constructor you would start directly with PyObject_Init/InitVar
```
It explains that `PyObject_New` is actually equal to `malloc` +
`PyObject_Init`.
Reviewers: llugovic
Reviewed By: llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2737
Summary:
This fixes an issue in Py(Vertex|Edge)GetProperty and prevents any
further issues of that type at the cost of additional typing effort.
Reviewers: ipaljak, llugovic
Reviewed By: ipaljak, llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2735
Summary:
Currently, when starting Memgraph with the production package (DEB/RPM),
Memgraph always outputs an error for not being able to replace an existing
query module (`example.so` with `example.c`). This diff introduces a precheck
so that the error message is correct - so that Memgraph doesn't try to replace
an `.so` file with a `.c` file before verifying that the `.c` file is a valid
query module (which it obviously isn't). Also, I have moved the source of the
example into a subdirectory so that it isn't even considered while loading
modules.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2724
Summary:
You should now be able to invoke query procedures written in Python. To
test the example you can run memgraph with PYTHONPATH set to `include`.
For example, assuming you are in the root of the repo, run this command.
PYTHONPATH=$PWD/include ./build/memgraph --query-modules-directory=./query_modules/
Alternatively, you can set a symlink inside the ./query_modules to point
to `include/mgp.py`, so there's no need to set PYTHONPATH. For example,
assuming you are in the root of the repo, run the following.
cd ./query_modules
ln -s ../include/mgp.py
cd ..
./build/memgraph --query-modules-directory=./query_modules/
Depends on D207
Reviewers: mferencevic, ipaljak, dsantl
Reviewed By: ipaljak
Subscribers: buda, tlastre, pullbot
Differential Revision: https://phabricator.memgraph.io/D2708
Summary:
When invoking a Python registered procedures we want to convert
`mgp_value` types to user facing 'mgp' types.
Depends on D2706
Reviewers: mferencevic, ipaljak
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2707
Summary:
Now that unique constraint feature is added to Memgraph database,
we should update `DUMP DATABASE` with list of existing unique constraints.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2698
Summary:
With this diff you should now be able to register `example.py` and read
procedures found there. The procedures will be listed through `CALL
mg.procedures() YIELD *` query, but invoking them will raise
`NotYetImplemented`.
If you wish to test this, you will need to run the Memgraph executable
with PYTHONPATH set to the `include` directory where `mgp.py` is found.
Additionally, you need to pass `--query-modules-directory` flag to
Memgraph, such that it points to where it will find the `example.py`.
For example, when running from the root directory of Memgraph repo, the
shell invocation below should do the trick (assuming `./build/memgraph`
is where is the executable). Make sure that `./query_modules/` does not
have `example.so` built, as that may interfere with loading
`example.py`.
PYTHONPATH=$PWD/include ./build/memgraph --query-modules-directory=./query_modules/
Reviewers: mferencevic, ipaljak, llugovic
Reviewed By: mferencevic, ipaljak, llugovic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2678
Summary:
Before this change, unique constraints supported only pairs of label
and a single property. With this change, unique constraints can be
created for label and set of properties.
Better tests for unique constraints in general are also included in
this diff.
Reviewers: mferencevic, teon.banek
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2653
Summary:
`query::procedure::Module` is now an interface class through which
users interact with loaded modules. At the moment, the interface is implemented
by three concrete classes: `query::procedure::BuiltinModule`,
`query::procedure::SharedLibraryModule` and `query::procedure::PythonModule`.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2664
