* Add base of e2e tests
* Add python dependencies
* Explicitly close customer in destructor
* Parametrize tests and add test for CHECK STREAM
* Add tests for SHOW STREAMS
* Add test for concurrent start/stop during check
* Add test for calling check with an already started stream
* Run streams e2e tests on CI servers
Co-authored-by: antonio2368 <antonio2368@users.noreply.github.com>
Co-authored-by: Jure Bajic <jbajic@users.noreply.github.com>
* Use the correct transformation result type
* Execute the result queries in streams
* Change the result type of parameters to nullable map
* Serialize transformation name
* Fix order of transformation parameters
* Use actual transformation in Streams
* Clear the Python transformations under GIL
* Add CHECK STREAM query
* Handle missing record fields properly
* Add CREATE, START, STOP and DROP queries
* Fix definition of port in replica query
* Explicitly stop the consumer before removing
* Fix offset committing in Consumer
* Add tests for basic stream queries
* Remove unnecessary WITH keywords from CREATE query
* Add tests
* Add STREAM privilege
* Disable not working test
The functionality is tested manually, but I couldn't make it work with
the mock kafka cluster.
* Add support for multiple topic names
* Replace skiplist by synchronized map
* Make Consumer::Test const and improve error handling
The improvement in the error handling is mostly done regarding to the
Test function. Instead of trying to revert the assignments, Test just
stores the last commited assignment. When Start or Test is called, they
check for the last commited assignments, and if it is saved, then they
try to restore it. This way:
1. All the failures are returned to the user (failed to save/restore)
2. Failed assignment cannot terminate Memgraph
* Make Test do not block creating/droping other streams
* Added public interface for registering mgp_trans and extended modules accordingly
* Added test for mgp_trans
* Added mg.transformations() to the module registry
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
* Stop the Consumer grafefully when it is destroyed
* Add Streams
* Add Streams to InterpreterContext
* Remove options to limit processed batches in Consumer
* Add Streams unit tests
* Stop waiting for a full batch if the Consumer stopped
* Add ReadLock functionality to Synchronized
* Use per Consumer-based locking
* Replace shared_mutex with RWLock
This PR introduces READ COMMITTED and READ UNCOMMITTED isolation levels.
The isolation level can be set with a config or with a query for different scopes.
* Disable sequential test
* Remove parent build and benchmark
* Save test data
* Save e2e logs in build folder
* Define different recovery time for each test
* Add LOAD CSV clause infrastructure
* Add LoadCsv operator
* Update csv::Reader class
* Support csv files with and without header
Co-authored-by: jseljan <josip.seljan@memgraph.io>
* Define additional commit log constructor which takes an oldest active id
* Delay commit log construction until the recovery process is finished
* Add test for commit log with initial id
* Silence the macro redefinition warning
* Set state to invalid after exception
* Add proper locking
* Start background replicating only if in valid state
* Freeze transaction timestamp on replica
* Timeout fixes
* Fix Jepsen run script
* Disable perf checker and enable nemesis
* Add documentation for some chunks of code
* Decrease timeout so main doesn't hang on network partitions too long
* Define replication config for tests
* Add support for final generator
* Add bank test
* Add host name resolution and basic replication setup
* Add timeout support
* Define helper macros for replication tests
* Add nemesis configuration
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.
To run the main instance use the following command:
```
./memgraph \
--main \
--data-directory main-data \
--storage-properties-on-edges \
--storage-wal-enabled \
--storage-snapshot-interval-sec 300
```
To run the replica instance use the following command:
```
./memgraph \
--replica \
--data-directory replica-data \
--storage-properties-on-edges \
--bolt-port 7688
```
You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.
NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.
* Add basic synchronous replication test
* Using RWLock for replication stuff
Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
* Added handshake support
* Add support for v4 hello and goodbye
* Add support for pulling n results
* Add support for transactions
* Add pull n for the dump
* Add support for NOOP
* Add support for multiple queries
* Update bolt session to support qid
* Update drivers test with multiple versions and go
* Extract failure handling into a function
* Use unique ptr instead of optional for query execution
* Destroy stream before query execution
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
Summary:
This change only adds streaming support to the client request. The client
response, server request and server response are still handled only when all of
the data is received.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2807
Summary:
SLK now correctly handles different CPU architectures (BIG/little endian).
Also, more string encoding functions have been added.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2806
Summary:
This diff fixes the variable expand operator to work correctly then the start
and destination nodes use the same symbol or when the destination symbol is an
existing symbol.
Previously, the variable expand operator produced paths that were both
completely wrong (they shouldn't have been produced) and nonexistent (they
didn't even exist in the storage). Invalid data was produced because of a
wrong equality check that was introduced in D1703.
This issue was reported externally and the supplied test case was:
```
CREATE (p1:Person {id: 1})-[:KNOWS]->(:Person {id: 2})-[:KNOWS]->(:Person {id: 3})-[:KNOWS]->(:Person {id: 4})-[:KNOWS]->(p1);
MATCH path = (pers:Person {id: 3})-[:KNOWS*2]->(pers) RETURN path;
```
Also, tests have been added so the behavior remains correct.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2793
Summary:
Semantics of 'all' and 'single' were updated to be
consistent with that of 'any' and 'none'
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2789
Summary: Change any function's handling of Null elements in a list
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2787
