The communication between the ShardRequestManager and the RsmClient
used to be direct. In this PR this changes into a future-based
communication type. The RsmClient stores state about the currently
processed future (either read or write request) and exposes blocking
and non-blocking functionality to obtain the filled future. The
ShardRequestManager -for now- will send of the set of requests present
in the ExecutionState and block on each of them until the requests are
completed or the set of paginated responses(caused by, for example the
batch-limit in ScanAll) are ready for the next round.
* Use query-v2 in the main executable
* Set up machine manager in memgraph
* Add `ShardRequestManager` to `Interpreter`
* Make vertex creation work
* Make scan all work
* Add edge type map in shard request manager
* Send schema over request
* Empty out DbAccessor
* Store shard mapping at creation
* Remove failing CI steps
Cooltura is the best place in Zagreb!
Co-authored-by: János Benjamin Antal <benjamin.antal@memgraph.io>
- Enable `mg-expr` in mg-storage-v3
- Adapt id_mapper
- Change conversion function from PropertyValue to TypedValue
- Remove memgraph functions
- Enable expression tests on for storage
Create shard-side handlers for basic messages
Implement the handlers for CreateVertices, CreateEdges and ScanAll. Use
or modify the defined messages to interact with individual Shards and
test their behavior. Shard is currently being owned by ShardRsm
instances. The two top level dispatching functions Read() and Apply()
are responsible for read- and write operations respectively. Currently
there are a handful of messages that are defined but not utilized, these
will be used in the near future, as well as a couple of handler
functions with empty implementations.
- Integrate schema and keystore on vertex creation
- Add GC test for storage v3
- Add tests for accessors
- Fix all tests related to this except for query v2
- Fix labels not returning primary label
* Create LocalTransport Io provider for sending messages to components on the same machine
* Move src/io/simulation/message_conversion.hpp to src/io/message_conversion.hpp for use in other Io providers
* Move schema to storage v3
* Remove schema from v2
* Move schema to query v2
* Remove schema from query v1
* Make glue v2
* Move schema related tests to newer versions of query and storage
* Fix typo in CMake
* Fix interpreter test
* Fix clang tidy errors
* Change temp dir name
* Fix aggregation functions on `null` and group-by inputs (#448)
* Upgrade Antrl to 4.10.1 and remove antlr_lock (#441)
* Update clang-tidy job (#476)
* Add parser stress test (#463)
NOTE: Doing this to have buildable comments on the project-pineapples branch
Co-authored-by: gvolfing <107616712+gvolfing@users.noreply.github.com>
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
* Make `IfOperator` return the `else_expression_` in case of `NULL`
* Add gql_behave tests
* Add gql_behave test to specifically check for the case when the test expression itself is null
* Modify `toString` to be able to handle `Date`, `LocalTime`, `LocalDateTime` and `Duration`
* Add unit tests
* Make `operator<<` use the `ToString()` implementations
* Add tests to verify the correctness of negative durations
* Add more tests to look for cases when the individual duration entities overflow.
* Add initial schema implementation
* Add index to schema
* List schemas and enable multiple properties
* Implement SchemaTypes
* Apply suggestions from code review
Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
* Address review comments
* Remove Map and List
* Add schema operations in storage
* Add create and show schema queries
* Add privileges for schema
* Add missing keywords into lexer
* Add drop schema query
* Add schema visitors
* Update metadata
* Add PrepareSchemaQuery function
* Implement show schemas
* Add show schema query
* Fix schema visitor
* Add common schema type
* Fix grammar
* Temporary create ddl logic
* Fix naming for schemaproperty type to schema type
* Rename schemaproperty to schemapropertytype
* Enable Create schema ddl
* Override visitPropertyType
* Add initial schema implementation
* Add initial schema implementation
* Add index to schema
* List schemas and enable multiple properties
* Implement SchemaTypes
* Apply suggestions from code review
Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
* Address review comments
* Remove Map and List
* Apply suggestions from code review
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
* Add verification on creation and deletion
* Rename DeleteSchema to DropSchema
* Remove list and map from lexer
* Fix grammar with schemaTypeMap
* Add privilege and cypher visitor tests
* Catch repeating type name in schema definition
* Fix conflicting keywords
* Add notifications
* Drop float support
* Finish interpreter tests
* Fix tests
* Fix clang tidy errors
* Fix GetSchema
* Replace for with transfrom
* Add cloning og schema_property_map
* Address review comments
* Rename SchemaPropertyType to SchemaType
* Remove inline
* Assert of schema properties
Co-authored-by: Jeremy B <97525434+42jeremy@users.noreply.github.com>
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
* Storage takes care of the saving of setting when a new replica is added
* Restore replicas at startup
* Modify interactive_mg_runner + memgraph to support that data-directory can be configured in CONTEXT
* Extend e2e test
* Correct typo
* Add flag to config to specify when replication should be stored (true by default when starting Memgraph)
* Remove un-necessary "--" in yaml file
* Make sure Memgraph stops if a replica can't be restored.
* Add UT covering the parsing of ReplicaStatus to/from json
* Add assert in e2e script to check that a port is free before using it
* Add test covering crash on Jepsen
* Make sure applciaiton crashes if it starts on corrupted replications' info
Starting with a non-reponsive replica is allowed.
* Add temporary startup flag: this is needed so jepsen do not automatically restore replica on startup of main. This will be removed in T0835
* Copy storage v2 to v3
* Integrate v3 to cmake
* Fix clang-tidy warnings
* Add dummy unit test for storage-v3 to trigger build for code analysis builds
* Add test
* Add implementation and adapted test
* Update workloads.yaml to have a timeout > 0
* Update tests (failing due to merging of "add replica state")
* Adding "raw message" column to the result return by CHECK STREAM query
* Update way results of CHECK STREAM are built
* Adapting CHECK STREAM integration tests (Pulsar/Kafka) to run with new result structure
* Adding new tests covering the check stream functionality
* Uppercase constants in stream tests
* Reformat f-strings
* Moving function add_query_module from CMakeLists from tests/e2e/magic_functions to tests/e2e
* Adding failing test copying behavior when loading c module (.so) into memgraph.
* Fixing issue where NO_ERROR status returned MgpTransAddFixedResult was converted to false
* Removing unnecessary transformation
* removing incorrect parameterization of test
* re-adding parametrized transformation
* Extend mgp_module with include adding functions
* Add return type to the function API
* Change Cypher grammar
* Add Python support for functions
* Implement error handling
* E2e tests for functions
* Write cpp e2e functions
* Create mg.functions() procedure
* Implement case insensitivity for user-defined Magic Functions.
* Fix doc of mgp_graph_vertices
* Make write_proc example meaningful write procedure example
* Improve wrap_exceptions
* Add check for write procedures for ReadWriteTypeChecker
* Change error code in case of invalid default value for optional arguments
* Add base of e2e tests
* Add python dependencies
* Explicitly close customer in destructor
* Parametrize tests and add test for CHECK STREAM
* Add tests for SHOW STREAMS
* Add test for concurrent start/stop during check
* Add test for calling check with an already started stream
* Run streams e2e tests on CI servers
Co-authored-by: antonio2368 <antonio2368@users.noreply.github.com>
Co-authored-by: Jure Bajic <jbajic@users.noreply.github.com>
* Use the correct transformation result type
* Execute the result queries in streams
* Change the result type of parameters to nullable map
* Serialize transformation name
* Fix order of transformation parameters
* Use actual transformation in Streams
* Clear the Python transformations under GIL
* Add CHECK STREAM query
* Handle missing record fields properly
* Add CREATE, START, STOP and DROP queries
* Fix definition of port in replica query
* Explicitly stop the consumer before removing
* Fix offset committing in Consumer
* Add tests for basic stream queries
* Remove unnecessary WITH keywords from CREATE query
* Add tests
* Add STREAM privilege
* Disable not working test
The functionality is tested manually, but I couldn't make it work with
the mock kafka cluster.
* Add support for multiple topic names
* Replace skiplist by synchronized map
* Make Consumer::Test const and improve error handling
The improvement in the error handling is mostly done regarding to the
Test function. Instead of trying to revert the assignments, Test just
stores the last commited assignment. When Start or Test is called, they
check for the last commited assignments, and if it is saved, then they
try to restore it. This way:
1. All the failures are returned to the user (failed to save/restore)
2. Failed assignment cannot terminate Memgraph
* Make Test do not block creating/droping other streams
* Added public interface for registering mgp_trans and extended modules accordingly
* Added test for mgp_trans
* Added mg.transformations() to the module registry
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
* Stop the Consumer grafefully when it is destroyed
* Add Streams
* Add Streams to InterpreterContext
* Remove options to limit processed batches in Consumer
* Add Streams unit tests
* Stop waiting for a full batch if the Consumer stopped
* Add ReadLock functionality to Synchronized
* Use per Consumer-based locking
* Replace shared_mutex with RWLock
This PR introduces READ COMMITTED and READ UNCOMMITTED isolation levels.
The isolation level can be set with a config or with a query for different scopes.
* Disable sequential test
* Remove parent build and benchmark
* Save test data
* Save e2e logs in build folder
* Define different recovery time for each test
* Add LOAD CSV clause infrastructure
* Add LoadCsv operator
* Update csv::Reader class
* Support csv files with and without header
Co-authored-by: jseljan <josip.seljan@memgraph.io>
* Define additional commit log constructor which takes an oldest active id
* Delay commit log construction until the recovery process is finished
* Add test for commit log with initial id
* Silence the macro redefinition warning
* Set state to invalid after exception
* Add proper locking
* Start background replicating only if in valid state
* Freeze transaction timestamp on replica
* Timeout fixes
* Fix Jepsen run script
* Disable perf checker and enable nemesis
* Add documentation for some chunks of code
* Decrease timeout so main doesn't hang on network partitions too long
* Define replication config for tests
* Add support for final generator
* Add bank test
* Add host name resolution and basic replication setup
* Add timeout support
* Define helper macros for replication tests
* Add nemesis configuration
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.
To run the main instance use the following command:
```
./memgraph \
--main \
--data-directory main-data \
--storage-properties-on-edges \
--storage-wal-enabled \
--storage-snapshot-interval-sec 300
```
To run the replica instance use the following command:
```
./memgraph \
--replica \
--data-directory replica-data \
--storage-properties-on-edges \
--bolt-port 7688
```
You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.
NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.
* Add basic synchronous replication test
* Using RWLock for replication stuff
Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
