* Throw OOMException while creating vertices and edges
* Throw on indices creation
* Throw on setting a property
* Throw oom exception while recovering
* Throw exception when query engine asks for extra memory
* Block out of memor exception during skip list GC
* Define additional commit log constructor which takes an oldest active id
* Delay commit log construction until the recovery process is finished
* Add test for commit log with initial id
* Silence the macro redefinition warning
* Set state to invalid after exception
* Add proper locking
* Start background replicating only if in valid state
* Freeze transaction timestamp on replica
* Timeout fixes
* Fix Jepsen run script
* Disable perf checker and enable nemesis
* Add documentation for some chunks of code
* Decrease timeout so main doesn't hang on network partitions too long
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.
To run the main instance use the following command:
```
./memgraph \
--main \
--data-directory main-data \
--storage-properties-on-edges \
--storage-wal-enabled \
--storage-snapshot-interval-sec 300
```
To run the replica instance use the following command:
```
./memgraph \
--replica \
--data-directory replica-data \
--storage-properties-on-edges \
--bolt-port 7688
```
You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.
NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.
* Add basic synchronous replication test
* Using RWLock for replication stuff
Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
Summary:
This diff improves the performance of `PropertyStore` with two main
techniques:
First:
`PropertyValue` has a very expensive constructor and destructor. The
`PropertyValue` was previously passed as a return value from many functions
wrapped in a `std::optional`. That caused the `PropertyValue`
constructor/destructor to be called for each intermediary value that was passed
between functions. This diff changes the functions to return a `bool` value
that imitates the `std::optional` "emptyness" flag and the `PropertyValue` is
modified using a pointer to it so that its constructor/destructor is called
only once.
Second:
The `PropertyStore` buffer was previously iterated through at least twice.
First to determine the exact position of the encoded property and then to
actually decode the property. This diff combines the two passes into a single
pass so that the property is immediately loaded if it is found.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2766
Summary:
In order for this race condition to cause damage, an index/constraint must be
created/dropped at the exact moment that the GC is cleaning
indices/constraints.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2733
Summary:
The storage now uses a file in the data directory (`.lock`) to determine
whether there is another instance of the storage running with the same data
directory. That helps notify the user/administrator that the system is running
in an unsupported configuration.
Reviewers: teon.banek, ipaljak
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2719
Summary:
This diff contains simple tests for unique constraints which tries to
change property values or labels in multiple threads at the same time.
During testing, a bug has been encountered in unique constraints, i.e.
one guard lock on vertices was missing.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: mferencevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D2711
Summary:
This diff contains a necessary functionality to save and restore unique
constraint operations. The previous snapshot/WAL version is backward
compatible. Integration tests for migration from older snapshot and WAL
versions are also included.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2680
Summary:
Before this change, unique constraints supported only pairs of label
and a single property. With this change, unique constraints can be
created for label and set of properties.
Better tests for unique constraints in general are also included in
this diff.
Reviewers: mferencevic, teon.banek
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2653
Summary:
This diff contains a basic implementation of unique constraints consistent with
the MVCC storage.
Stale records in the unique constraints are collected by the garbage collector.
Tests for checking correctness of unique constraints and violations are included.
Note: currently we only support a pair of label and a single property. Support for
multiple properties will be added later.
Reviewers: mferencevic, teon.banek
Reviewed By: mferencevic, teon.banek
Subscribers: buda, ipaljak, pullbot
Differential Revision: https://phabricator.memgraph.io/D2608
Summary:
The property store stores a map of `PropertyId` to `PropertyValue` mappings. It
compresses all of the values in order to use as little memory as possible.
Reviewers: teon.banek, ipaljak
Reviewed By: teon.banek
Subscribers: buda, pullbot
Differential Revision: https://phabricator.memgraph.io/D2604
Summary:
The garbage collector had a race condition when it would delete deltas that
were in the middle of an object's delta chain. In the process of deleting
(unlinking) the delta, the garbage collector previously wouldn't acquire any
locks. That operation was then racing with the standard MVCC
`CreateAndLinkDelta` function that adds a new delta into the chain.
Fortunately, `CreateAndLinkDelta` always does its modifications while holding a
lock to the owner of the chain (either a vertex or an edge) so this change just
adds the lock acquiring to the garbage collector.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2582
Summary:
The functions that previously had locks in them are always called while the
vertex lock is already being held. Also, the lock guards were implemented
incorrectly.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2580
Summary:
Edge filters (edge type and destination vertex) are now handled natively in the
storage API. The API is implemented to be the fastest possible when using the
filters with the assumption that the number of edge types will be small.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2576
Summary:
Now when iterating over a label+property index the index verifies that the
bounds meet the criteria imposed by openCypher.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2552
Summary:
Previously, when accessing the labels/properties/edges of a vertex/edge that
was just created the NEW view would correctly display the change, but the OLD
view would be invalid and would crash the database. With this change the OLD
view of a freshly created vertex/edge won't cause a crash, but will instead
report an error.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2549
