Commit Graph

2914 Commits

Author SHA1 Message Date
antonio2368
28413fd626 Change log library to spdlog, expose log levels to user (#72)
* Change from glog to spdlog

* Remove HA tests

* Remove logrotate log configuration

* Define custom main for unit gtests
2021-01-21 16:30:55 +01:00
antonio2368
c0bd59bb09 Fix recovery process (#78)
* Use correct last loaded timestamp for WAL
2021-01-21 15:56:21 +01:00
antonio2368
a84ffe86c1 Add LOCK DATA DIRECTORY query (#68) 2021-01-21 15:56:21 +01:00
Josip Seljan
8f5b88f24a Fix how query read/write type RWType is determined (#71)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
afbf672915 Add end2end replication tests based on mgclient (#69)
* Remove old HA benchmark and integration tests
2021-01-21 15:56:21 +01:00
antonio2368
10c8256ec9 Fix epoch id handling (#73) 2021-01-21 15:56:21 +01:00
Josip Seljan
b19cd4f5d1 Fix advanced part in replication feature spec (#75)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
8fc9298832 Avoid crashing the db when the ip address is invalid in a REGISTER REPLICA query (#74)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
9966ba1d52 Improve Jepsen run script (#64)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
b7bbd026de Add large Jepsen test (#67)
* Fix WAL recovery step
2021-01-21 15:56:21 +01:00
antonio2368
855c2ea9ca Add sequential test (#66) 2021-01-21 15:56:21 +01:00
antonio2368
adc355a22a Small replication fixes (#59)
* Set state to invalid after exception

* Add proper locking

* Start background replicating only if in valid state

* Freeze transaction timestamp on replica

* Timeout fixes

* Fix Jepsen run script

* Disable perf checker and enable nemesis

* Add documentation for some chunks of code

* Decrease timeout so main doesn't hang on network partitions too long
2021-01-21 15:56:21 +01:00
Josip Seljan
bff9cf07de Fix replication queries (#65)
* Fix replication query visitors
* Improve error handling in ReplQueryHandler methods

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
60d742a2dc Jepsen test infrastracture improvements and bank test (#62)
* Define replication config for tests

* Add support for final generator

* Add bank test

* Add host name resolution and basic replication setup

* Add timeout support

* Define helper macros for replication tests

* Add nemesis configuration
2021-01-21 15:56:21 +01:00
Josip Seljan
a0fb3fc463 Implement ReplicationQueryHandler class (#52)
* Refactor io::network::Endpoint class
* Add ParseSocketOrIpAddress static method to Endpoint class
* Implement ReplQueryHandler methods
* Add implementation of SetReplicationRole to ReplQueryHandler
* Fix PrepareReplicationQuery (create PullPlanVector)

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
f23e2e12c4 Add Jepsen initialization (#53)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
87e00f4fef Add support for specifying the replica port to SET REPLICATION ROLE query (#61)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
f7b764607d Improve error handling (#56) 2021-01-21 15:49:32 +01:00
antonio2368
200ce5f45e Add configs and support for semi-sync and SSL (#55)
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
2021-01-21 15:49:32 +01:00
antonio2368
a0705746cb Add epoch id and refactor replication client/server (#51) 2021-01-21 15:49:32 +01:00
Josip Seljan
4e36b646df Replace "CREATE REPLICA" query with "REGISTER REPLICA" (#50)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
Josip Seljan
ec909ced57 Update nomenclature in the replication feature spec (#47)
* Update nomenclature in the replication feature spec

* Update the replication feature spec to contain the changes to the new replica registering syntax

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
7e9175052a Define communication process (#49)
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
2021-01-21 15:49:32 +01:00
Josip Seljan
3c85319701 Update replication queries to follow the new replication nomenclature (#48)
* MODE -> ROLE

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
76f0d5873b Use unfinished tasks num instead of idle thread num (#45) 2021-01-21 15:49:32 +01:00
antonio2368
03cc568e39 Add support for async replication (#41)
* Add thread pool
* Define async replication
* Expose replication state
* Rename TransactionHandler to ReplicaStream
2021-01-21 15:49:32 +01:00
antonio2368
bc0c944910 Add replica recovery process (#40)
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
42f6118c00 Add safe file deletion utility FileRetainer (#38)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
b10255a12f Add initial support for multiple clients (#31)
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
c68ed8d94e Add implementation of synchronous replication (#7)
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.

To run the main instance use the following command:
```
./memgraph \
    --main \
    --data-directory main-data \
    --storage-properties-on-edges \
    --storage-wal-enabled \
    --storage-snapshot-interval-sec 300
```

To run the replica instance use the following command:
```
./memgraph \
    --replica \
    --data-directory replica-data \
    --storage-properties-on-edges \
    --bolt-port 7688
```

You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.

NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.

* Add basic synchronous replication test
* Using RWLock for replication stuff

Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
d5b02eafb1 Add syntactic support for replication queries - prototype (#8)
Extend parser to handle replication queries - first implementation.

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
1580c46abc Write replication feature spec (#6)
Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
367cb44983
Fix CentOS 8 powertools setup (#76) 2021-01-19 10:07:14 +01:00
Marko Budiselić
77bda187d2
Add install_all_apt environment util function (#77) 2021-01-15 09:26:15 +01:00
Marko Budiselić
57806544cd
Add new feature specs (#43) 2021-01-13 11:20:05 +01:00
antonio2368
7bff678cd9
Clear query executions if not in explicit transaction (#70) 2021-01-12 19:10:32 +01:00
Marko Budiselić
647388792a Add upload of Diff binaries (#58)
* Check RESULT_VARIABLE after get_version script call
2020-12-21 16:41:46 +01:00
Marko Budiselić
b42ee31c1d
Add ability to specify the exact Python version (#57) 2020-12-21 13:21:51 +01:00
Marko Budiselić
53fa1482cb
Update GHA jobs tags (#60) 2020-12-07 15:02:14 +01:00
antonio2368
c5f722b724
Document defining constructors for member variables (#54) 2020-12-02 10:11:51 +01:00
Marko Budiselić
573b6cb045
Clean and upgrade openCypher and stress tests (#27)
* Add initial behave test improvements
* Make HTML human-readable
* Remove tutorial tests
* Migrate to neo4j-driver==4.1.1
2020-11-24 13:09:14 +01:00
Marko Budiselić
4afcf1b655
Remove release examples (#42) 2020-11-23 15:46:03 +01:00
antonio2368
4e7c569071 Migrate code to C++20 (#44) 2020-11-17 11:03:08 +01:00
Marko Budiselić
ea27ac9391
Update workflow documentation (#36)
Co-authored-by: antonio2368 <antonio2368@users.noreply.github.com>
2020-11-12 20:45:29 +01:00
Marko Budiselić
958bc870b3
Migrate to toolchain-v2 (#33)
* GCC_VERSION=10.2.0
* BINUTILS_VERSION=2.35.1
* GDB_VERSION=10.1 (except centos-7 8.3)
* CMAKE_VERSION=3.18.4
* CPPCHECK_VERSION=2.2
* LLVM_VERSION=11.0.0
* SWIG_VERSION=4.0.2
2020-11-12 20:18:11 +01:00
antonio2368
f0382c82cd
Potential memory leak fix (#39)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2020-11-09 12:47:15 +01:00
Marko Budiselić
6fab357430
Upgrade cppitertools to v2.0 (#17) 2020-10-27 12:14:18 +01:00
antonio2368
f273be63a2
RPC client dangling reference fix (#34)
* Fix dangling reference
* Add docs for the dangling reference in lambda captures

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2020-10-26 13:33:48 +01:00
Marko Budiselić
9b984ab2f2
Add DISCARD fix to the CHANGELOG (#32) 2020-10-23 20:08:47 +02:00
Marko Budiselić
92bad4f62b
Add docs examples as driver tests (#29) 2020-10-23 18:02:33 +02:00