Commit Graph

2926 Commits

Author SHA1 Message Date
antonio2368
435af8b833
Remove clang tidy from hooks (#94)
* Removed clang-tidy from hooks

* Improved clang-format output

* Remove lcp from types to format
2021-02-16 11:22:59 +01:00
antonio2368
cc1c1513ef
Increase column width (#91)
* Increase column width

* Prevent clang-format/tidy from triggering diff build
2021-02-12 16:58:55 +01:00
Marko Budiselić
fae407d3fe
Add implementation of average number of equals estimate in SkipList (#9)
* Implement average number of equals estimate in SkipList
2021-02-10 14:38:54 +01:00
Marko Budiselić
42c245df8a
Fix CentOS8 packaging (#87) 2021-01-29 14:48:55 +01:00
Marko Budiselić
e4852cc5e3
Migrate release docs to Notion (#86) 2021-01-28 16:29:00 +01:00
Marko Budiselić
2caf0e617e
Remove docs/dev (migrated to Notion) (#84) 2021-01-26 20:08:40 +01:00
Marko Budiselić
90d4ebdb1e
Polish CI scripts (#83)
* Add Jepsen to the release workflow
* Increase the long-running stress timeout to 16 minutes
* Remove ha_client code
2021-01-26 18:57:23 +01:00
Marko Budiselić
151812da1a
Add v1.3.0 CHANGELOG (#82) 2021-01-26 18:27:23 +01:00
Marko Budiselić
1e10b9cb77
Merge pull request #11
* Add replication
* Add Jepsen tests
* Change from glog to spdlog
* Add LOCK DATA DIRECTORY
* Add query type deduction
2021-01-25 17:49:14 +01:00
Marko Budiselić
c44dbdda86
Remove --min-log-level from mgbench (#81) 2021-01-25 09:14:08 +01:00
Josip Seljan
46b1783618
Forbid write queries on a replica (#80)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-25 09:13:32 +01:00
antonio2368
e866526c7a
Fix Centos build (#79)
* Don't run DocsHowToQuery.java on CentOS

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.com>
2021-01-24 08:17:21 +01:00
antonio2368
28413fd626 Change log library to spdlog, expose log levels to user (#72)
* Change from glog to spdlog

* Remove HA tests

* Remove logrotate log configuration

* Define custom main for unit gtests
2021-01-21 16:30:55 +01:00
antonio2368
c0bd59bb09 Fix recovery process (#78)
* Use correct last loaded timestamp for WAL
2021-01-21 15:56:21 +01:00
antonio2368
a84ffe86c1 Add LOCK DATA DIRECTORY query (#68) 2021-01-21 15:56:21 +01:00
Josip Seljan
8f5b88f24a Fix how query read/write type RWType is determined (#71)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
afbf672915 Add end2end replication tests based on mgclient (#69)
* Remove old HA benchmark and integration tests
2021-01-21 15:56:21 +01:00
antonio2368
10c8256ec9 Fix epoch id handling (#73) 2021-01-21 15:56:21 +01:00
Josip Seljan
b19cd4f5d1 Fix advanced part in replication feature spec (#75)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
8fc9298832 Avoid crashing the db when the ip address is invalid in a REGISTER REPLICA query (#74)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
9966ba1d52 Improve Jepsen run script (#64)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
b7bbd026de Add large Jepsen test (#67)
* Fix WAL recovery step
2021-01-21 15:56:21 +01:00
antonio2368
855c2ea9ca Add sequential test (#66) 2021-01-21 15:56:21 +01:00
antonio2368
adc355a22a Small replication fixes (#59)
* Set state to invalid after exception

* Add proper locking

* Start background replicating only if in valid state

* Freeze transaction timestamp on replica

* Timeout fixes

* Fix Jepsen run script

* Disable perf checker and enable nemesis

* Add documentation for some chunks of code

* Decrease timeout so main doesn't hang on network partitions too long
2021-01-21 15:56:21 +01:00
Josip Seljan
bff9cf07de Fix replication queries (#65)
* Fix replication query visitors
* Improve error handling in ReplQueryHandler methods

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
60d742a2dc Jepsen test infrastracture improvements and bank test (#62)
* Define replication config for tests

* Add support for final generator

* Add bank test

* Add host name resolution and basic replication setup

* Add timeout support

* Define helper macros for replication tests

* Add nemesis configuration
2021-01-21 15:56:21 +01:00
Josip Seljan
a0fb3fc463 Implement ReplicationQueryHandler class (#52)
* Refactor io::network::Endpoint class
* Add ParseSocketOrIpAddress static method to Endpoint class
* Implement ReplQueryHandler methods
* Add implementation of SetReplicationRole to ReplQueryHandler
* Fix PrepareReplicationQuery (create PullPlanVector)

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
f23e2e12c4 Add Jepsen initialization (#53)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
87e00f4fef Add support for specifying the replica port to SET REPLICATION ROLE query (#61)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
f7b764607d Improve error handling (#56) 2021-01-21 15:49:32 +01:00
antonio2368
200ce5f45e Add configs and support for semi-sync and SSL (#55)
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
2021-01-21 15:49:32 +01:00
antonio2368
a0705746cb Add epoch id and refactor replication client/server (#51) 2021-01-21 15:49:32 +01:00
Josip Seljan
4e36b646df Replace "CREATE REPLICA" query with "REGISTER REPLICA" (#50)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
Josip Seljan
ec909ced57 Update nomenclature in the replication feature spec (#47)
* Update nomenclature in the replication feature spec

* Update the replication feature spec to contain the changes to the new replica registering syntax

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
7e9175052a Define communication process (#49)
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
2021-01-21 15:49:32 +01:00
Josip Seljan
3c85319701 Update replication queries to follow the new replication nomenclature (#48)
* MODE -> ROLE

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
76f0d5873b Use unfinished tasks num instead of idle thread num (#45) 2021-01-21 15:49:32 +01:00
antonio2368
03cc568e39 Add support for async replication (#41)
* Add thread pool
* Define async replication
* Expose replication state
* Rename TransactionHandler to ReplicaStream
2021-01-21 15:49:32 +01:00
antonio2368
bc0c944910 Add replica recovery process (#40)
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
42f6118c00 Add safe file deletion utility FileRetainer (#38)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
b10255a12f Add initial support for multiple clients (#31)
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
c68ed8d94e Add implementation of synchronous replication (#7)
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.

To run the main instance use the following command:
```
./memgraph \
    --main \
    --data-directory main-data \
    --storage-properties-on-edges \
    --storage-wal-enabled \
    --storage-snapshot-interval-sec 300
```

To run the replica instance use the following command:
```
./memgraph \
    --replica \
    --data-directory replica-data \
    --storage-properties-on-edges \
    --bolt-port 7688
```

You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.

NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.

* Add basic synchronous replication test
* Using RWLock for replication stuff

Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
d5b02eafb1 Add syntactic support for replication queries - prototype (#8)
Extend parser to handle replication queries - first implementation.

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
1580c46abc Write replication feature spec (#6)
Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
367cb44983
Fix CentOS 8 powertools setup (#76) 2021-01-19 10:07:14 +01:00
Marko Budiselić
77bda187d2
Add install_all_apt environment util function (#77) 2021-01-15 09:26:15 +01:00
Marko Budiselić
57806544cd
Add new feature specs (#43) 2021-01-13 11:20:05 +01:00
antonio2368
7bff678cd9
Clear query executions if not in explicit transaction (#70) 2021-01-12 19:10:32 +01:00
Marko Budiselić
647388792a Add upload of Diff binaries (#58)
* Check RESULT_VARIABLE after get_version script call
2020-12-21 16:41:46 +01:00
Marko Budiselić
b42ee31c1d
Add ability to specify the exact Python version (#57) 2020-12-21 13:21:51 +01:00