Commit Graph

2934 Commits

Author SHA1 Message Date
Marko Budiselić
866ed45562
Add daily build for all major latest OSs () 2021-03-16 09:13:50 +01:00
János Benjamin Antal
1598cb24ea
Add directory argument to get_version.py ()
* Add root-dir option to get_version.py

* Make CMake callable outside from the repository
2021-03-15 11:55:39 +01:00
antonio2368
35d789c56b
Allow multiple folders for query modules ()
* Allow multiple query modules directories as arg
2021-03-10 12:18:09 +01:00
antonio2368
16715d5005
Enable replication in community ()
* Enable replication in community
2021-03-10 10:36:38 +01:00
antonio2368
a9f5f45b3d
Improve telemetry data ()
* Add query type count to telemetry

* Extract telemetry data in different structure

* Add global event counter
2021-03-09 19:55:58 +01:00
antonio2368
2e0dd19bac
Add machine-id to telemetry if not docker ()
* Add machine-id to telemetry if not docker

* Improve telemetry integration test
2021-03-08 15:15:58 +01:00
antonio2368
f807b495ab
Update commit log after recovery ()
* Define additional commit log constructor which takes an oldest active id

* Delay commit log construction until the recovery process is finished

* Add test for commit log with initial id

* Silence the macro redefinition warning
2021-02-19 11:00:10 +01:00
antonio2368
3f3c55a4aa
Format all the memgraph and test source files () 2021-02-18 15:32:43 +01:00
antonio2368
435af8b833
Remove clang tidy from hooks ()
* Removed clang-tidy from hooks

* Improved clang-format output

* Remove lcp from types to format
2021-02-16 11:22:59 +01:00
antonio2368
cc1c1513ef
Increase column width ()
* Increase column width

* Prevent clang-format/tidy from triggering diff build
2021-02-12 16:58:55 +01:00
Marko Budiselić
fae407d3fe
Add implementation of average number of equals estimate in SkipList ()
* Implement average number of equals estimate in SkipList
2021-02-10 14:38:54 +01:00
Marko Budiselić
42c245df8a
Fix CentOS8 packaging () 2021-01-29 14:48:55 +01:00
Marko Budiselić
e4852cc5e3
Migrate release docs to Notion () 2021-01-28 16:29:00 +01:00
Marko Budiselić
2caf0e617e
Remove docs/dev (migrated to Notion) () 2021-01-26 20:08:40 +01:00
Marko Budiselić
90d4ebdb1e
Polish CI scripts ()
* Add Jepsen to the release workflow
* Increase the long-running stress timeout to 16 minutes
* Remove ha_client code
2021-01-26 18:57:23 +01:00
Marko Budiselić
151812da1a
Add v1.3.0 CHANGELOG () 2021-01-26 18:27:23 +01:00
Marko Budiselić
1e10b9cb77
Merge pull request
* Add replication
* Add Jepsen tests
* Change from glog to spdlog
* Add LOCK DATA DIRECTORY
* Add query type deduction
2021-01-25 17:49:14 +01:00
Marko Budiselić
c44dbdda86
Remove --min-log-level from mgbench () 2021-01-25 09:14:08 +01:00
Josip Seljan
46b1783618
Forbid write queries on a replica ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-25 09:13:32 +01:00
antonio2368
e866526c7a
Fix Centos build ()
* Don't run DocsHowToQuery.java on CentOS

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.com>
2021-01-24 08:17:21 +01:00
antonio2368
28413fd626 Change log library to spdlog, expose log levels to user ()
* Change from glog to spdlog

* Remove HA tests

* Remove logrotate log configuration

* Define custom main for unit gtests
2021-01-21 16:30:55 +01:00
antonio2368
c0bd59bb09 Fix recovery process ()
* Use correct last loaded timestamp for WAL
2021-01-21 15:56:21 +01:00
antonio2368
a84ffe86c1 Add LOCK DATA DIRECTORY query () 2021-01-21 15:56:21 +01:00
Josip Seljan
8f5b88f24a Fix how query read/write type RWType is determined ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
afbf672915 Add end2end replication tests based on mgclient ()
* Remove old HA benchmark and integration tests
2021-01-21 15:56:21 +01:00
antonio2368
10c8256ec9 Fix epoch id handling () 2021-01-21 15:56:21 +01:00
Josip Seljan
b19cd4f5d1 Fix advanced part in replication feature spec ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
8fc9298832 Avoid crashing the db when the ip address is invalid in a REGISTER REPLICA query ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
9966ba1d52 Improve Jepsen run script ()
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
b7bbd026de Add large Jepsen test ()
* Fix WAL recovery step
2021-01-21 15:56:21 +01:00
antonio2368
855c2ea9ca Add sequential test () 2021-01-21 15:56:21 +01:00
antonio2368
adc355a22a Small replication fixes ()
* Set state to invalid after exception

* Add proper locking

* Start background replicating only if in valid state

* Freeze transaction timestamp on replica

* Timeout fixes

* Fix Jepsen run script

* Disable perf checker and enable nemesis

* Add documentation for some chunks of code

* Decrease timeout so main doesn't hang on network partitions too long
2021-01-21 15:56:21 +01:00
Josip Seljan
bff9cf07de Fix replication queries ()
* Fix replication query visitors
* Improve error handling in ReplQueryHandler methods

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
60d742a2dc Jepsen test infrastracture improvements and bank test ()
* Define replication config for tests

* Add support for final generator

* Add bank test

* Add host name resolution and basic replication setup

* Add timeout support

* Define helper macros for replication tests

* Add nemesis configuration
2021-01-21 15:56:21 +01:00
Josip Seljan
a0fb3fc463 Implement ReplicationQueryHandler class ()
* Refactor io::network::Endpoint class
* Add ParseSocketOrIpAddress static method to Endpoint class
* Implement ReplQueryHandler methods
* Add implementation of SetReplicationRole to ReplQueryHandler
* Fix PrepareReplicationQuery (create PullPlanVector)

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
f23e2e12c4 Add Jepsen initialization ()
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
87e00f4fef Add support for specifying the replica port to SET REPLICATION ROLE query ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
f7b764607d Improve error handling () 2021-01-21 15:49:32 +01:00
antonio2368
200ce5f45e Add configs and support for semi-sync and SSL ()
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
2021-01-21 15:49:32 +01:00
antonio2368
a0705746cb Add epoch id and refactor replication client/server () 2021-01-21 15:49:32 +01:00
Josip Seljan
4e36b646df Replace "CREATE REPLICA" query with "REGISTER REPLICA" ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
Josip Seljan
ec909ced57 Update nomenclature in the replication feature spec ()
* Update nomenclature in the replication feature spec

* Update the replication feature spec to contain the changes to the new replica registering syntax

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
7e9175052a Define communication process ()
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
2021-01-21 15:49:32 +01:00
Josip Seljan
3c85319701 Update replication queries to follow the new replication nomenclature ()
* MODE -> ROLE

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
76f0d5873b Use unfinished tasks num instead of idle thread num () 2021-01-21 15:49:32 +01:00
antonio2368
03cc568e39 Add support for async replication ()
* Add thread pool
* Define async replication
* Expose replication state
* Rename TransactionHandler to ReplicaStream
2021-01-21 15:49:32 +01:00
antonio2368
bc0c944910 Add replica recovery process ()
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
42f6118c00 Add safe file deletion utility FileRetainer ()
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
b10255a12f Add initial support for multiple clients ()
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
c68ed8d94e Add implementation of synchronous replication ()
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.

To run the main instance use the following command:
```
./memgraph \
    --main \
    --data-directory main-data \
    --storage-properties-on-edges \
    --storage-wal-enabled \
    --storage-snapshot-interval-sec 300
```

To run the replica instance use the following command:
```
./memgraph \
    --replica \
    --data-directory replica-data \
    --storage-properties-on-edges \
    --bolt-port 7688
```

You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.

NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.

* Add basic synchronous replication test
* Using RWLock for replication stuff

Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00