Commit Graph

1493 Commits

Author SHA1 Message Date
antonio2368
5c93f81881
Disable failing tests and add logs for replication e2e ()
* Disable sequential test

* Remove parent build and benchmark

* Save test data

* Save e2e logs in build folder

* Define different recovery time for each test
2021-04-02 12:29:10 +02:00
János Benjamin Antal
6d4fe5cdd5
Explore and implement the usage of clang-tidy and sanitizers ()
* Run clang-tidy on the full code base

* Run clang-tidy on diffs

* Enable ASAN in coverage build

* Add UBSAN to code analysis
2021-04-01 17:08:40 +02:00
antonio2368
276e09d7d3
Fix reading rows with empty columns at the end ()
* Fix reading rows with empty columns at the end

* Update CHANGELOG for the recovery logs
2021-03-27 09:47:41 +01:00
János Benjamin Antal
06f761bdf9
Add logs for loading snapshot and WAL files ()
* Add logs for loading snapshot and WAL files
2021-03-26 15:02:35 +01:00
antonio2368
50ddd59450
Fix Centos8 build ()
* Increase server wait

* Use standard shebang for python scripts
2021-03-26 12:05:58 +01:00
antonio2368
e8e4cd7f97 Memory control e2e test ()
* Add memory control e2e test

* Fix cmake for jemalloc
2021-03-24 15:21:18 +01:00
antonio2368
ad4c80af13 Add memory limit clause for cypher queries ()
* Add memory limit clause for cypher queries

* Add PROCEDURE and QUERY keywords

* Improve memory limit logs

* Update CHANGELOG
2021-03-24 15:21:11 +01:00
antonio2368
cc56ac3dd8 Expose query for cleaning memory and add memory limit flag ()
* Add memory flag, add additional meminfo utilities

* Add free memory query
2021-03-24 15:10:28 +01:00
antonio2368
bbed7a2397 Add memory tracker definition ()
* Allow size 0 in MemoryTracker
* Block only exception throwing
* Subtract unsuccessfully allocated size in memory tracker
* Add oom exception enablers and blockers
2021-03-24 14:53:15 +01:00
antonio2368
e8810a4152 Add and use jemalloc lib ()
* Add and use jemalloc lib

* Add autoconf

* Silence macro redefinition warning
2021-03-24 14:53:15 +01:00
Josip Seljan
25eb2c147a LOAD CSV implementation fixes ()
* Change how csv::Reader handles memory resources
* Add multiline quoted string test
2021-03-24 12:21:07 +01:00
Josip Seljan
f950a91732 Add LOAD CSV clause infrastructure ()
* Add LOAD CSV clause infrastructure
* Add LoadCsv operator
* Update csv::Reader class
* Support csv files with and without header

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-03-24 12:21:07 +01:00
Josip Seljan
f6d5f576d5 CSV reader class ()
* CSV parsing class
* Basic CSV parsing tests
2021-03-24 12:21:07 +01:00
antonio2368
c240b5b564
Remove v1 driver test for python () 2021-03-16 11:42:36 +01:00
antonio2368
16715d5005
Enable replication in community ()
* Enable replication in community
2021-03-10 10:36:38 +01:00
antonio2368
2e0dd19bac
Add machine-id to telemetry if not docker ()
* Add machine-id to telemetry if not docker

* Improve telemetry integration test
2021-03-08 15:15:58 +01:00
antonio2368
f807b495ab
Update commit log after recovery ()
* Define additional commit log constructor which takes an oldest active id

* Delay commit log construction until the recovery process is finished

* Add test for commit log with initial id

* Silence the macro redefinition warning
2021-02-19 11:00:10 +01:00
antonio2368
3f3c55a4aa
Format all the memgraph and test source files () 2021-02-18 15:32:43 +01:00
Marko Budiselić
fae407d3fe
Add implementation of average number of equals estimate in SkipList ()
* Implement average number of equals estimate in SkipList
2021-02-10 14:38:54 +01:00
Marko Budiselić
90d4ebdb1e
Polish CI scripts ()
* Add Jepsen to the release workflow
* Increase the long-running stress timeout to 16 minutes
* Remove ha_client code
2021-01-26 18:57:23 +01:00
Marko Budiselić
c44dbdda86
Remove --min-log-level from mgbench () 2021-01-25 09:14:08 +01:00
antonio2368
e866526c7a
Fix Centos build ()
* Don't run DocsHowToQuery.java on CentOS

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.com>
2021-01-24 08:17:21 +01:00
antonio2368
28413fd626 Change log library to spdlog, expose log levels to user ()
* Change from glog to spdlog

* Remove HA tests

* Remove logrotate log configuration

* Define custom main for unit gtests
2021-01-21 16:30:55 +01:00
antonio2368
a84ffe86c1 Add LOCK DATA DIRECTORY query () 2021-01-21 15:56:21 +01:00
Josip Seljan
8f5b88f24a Fix how query read/write type RWType is determined ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
afbf672915 Add end2end replication tests based on mgclient ()
* Remove old HA benchmark and integration tests
2021-01-21 15:56:21 +01:00
Josip Seljan
8fc9298832 Avoid crashing the db when the ip address is invalid in a REGISTER REPLICA query ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
9966ba1d52 Improve Jepsen run script ()
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
b7bbd026de Add large Jepsen test ()
* Fix WAL recovery step
2021-01-21 15:56:21 +01:00
antonio2368
855c2ea9ca Add sequential test () 2021-01-21 15:56:21 +01:00
antonio2368
adc355a22a Small replication fixes ()
* Set state to invalid after exception

* Add proper locking

* Start background replicating only if in valid state

* Freeze transaction timestamp on replica

* Timeout fixes

* Fix Jepsen run script

* Disable perf checker and enable nemesis

* Add documentation for some chunks of code

* Decrease timeout so main doesn't hang on network partitions too long
2021-01-21 15:56:21 +01:00
antonio2368
60d742a2dc Jepsen test infrastracture improvements and bank test ()
* Define replication config for tests

* Add support for final generator

* Add bank test

* Add host name resolution and basic replication setup

* Add timeout support

* Define helper macros for replication tests

* Add nemesis configuration
2021-01-21 15:56:21 +01:00
Josip Seljan
a0fb3fc463 Implement ReplicationQueryHandler class ()
* Refactor io::network::Endpoint class
* Add ParseSocketOrIpAddress static method to Endpoint class
* Implement ReplQueryHandler methods
* Add implementation of SetReplicationRole to ReplQueryHandler
* Fix PrepareReplicationQuery (create PullPlanVector)

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
f23e2e12c4 Add Jepsen initialization ()
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
87e00f4fef Add support for specifying the replica port to SET REPLICATION ROLE query ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
200ce5f45e Add configs and support for semi-sync and SSL ()
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
2021-01-21 15:49:32 +01:00
antonio2368
a0705746cb Add epoch id and refactor replication client/server () 2021-01-21 15:49:32 +01:00
Josip Seljan
4e36b646df Replace "CREATE REPLICA" query with "REGISTER REPLICA" ()
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
7e9175052a Define communication process ()
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
2021-01-21 15:49:32 +01:00
Josip Seljan
3c85319701 Update replication queries to follow the new replication nomenclature ()
* MODE -> ROLE

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
76f0d5873b Use unfinished tasks num instead of idle thread num () 2021-01-21 15:49:32 +01:00
antonio2368
03cc568e39 Add support for async replication ()
* Add thread pool
* Define async replication
* Expose replication state
* Rename TransactionHandler to ReplicaStream
2021-01-21 15:49:32 +01:00
antonio2368
bc0c944910 Add replica recovery process ()
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
42f6118c00 Add safe file deletion utility FileRetainer ()
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
b10255a12f Add initial support for multiple clients ()
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
c68ed8d94e Add implementation of synchronous replication ()
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.

To run the main instance use the following command:
```
./memgraph \
    --main \
    --data-directory main-data \
    --storage-properties-on-edges \
    --storage-wal-enabled \
    --storage-snapshot-interval-sec 300
```

To run the replica instance use the following command:
```
./memgraph \
    --replica \
    --data-directory replica-data \
    --storage-properties-on-edges \
    --bolt-port 7688
```

You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.

NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.

* Add basic synchronous replication test
* Using RWLock for replication stuff

Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
d5b02eafb1 Add syntactic support for replication queries - prototype ()
Extend parser to handle replication queries - first implementation.

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
573b6cb045
Clean and upgrade openCypher and stress tests ()
* Add initial behave test improvements
* Make HTML human-readable
* Remove tutorial tests
* Migrate to neo4j-driver==4.1.1
2020-11-24 13:09:14 +01:00
antonio2368
4e7c569071 Migrate code to C++20 () 2020-11-17 11:03:08 +01:00
Marko Budiselić
958bc870b3
Migrate to toolchain-v2 ()
* GCC_VERSION=10.2.0
* BINUTILS_VERSION=2.35.1
* GDB_VERSION=10.1 (except centos-7 8.3)
* CMAKE_VERSION=3.18.4
* CPPCHECK_VERSION=2.2
* LLVM_VERSION=11.0.0
* SWIG_VERSION=4.0.2
2020-11-12 20:18:11 +01:00