Commit Graph

1493 Commits

Author SHA1 Message Date
antonio2368
5c93f81881
Disable failing tests and add logs for replication e2e (#132)
* Disable sequential test

* Remove parent build and benchmark

* Save test data

* Save e2e logs in build folder

* Define different recovery time for each test
2021-04-02 12:29:10 +02:00
János Benjamin Antal
6d4fe5cdd5
Explore and implement the usage of clang-tidy and sanitizers (#125)
* Run clang-tidy on the full code base

* Run clang-tidy on diffs

* Enable ASAN in coverage build

* Add UBSAN to code analysis
2021-04-01 17:08:40 +02:00
antonio2368
276e09d7d3
Fix reading rows with empty columns at the end (#127)
* Fix reading rows with empty columns at the end

* Update CHANGELOG for the recovery logs
2021-03-27 09:47:41 +01:00
János Benjamin Antal
06f761bdf9
Add logs for loading snapshot and WAL files (#121)
* Add logs for loading snapshot and WAL files
2021-03-26 15:02:35 +01:00
antonio2368
50ddd59450
Fix Centos8 build (#126)
* Increase server wait

* Use standard shebang for python scripts
2021-03-26 12:05:58 +01:00
antonio2368
e8e4cd7f97 Memory control e2e test (#115)
* Add memory control e2e test

* Fix cmake for jemalloc
2021-03-24 15:21:18 +01:00
antonio2368
ad4c80af13 Add memory limit clause for cypher queries (#106)
* Add memory limit clause for cypher queries

* Add PROCEDURE and QUERY keywords

* Improve memory limit logs

* Update CHANGELOG
2021-03-24 15:21:11 +01:00
antonio2368
cc56ac3dd8 Expose query for cleaning memory and add memory limit flag (#100)
* Add memory flag, add additional meminfo utilities

* Add free memory query
2021-03-24 15:10:28 +01:00
antonio2368
bbed7a2397 Add memory tracker definition (#93)
* Allow size 0 in MemoryTracker
* Block only exception throwing
* Subtract unsuccessfully allocated size in memory tracker
* Add oom exception enablers and blockers
2021-03-24 14:53:15 +01:00
antonio2368
e8810a4152 Add and use jemalloc lib (#90)
* Add and use jemalloc lib

* Add autoconf

* Silence macro redefinition warning
2021-03-24 14:53:15 +01:00
Josip Seljan
25eb2c147a LOAD CSV implementation fixes (#120)
* Change how csv::Reader handles memory resources
* Add multiline quoted string test
2021-03-24 12:21:07 +01:00
Josip Seljan
f950a91732 Add LOAD CSV clause infrastructure (#101)
* Add LOAD CSV clause infrastructure
* Add LoadCsv operator
* Update csv::Reader class
* Support csv files with and without header

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-03-24 12:21:07 +01:00
Josip Seljan
f6d5f576d5 CSV reader class (#98)
* CSV parsing class
* Basic CSV parsing tests
2021-03-24 12:21:07 +01:00
antonio2368
c240b5b564
Remove v1 driver test for python (#109) 2021-03-16 11:42:36 +01:00
antonio2368
16715d5005
Enable replication in community (#104)
* Enable replication in community
2021-03-10 10:36:38 +01:00
antonio2368
2e0dd19bac
Add machine-id to telemetry if not docker (#103)
* Add machine-id to telemetry if not docker

* Improve telemetry integration test
2021-03-08 15:15:58 +01:00
antonio2368
f807b495ab
Update commit log after recovery (#96)
* Define additional commit log constructor which takes an oldest active id

* Delay commit log construction until the recovery process is finished

* Add test for commit log with initial id

* Silence the macro redefinition warning
2021-02-19 11:00:10 +01:00
antonio2368
3f3c55a4aa
Format all the memgraph and test source files (#97) 2021-02-18 15:32:43 +01:00
Marko Budiselić
fae407d3fe
Add implementation of average number of equals estimate in SkipList (#9)
* Implement average number of equals estimate in SkipList
2021-02-10 14:38:54 +01:00
Marko Budiselić
90d4ebdb1e
Polish CI scripts (#83)
* Add Jepsen to the release workflow
* Increase the long-running stress timeout to 16 minutes
* Remove ha_client code
2021-01-26 18:57:23 +01:00
Marko Budiselić
c44dbdda86
Remove --min-log-level from mgbench (#81) 2021-01-25 09:14:08 +01:00
antonio2368
e866526c7a
Fix Centos build (#79)
* Don't run DocsHowToQuery.java on CentOS

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.com>
2021-01-24 08:17:21 +01:00
antonio2368
28413fd626 Change log library to spdlog, expose log levels to user (#72)
* Change from glog to spdlog

* Remove HA tests

* Remove logrotate log configuration

* Define custom main for unit gtests
2021-01-21 16:30:55 +01:00
antonio2368
a84ffe86c1 Add LOCK DATA DIRECTORY query (#68) 2021-01-21 15:56:21 +01:00
Josip Seljan
8f5b88f24a Fix how query read/write type RWType is determined (#71)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
afbf672915 Add end2end replication tests based on mgclient (#69)
* Remove old HA benchmark and integration tests
2021-01-21 15:56:21 +01:00
Josip Seljan
8fc9298832 Avoid crashing the db when the ip address is invalid in a REGISTER REPLICA query (#74)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
9966ba1d52 Improve Jepsen run script (#64)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
antonio2368
b7bbd026de Add large Jepsen test (#67)
* Fix WAL recovery step
2021-01-21 15:56:21 +01:00
antonio2368
855c2ea9ca Add sequential test (#66) 2021-01-21 15:56:21 +01:00
antonio2368
adc355a22a Small replication fixes (#59)
* Set state to invalid after exception

* Add proper locking

* Start background replicating only if in valid state

* Freeze transaction timestamp on replica

* Timeout fixes

* Fix Jepsen run script

* Disable perf checker and enable nemesis

* Add documentation for some chunks of code

* Decrease timeout so main doesn't hang on network partitions too long
2021-01-21 15:56:21 +01:00
antonio2368
60d742a2dc Jepsen test infrastracture improvements and bank test (#62)
* Define replication config for tests

* Add support for final generator

* Add bank test

* Add host name resolution and basic replication setup

* Add timeout support

* Define helper macros for replication tests

* Add nemesis configuration
2021-01-21 15:56:21 +01:00
Josip Seljan
a0fb3fc463 Implement ReplicationQueryHandler class (#52)
* Refactor io::network::Endpoint class
* Add ParseSocketOrIpAddress static method to Endpoint class
* Implement ReplQueryHandler methods
* Add implementation of SetReplicationRole to ReplQueryHandler
* Fix PrepareReplicationQuery (create PullPlanVector)

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Marko Budiselić
f23e2e12c4 Add Jepsen initialization (#53)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
87e00f4fef Add support for specifying the replica port to SET REPLICATION ROLE query (#61)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
200ce5f45e Add configs and support for semi-sync and SSL (#55)
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
2021-01-21 15:49:32 +01:00
antonio2368
a0705746cb Add epoch id and refactor replication client/server (#51) 2021-01-21 15:49:32 +01:00
Josip Seljan
4e36b646df Replace "CREATE REPLICA" query with "REGISTER REPLICA" (#50)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
7e9175052a Define communication process (#49)
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
2021-01-21 15:49:32 +01:00
Josip Seljan
3c85319701 Update replication queries to follow the new replication nomenclature (#48)
* MODE -> ROLE

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
76f0d5873b Use unfinished tasks num instead of idle thread num (#45) 2021-01-21 15:49:32 +01:00
antonio2368
03cc568e39 Add support for async replication (#41)
* Add thread pool
* Define async replication
* Expose replication state
* Rename TransactionHandler to ReplicaStream
2021-01-21 15:49:32 +01:00
antonio2368
bc0c944910 Add replica recovery process (#40)
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
42f6118c00 Add safe file deletion utility FileRetainer (#38)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
b10255a12f Add initial support for multiple clients (#31)
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
c68ed8d94e Add implementation of synchronous replication (#7)
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.

To run the main instance use the following command:
```
./memgraph \
    --main \
    --data-directory main-data \
    --storage-properties-on-edges \
    --storage-wal-enabled \
    --storage-snapshot-interval-sec 300
```

To run the replica instance use the following command:
```
./memgraph \
    --replica \
    --data-directory replica-data \
    --storage-properties-on-edges \
    --bolt-port 7688
```

You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.

NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.

* Add basic synchronous replication test
* Using RWLock for replication stuff

Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
d5b02eafb1 Add syntactic support for replication queries - prototype (#8)
Extend parser to handle replication queries - first implementation.

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
573b6cb045
Clean and upgrade openCypher and stress tests (#27)
* Add initial behave test improvements
* Make HTML human-readable
* Remove tutorial tests
* Migrate to neo4j-driver==4.1.1
2020-11-24 13:09:14 +01:00
antonio2368
4e7c569071 Migrate code to C++20 (#44) 2020-11-17 11:03:08 +01:00
Marko Budiselić
958bc870b3
Migrate to toolchain-v2 (#33)
* GCC_VERSION=10.2.0
* BINUTILS_VERSION=2.35.1
* GDB_VERSION=10.1 (except centos-7 8.3)
* CMAKE_VERSION=3.18.4
* CPPCHECK_VERSION=2.2
* LLVM_VERSION=11.0.0
* SWIG_VERSION=4.0.2
2020-11-12 20:18:11 +01:00