Commit Graph

1064 Commits

Author SHA1 Message Date
Kostas Kyrimis
4004e94ca1 Introduce messages C-API (#168) 2021-07-07 15:38:51 +02:00
János Benjamin Antal
36afc6c5f3 Add Kafka consumer (#167)
* Add CMake config for Kafka integration

* Add Consumer

* Add simple unit test for consumer

* Add explicit offset handling and test for it
2021-07-07 15:38:51 +02:00
antonio2368
3b336e3e0b
Add CREATE SNAPSHOT query (#182) 2021-06-30 12:31:30 +02:00
antonio2368
715162e205
Disable LOAD CSV with config (#180) 2021-06-30 11:19:13 +02:00
antonio2368
e016c74e4b
Trigger restore after query modules load (#183) 2021-06-30 10:01:51 +02:00
antonio2368
15911b64dc
Use timers for query timeout thread (#163)
Co-authored-by: Benjamin Antal <benjamin.antal@memgraph.io>
2021-06-16 16:03:58 +02:00
antonio2368
8cd9f696cf
Multiple isolation levels (#171)
This PR introduces READ COMMITTED and READ UNCOMMITTED isolation levels.
The isolation level can be set with a config or with a query for different scopes.
2021-06-14 15:47:57 +02:00
János Benjamin Antal
a3ecc52429 Use all available predefined variables (#157) 2021-05-26 14:37:11 +02:00
János Benjamin Antal
e8a1d15a55 Add filtering based on registered event types (#155)
* Add filtering to TriggerContextCollector

* Add all predefined variable to ANY triggers

* Make variable names consistent with event types
2021-05-26 14:37:11 +02:00
János Benjamin Antal
5af3d0ff68 Add e2e tests for triggers (#152) 2021-05-26 14:36:41 +02:00
antonio2368
62a628c51f Add trigger queries and trigger store (#145)
Co-authored-by: Benjamin Antal <benjamin.antal@memgraph.io>
2021-05-26 14:36:41 +02:00
antonio2368
883f9c7ed3 Trigger event types (#144)
Co-authored-by: Benjamin Antal <benjamin.antal@memgraph.io>
2021-05-26 14:13:14 +02:00
antonio2368
b459639968 Other predefined variables (#143) 2021-05-26 14:13:14 +02:00
antonio2368
11c0dde11c Support removed vertices in the triggers (#140) 2021-05-26 14:13:14 +02:00
antonio2368
2f3fa656d9 Predefined variables (#138)
* Add support for injecting predefined variables in any scope
2021-05-26 14:13:14 +02:00
antonio2368
13ea35af2d
Fix permission for newer queries (#156) 2021-05-26 10:02:25 +02:00
antonio2368
30413a7b4f
Ignore carriage return at the end of line for a CSV file (#151) 2021-05-18 13:44:29 +02:00
János Benjamin Antal
6d4fe5cdd5
Explore and implement the usage of clang-tidy and sanitizers (#125)
* Run clang-tidy on the full code base

* Run clang-tidy on diffs

* Enable ASAN in coverage build

* Add UBSAN to code analysis
2021-04-01 17:08:40 +02:00
antonio2368
276e09d7d3
Fix reading rows with empty columns at the end (#127)
* Fix reading rows with empty columns at the end

* Update CHANGELOG for the recovery logs
2021-03-27 09:47:41 +01:00
antonio2368
ad4c80af13 Add memory limit clause for cypher queries (#106)
* Add memory limit clause for cypher queries

* Add PROCEDURE and QUERY keywords

* Improve memory limit logs

* Update CHANGELOG
2021-03-24 15:21:11 +01:00
antonio2368
cc56ac3dd8 Expose query for cleaning memory and add memory limit flag (#100)
* Add memory flag, add additional meminfo utilities

* Add free memory query
2021-03-24 15:10:28 +01:00
antonio2368
bbed7a2397 Add memory tracker definition (#93)
* Allow size 0 in MemoryTracker
* Block only exception throwing
* Subtract unsuccessfully allocated size in memory tracker
* Add oom exception enablers and blockers
2021-03-24 14:53:15 +01:00
antonio2368
e8810a4152 Add and use jemalloc lib (#90)
* Add and use jemalloc lib

* Add autoconf

* Silence macro redefinition warning
2021-03-24 14:53:15 +01:00
Josip Seljan
25eb2c147a LOAD CSV implementation fixes (#120)
* Change how csv::Reader handles memory resources
* Add multiline quoted string test
2021-03-24 12:21:07 +01:00
Josip Seljan
f950a91732 Add LOAD CSV clause infrastructure (#101)
* Add LOAD CSV clause infrastructure
* Add LoadCsv operator
* Update csv::Reader class
* Support csv files with and without header

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-03-24 12:21:07 +01:00
Josip Seljan
f6d5f576d5 CSV reader class (#98)
* CSV parsing class
* Basic CSV parsing tests
2021-03-24 12:21:07 +01:00
antonio2368
16715d5005
Enable replication in community (#104)
* Enable replication in community
2021-03-10 10:36:38 +01:00
antonio2368
f807b495ab
Update commit log after recovery (#96)
* Define additional commit log constructor which takes an oldest active id

* Delay commit log construction until the recovery process is finished

* Add test for commit log with initial id

* Silence the macro redefinition warning
2021-02-19 11:00:10 +01:00
antonio2368
3f3c55a4aa
Format all the memgraph and test source files (#97) 2021-02-18 15:32:43 +01:00
Marko Budiselić
fae407d3fe
Add implementation of average number of equals estimate in SkipList (#9)
* Implement average number of equals estimate in SkipList
2021-02-10 14:38:54 +01:00
antonio2368
28413fd626 Change log library to spdlog, expose log levels to user (#72)
* Change from glog to spdlog

* Remove HA tests

* Remove logrotate log configuration

* Define custom main for unit gtests
2021-01-21 16:30:55 +01:00
antonio2368
a84ffe86c1 Add LOCK DATA DIRECTORY query (#68) 2021-01-21 15:56:21 +01:00
Josip Seljan
8f5b88f24a Fix how query read/write type RWType is determined (#71)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
8fc9298832 Avoid crashing the db when the ip address is invalid in a REGISTER REPLICA query (#74)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
a0fb3fc463 Implement ReplicationQueryHandler class (#52)
* Refactor io::network::Endpoint class
* Add ParseSocketOrIpAddress static method to Endpoint class
* Implement ReplQueryHandler methods
* Add implementation of SetReplicationRole to ReplQueryHandler
* Fix PrepareReplicationQuery (create PullPlanVector)

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:56:21 +01:00
Josip Seljan
87e00f4fef Add support for specifying the replica port to SET REPLICATION ROLE query (#61)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
200ce5f45e Add configs and support for semi-sync and SSL (#55)
* Add config for replication client/server
* Add SSL to replication
* Add semi-sync replication
* Expose necessary information about replication
* Thread pool fix
* Set BasicResult value type to void
2021-01-21 15:49:32 +01:00
antonio2368
a0705746cb Add epoch id and refactor replication client/server (#51) 2021-01-21 15:49:32 +01:00
Josip Seljan
4e36b646df Replace "CREATE REPLICA" query with "REGISTER REPLICA" (#50)
Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
7e9175052a Define communication process (#49)
* Add basic communication process using commit timestamp
* Add file number to req
* Add proper recovery handling
* Allow loading of WALs with same seq num
* Allow always desired commit timestamp
* Set replica timestamp for operation
* Mark non-transactional timestamp as finished
2021-01-21 15:49:32 +01:00
Josip Seljan
3c85319701 Update replication queries to follow the new replication nomenclature (#48)
* MODE -> ROLE

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
76f0d5873b Use unfinished tasks num instead of idle thread num (#45) 2021-01-21 15:49:32 +01:00
antonio2368
03cc568e39 Add support for async replication (#41)
* Add thread pool
* Define async replication
* Expose replication state
* Rename TransactionHandler to ReplicaStream
2021-01-21 15:49:32 +01:00
antonio2368
bc0c944910 Add replica recovery process (#40)
* Add file transfer over RPC
* Snapshot transfer implementation
* Allow snapshot creation only for MAIN instances
* Replica and main can have replication clients
* Use only snapshots and WALs that are from the Main storage
* Add flush lock and expose buffer
* Add fstat for file size and TryFlushing method
* Use lseek for size

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
42f6118c00 Add safe file deletion utility FileRetainer (#38)
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
b10255a12f Add initial support for multiple clients (#31)
* Add tests for multiple clients
* Use variant for RPC server and clients
* Using synchronized list for replication clients, extracted variant access to a function
* Set MAIN as default, add unregister function, add a name for replication clients
* Use the regular list for clients
* Use test fixture so storage directory is cleaned
* Use seq_cst for replication_state

Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
c68ed8d94e Add implementation of synchronous replication (#7)
This implements the initial version of synchronous replication.
Currently, only one replica is supported and that isn't configurable.

To run the main instance use the following command:
```
./memgraph \
    --main \
    --data-directory main-data \
    --storage-properties-on-edges \
    --storage-wal-enabled \
    --storage-snapshot-interval-sec 300
```

To run the replica instance use the following command:
```
./memgraph \
    --replica \
    --data-directory replica-data \
    --storage-properties-on-edges \
    --bolt-port 7688
```

You can then write/read data to Bolt port 7687 (the main instance) and also you
can read the data from the replica instance using Bolt port 7688.

NOTE: The main instance *must* be started without any data and the replica
*must* be started before any data is added to the main instance.

* Add basic synchronous replication test
* Using RWLock for replication stuff

Co-authored-by: Matej Ferencevic <matej.ferencevic@memgraph.io>
Co-authored-by: Antonio Andelic <antonio.andelic@memgraph.io>
2021-01-21 15:49:32 +01:00
Marko Budiselić
d5b02eafb1 Add syntactic support for replication queries - prototype (#8)
Extend parser to handle replication queries - first implementation.

Co-authored-by: jseljan <josip.seljan@memgraph.io>
2021-01-21 15:49:32 +01:00
antonio2368
4e7c569071 Migrate code to C++20 (#44) 2020-11-17 11:03:08 +01:00
Marko Budiselić
958bc870b3
Migrate to toolchain-v2 (#33)
* GCC_VERSION=10.2.0
* BINUTILS_VERSION=2.35.1
* GDB_VERSION=10.1 (except centos-7 8.3)
* CMAKE_VERSION=3.18.4
* CPPCHECK_VERSION=2.2
* LLVM_VERSION=11.0.0
* SWIG_VERSION=4.0.2
2020-11-12 20:18:11 +01:00