Commit Graph

501 Commits

Author SHA1 Message Date
Andi
1a3c5af797
Improve expansions on disk (#1335)
* Improve disk expansions
2023-10-11 10:18:50 +02:00
Gareth Andrew Lloyd
d278a33f31
Decouple pure replication state from storage [part 1] (#1325)
A major refactor to decouple replication state from storage.
ATM it is still owned by storage but a following part should fix that.
2023-10-10 11:44:19 +01:00
Gareth Andrew Lloyd
3cc2bc2791
Refactor interpreter to support multiple distributed clocks (Part 1) (#1281)
* Interpreter transaction ID decoupled from storage transaction ID
* Transactional scope for indices, statistics and constraints
* Storage::Accessor now has 2 modes (unique and shared)
* Introduced ResourceLock to fix pthread mutex problems
* Split InfoQuery in two: non-transactional SystemInfoQuery and transactional DatabaseInfoQuery
* Replicable and durable statistics
* Bumped WAL/Snapshot versions
* Initial implementation of the Lamport clock

---------

Co-authored-by: Andreja Tonev <andreja.tonev@memgraph.io>
2023-10-05 16:58:39 +02:00
Gareth Andrew Lloyd
d71b6a5007
Refactor replication client/server (#1311) 2023-09-29 11:21:42 +01:00
Andi
61ac7e1b11
Add --storage-mode flag (#1282)
* Add --storage-mode flag
2023-09-26 14:47:30 +02:00
Andi
1553fcb958
Improve deserialization performance
* Change std::stoull to std::from_chars
---------

Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-09-20 14:25:17 +02:00
andrejtonev
bce48361ca
Decoupling Interpreter from Storage (#1186)
Unique/global InterpreterContext that is Storage agnostic (has a reference to the DbmsHandler instead)

* InterpreterContext is no longer the owner of Storage
* New Database structure that handles Storage, Triggers, Streams
* Renamed SessinContextHandler to DbmsHandler and simplified the multi-tenant logic
* Added Gatekeeper and updated handlers to use it

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2023-09-20 13:13:54 +02:00
andrejtonev
5e5f4ffc5d
Add more runtime configurable settings (#1183)
server name, query timeout settings, log.level, log.to_stderr
2023-09-11 11:30:54 -04:00
Josipmrden
58546a9fe1
Add detach delete in bulk (#1078) 2023-09-10 18:53:03 +02:00
Gareth Andrew Lloyd
ab56abf4ca
Optimize scanning vertices (#1227) 2023-09-09 10:09:25 -04:00
Gareth Andrew Lloyd
1bd47318cd
Improve PropertyStore (#1142)
Improve AnyVersionHasLabelProperty by doing less work in some instances.
Improve FindSpecificProperty.
2023-09-09 08:00:43 -04:00
Antonio Filipovic
b094fdbadc
Fix API bug on accessing deleted object (#1209) 2023-09-08 13:52:21 -04:00
Josipmrden
07dea328d8
[master < T1110] Add merge optimization to expand dynamically during runtime (#1110) 2023-09-08 17:12:25 +02:00
imilinovic
9c51dbbb01
Implement changing from and to vertices in relationships (#1221) 2023-09-08 12:52:40 +02:00
Ante Javor
312d01bd0c
Remove repeated log lines from TRACE log level. (#1054) 2023-09-06 23:09:51 +02:00
Antonio Filipovic
b6b32bec03
Improve performance of delta creation (#1129) 2023-09-06 11:30:21 +02:00
Andi
b5413c6f82
Add edge import mode into the on-disk storage (#1157) 2023-09-05 19:00:53 +02:00
Gareth Andrew Lloyd
9661c52179
Introduce a reader writer spin lock (#1187)
It is possible for multiple read only queries to be accessing the same
sequence of vertices/edges. The reader mode of the spin lock will ensure
multiple threads can make progress at the same time.
2023-09-01 14:21:15 +01:00
Gareth Andrew Lloyd
e928eed028
Replication refactor (part 4) (#1211)
More refactoring to isolate generic replication behavior. Making the 
InMemory* types even more decoupled from replication logic.
2023-08-31 16:06:44 +01:00
Andi
a6ec81b179
Add deterministic disk vertex_count and edge_count (#1146)
* Add exact vertex_count and edge_count to disk storage
2023-08-29 13:07:23 +02:00
Aidar Samerkhanov
5f509532f2
Add timestamp to DELETE_DESERIALIZED_OBJECT delta at which this object was created. (#1179)
Add timestamp to DELETE_DESERIALIZED_OBJECT delta at which this object was created.
RocksDB currently doesn't provide timestamp() functionality in iterators of TransationDB.
Because of that we are using constant "0" timestamp for DELETE_DESERIALIZED_OBJECT.
2023-08-28 10:56:17 +04:00
Andi
4b3ba908c7
Code improvements on disk storage (#1153)
* Improvements based on a code review

---------

Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-08-26 14:16:12 +02:00
Andi
030b554ffd
Improve concurrency control for on-disk storage (#1154)
* Remove locking vertices and serialization checks
---------

Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-08-25 14:42:52 +02:00
Gareth Andrew Lloyd
4bc5d749b2
Refactor replication, part 3 (#1177)
Changes to make replication code agnostic of the storage kind being used.

Co-authored-by: Andreja Tonev <andreja.tonev@memgraph.io>
2023-08-25 10:52:07 +01:00
Ante Pušić
60e167d676
Optimize index and constraint updates (#1159) 2023-08-23 14:52:44 +02:00
andrejtonev
9355e58e73
Decoupling replication logic from InMemoryStorage (#1169) 2023-08-22 13:29:25 +02:00
Gareth Andrew Lloyd
97183fb9da
Fix FLAGS_delta_chain_cache_threshold typo (#1181) 2023-08-21 13:16:02 +02:00
Gareth Andrew Lloyd
2e51e703c3
Add supernode vertex cache (#1124)
Add supernode vertex cache to account for long delta chains and modifications in the same module being independent of scanning of the nodes in the next iteration of the pulling mechanism.
2023-08-11 10:18:28 +02:00
Andi
adf7533751
Optimize import of edges on disk (#1132) 2023-08-10 11:53:07 +02:00
Antonio Filipovic
509183e985
Improve performance on set properties (#1115) 2023-08-10 09:06:44 +02:00
Aidar Samerkhanov
1fe2190747
Filter deleted edges during edge prefetch (#1145) 2023-08-09 13:56:34 +02:00
Andi
762fe6a65d
Improve disk indices (#1139) 2023-08-09 10:16:49 +02:00
gvolfing
260660f1dd
Fix sequential label-property index recovery (#1135)
The parallel_exec_info should have been passed to this function before,
otherwise, the recovery of label-property indices would never have been
parallelized.
2023-08-05 23:20:15 +02:00
Andi
f0bac53e7b
Improve restore replication role (#1089) 2023-08-01 21:51:52 +02:00
andrejtonev
e8850549d2
Add multi-tenancy v1 (#952)
* Decouple BoltSession and communication::bolt::Session
* Add CREATE/USE/DROP DATABASE
* Add SHOW DATABASES
* Cover WebSocket session
* Simple session safety implemented via RWLock
* Storage symlinks for backward. compatibility
* Extend the audit log with the DB info
* Add auth part
* Add tenant recovery
2023-08-01 18:49:11 +02:00
Andi
60f4ffc6a1
Improve logging if replica cannot recover using curr WAL file (#1086) 2023-08-01 10:33:46 +02:00
Andi
be4eb95a98
Fix Jepsen replication pause (#1082) 2023-07-30 02:36:11 +02:00
Andi
18bd02423a
Fix PropertyStore buffer serialization (#1111) 2023-07-29 19:14:27 +02:00
Marko Budiselić
3b9133fd5a
Improve e2e and replication testing setup (#1061)
* Add `--replication-restore-state-on-startup` with `false` as default

Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
Co-authored-by: Andi Skrgat <andi8647@gmail.com>
2023-07-19 21:18:43 +02:00
Marko Budiselić
9d056e7649
Add experimental/v1 of ON_DISK_TRANSACTIONAL storage (#850)
Co-authored-by: Andi Skrgat <andi8647@gmail.com>
Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-06-29 11:44:55 +02:00
Josipmrden
84721f7e0a
Add vertex degree to index statistics (#1026)
Add graph analysis of vertex degrees when doing ANALYZE GRAPH.
2023-06-27 18:06:20 +02:00
Ante Javor
261aa4f49b
Improve replication logging (#1030) 2023-06-27 17:57:51 +02:00
Gareth Andrew Lloyd
5b1ba10183
Fix IN_MEMORY_ANALYTICAL storage GC (#1025) 2023-06-23 12:50:03 +02:00
Ante Javor
0ea96663ba
Add check for opening snapshots (#966) 2023-06-22 13:29:49 +02:00
Josipmrden
b875649270
Add restoring of replication roles upon database startup (#791)
Fix replica node restoration on startup so it is restored as replica and not as main.
2023-06-21 19:08:58 +02:00
Josipmrden
651b6f3a5a Expose system metrics over HTTP Endpoint (#940) 2023-05-18 05:10:57 +00:00
andrejtonev
802f8aceda
Add data directory status and (un)lock query (#933) 2023-05-16 18:36:04 +02:00
gvolfing
c3e4f81026
Include additional info inside storage mode info query (#883) 2023-05-16 14:25:41 +02:00
gvolfing
00f8d54249
Parallelize index creation (#882) 2023-04-26 16:28:02 +02:00
János Benjamin Antal
6c947947eb Parallelize recovery (#868)
* Parallelize edge recovery

* Load vertex labels and properties parallel

* Add parallel connectivity loading

* Add batches information to snapshot

* Introduce `items_per_batch` and `recovery_thread_count` flags

* Make possible to load snapshots with old version

* Add vertex batches to `RecoveryInfo`

* Extend durability integration tests with v15 test cases

* Add `std::vector` based `InitProperties`

* Use `InitProperties` in snapshot loading
2023-04-25 16:25:25 +02:00