Summary:
Start removing `is_remote` from `Address`
Remove `GlobalAddress`
Remove `GlobalizedAddress`
Remove bitmasks from `Address`
Remove `is_local` from `Address`
Remove `is_local` from `RecordAccessor`
Remove `worker_id` from `Address`
Remove `worker_id` from `GidGenerator`
Unfriend `IndexRpcServer` from `Storage`
Remove `LocalizedAddressIfPossible`
Make member private
Remove `worker_id` from `Storage`
Copy function to ease removal of distributed logic
Remove `worker_id` from `WriteAheadLog`
Remove `worker_id` from `GraphDb`
Remove `worker_id` from durability
Remove nonexistant function
Remove `gid` from `Address`
Remove usage of `Address`
Remove `Address`
Remove `VertexAddress` and `EdgeAddress`
Fix Id test
Remove `cypher_id` from `VersionList`
Remove `cypher_id` from durability
Remove `cypher_id` member from `VersionList`
Remove `cypher_id` from database
Fix recovery (revert D1142)
Remove unnecessary functions from `GraphDbAccessor`
Revert `InsertEdge` implementation to the way it was in/before D1142
Remove leftover `VertexAddress` from `Edge`
Remove `PostCreateIndex` and `PopulateIndexFromBuildIndex`
Split durability paths into single node and distributed
Fix `TransactionIdFromWalFilename` implementation
Fix tests
Remove `cypher_id` from `snapshooter` and `durability` test
Reviewers: msantl, teon.banek
Reviewed By: msantl
Subscribers: msantl, pullbot
Differential Revision: https://phabricator.memgraph.io/D1647
Summary:
This diff splits single node and distributed storage from each other.
Currently all of the storage code is copied into two directories (one single
node, one distributed). The logic used in the storage implementation isn't
touched, it will be refactored in following diffs.
To clean the working directory after this diff you should execute:
```
rm database/state_delta.capnp
rm database/state_delta.hpp
rm storage/concurrent_id_mapper_rpc_messages.capnp
rm storage/concurrent_id_mapper_rpc_messages.hpp
```
Reviewers: teon.banek, buda, msantl
Reviewed By: teon.banek, msantl
Subscribers: teon.banek, pullbot
Differential Revision: https://phabricator.memgraph.io/D1625
Summary:
This change introduces a pure virtual initial implementation of the transaction
engine which is then implemented in two versions: single node and distributed.
The interface classes now have the following hierarchy:
```
Engine (pure interface)
|
+----+---------- EngineDistributed (common logic)
| |
EngineSingleNode +-------+--------+
| |
EngineMaster EngineWorker
```
In addition to this layout the `EngineMaster` uses `EngineSingleNode` as its
underlying storage engine and only changes the necessary functions to make
them work with the `EngineWorker`.
After this change I recommend that you delete the following leftover files:
```
rm src/distributed/transactional_cache_cleaner_rpc_messages.*
rm src/transactions/common.*
rm src/transactions/engine_rpc_messages.*
```
Reviewers: teon.banek, msantl, buda
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1589
Summary:
No logic changes, just split `tx::MasterEngine` into
`tx::SingleNodeEngine` and `tx::MasterEngine`. This gives better
responsibility separation and is more appropriate now there is no
Start/Shutdown.
Reviewers: dgleich, teon.banek, buda
Reviewed By: dgleich, teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1099
Summary:
The current idea is that the same MG binary can be used for single-node,
distributed master and distributed worker. The transactional engine in
the single-node and distributed master is the same: it determines the
transactional time and exposes all the "global" functionalities. In the
distributed worker the "global" functions must contact the master.
Reviewers: dgleich, mislav.bradac, buda
Reviewed By: dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1013
Summary:
My dear fellow Memgraphians. It's friday afternoon, and I am as ready to pop as WAL is to get reviewed...
What's done:
- Vertices and Edges have global IDs, stored in `VersionList`. Main storage is now a concurrent map ID->vlist_ptr.
- WriteAheadLog class added. It's based around buffering WAL::Op objects (elementraly DB changes) and periodically serializing and flusing them to disk.
- Snapshot recovery refactored, WAL recovery added. Snapshot format changed again to include necessary info.
- Durability testing completely reworked.
What's not done (and should be when we decide how):
- Old WAL file purging.
- Config refactor (naming and organization). Will do when we discuss what we want.
- Changelog and new feature documentation (both depending on the point above).
- Better error handling and recovery feedback. Currently it's all returning bools, which is not fine-grained enough (neither for errors nor partial successes, also EOF is reported as a failure at the moment).
- Moving the implementation of WAL stuff to .cpp where possible.
- Not sure if there are transactions being created outside of `GraphDbAccessor` and it's `BuildIndex`. Need to look into.
- True write-ahead logic (flag controlled): not committing a DB transaction if the WAL has not flushed it's data. We can discuss the gain/effort ratio for this feature.
Reviewers: buda, mislav.bradac, teon.banek, dgleich
Reviewed By: dgleich
Subscribers: mtomic, pullbot
Differential Revision: https://phabricator.memgraph.io/D958
Summary:
- GC changed to evaluate old records w.r.t. the oldest transaction's id AND snapshot, as opposed to only id
- MVCC hints exp+aborted race condition prevented
- minor MVCC refactors and cleanups
- minor Transaction refactors and cleanups
Reviewers: buda, dgleich
Reviewed By: buda, dgleich
Subscribers: dtomicevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D434
Summary:
When running tck tests there was a peculiar behavior where sometimes some queries worked, and sometimes they failed. Nothing was failing when memgraph was restarted between each query (scenario) - which points to MATCH DETACH DELETE not working correctly.
What was happening is the following: some transaction would update the record in version list and set it's expiry to it's id. Along with that some transaction would query the mentioned record - and would set the hints flags for that expiration transaction status (which was aborted - which is fine at this moment). After some while, because the record is not really deleted because it's not aborted some other transaction would modify it's expiry transaction (this time making the transaction commited), but because the hints flags were not updated - they would still return the status for the old transaction - which was aborted. This made some records available even though they were deleted.
Reviewers: mislav.bradac, florijan, teon.banek, matej.gradicek, buda
Reviewed By: buda
Subscribers: buda, lion, pullbot
Differential Revision: https://phabricator.memgraph.io/D370
Summary:
Gradicek's Mvcc test have seen the following changes:
- provided a test infrastructure (fixture and macros) to facilitate testing and increase readability
- split tests into multi-transaction update, VersionList::find and general Mvcc testing
- multi-transaction update tests have been refactored (i *think* nothing got deleted, but it was a mess so I don't guarantee)
- changed all multithreaded tests to be single-threaded because multiple threads were not necessary
- changed transaction naming from T5, T8, T10 to T1, T2... for consistency with actual transaction indices
What still needs to be done:
- Gleich and Gradicek need to review the infrastructure (possible improvements)
- multi-transaction update tests need to be addressed by Gradicek (see "TODO gradicek" in code, discuss with Flor)
- the wiki/draw.io documentation needs to be updated. it is not imperative that all the tests be drawn in draw.io, only the general infrastructure explained. perhaps only a few examples drawn. Gradicek discuss with Flor
- Gleich see the "TODO Gleich" lines in the diff and discuss with flor
Suggested workflow:
- review this diff, hopefully land (before resolving all the TODOs)
- discard D169
- Gradicek and Gleich address the TODOs
- Flor reviews the results (in following diffs)
Reviewers: dgleich, matej.gradicek, buda
Reviewed By: matej.gradicek, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D348
Summary: Changing mvcc again. This should be double-checked since it's quite a substantial change. The test here tests for the new behaviour - which should be correct.
Reviewers: matej.gradicek, dtomicevic, buda
Reviewed By: matej.gradicek, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D156
