Commit Graph

292 Commits

Author SHA1 Message Date
Matej Ferencevic
d9a6775656 Move transaction to stack in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2196
2019-07-11 15:04:10 +02:00
Marin Tomic
8414479abe [StorageV2] Implement GC
Summary:
Here are some numbers from the benchmark:

```
(TOOLCHAIN) mtomic@poso:~/memgraph/build_release$ tests/benchmark/storage_v2_gc --num-threads 8
Config: NoGc, Time: 25.9836
Config: OnFinishGc, Time: 49.012
Config: 100msPeriodicGc, Time: 45.9856
Config: 1000msPeriodicGc, Time: 40.3094
```

```
(TOOLCHAIN) mtomic@poso:~/memgraph/build_release$ tests/benchmark/storage_v2_gc --num-threads 7
Config: NoGc, Time: 20.4256
Config: OnFinishGc, Time: 39.6669
Config: 100msPeriodicGc, Time: 30.7956
Config: 1000msPeriodicGc, Time: 35.128
```

It is not that bad if there is a core dedicated to doing garbage collection.

Reviewers: mferencevic, teon.banek

Reviewed By: mferencevic, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2168
2019-07-10 16:12:08 +02:00
Marin Tomic
a4da2f123c [StorageV2] Fix object deletion in Abort function
Summary:
`lock_guard` is holding vertex lock while we're deleting the vertex,
and then it might try to unlock it in its destructor and access freed memory.

Reviewers: teon.banek, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2192
2019-07-10 10:20:58 +02:00
Matej Ferencevic
18e2f9eeda Make previous pointer atomic in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2191
2019-07-09 16:26:10 +02:00
Matej Ferencevic
4d94aa740a Implement multiple previous pointer in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2190
2019-07-09 14:24:28 +02:00
Matej Ferencevic
b1f7bbf051 Implement properties for edges in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2187
2019-07-08 16:55:47 +02:00
Matej Ferencevic
5b8e7ff432 Implement edges in storage v2
Summary:
This change implements full edges support in storage v2. Edges can be created
and deleted. Support for detach-deleting vertices is added and regular vertex
deletion verifies existance of edges.

Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2180
2019-07-08 16:06:57 +02:00
Matej Ferencevic
069000a92a Fix CommitLog namespace in storage v2
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2179
2019-07-05 13:53:32 +02:00
Matej Ferencevic
26f14443c2 Move vertex deletion to storage accessor
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2174
2019-07-04 14:47:14 +02:00
Matej Ferencevic
6e6dff81e0 Split storage and vertex accessor into cpp in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2172
2019-07-04 12:56:06 +02:00
Matej Ferencevic
0833a74fd8 Refactor storage delta in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2171
2019-07-03 15:22:38 +02:00
Matej Ferencevic
db72ef05f8 Add support for vertex properties to storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2163
2019-07-03 11:07:57 +02:00
Matej Ferencevic
83a99aaf7d Implement vertex deletion in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2157
2019-07-01 16:54:34 +02:00
Matej Ferencevic
9e649794fc Rework delta creation in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2159
2019-07-01 15:47:43 +02:00
Marin Tomic
c7a8be31cb [StorageV2] Fix move constructor for delta
Reviewers: mferencevic, teon.banek

Reviewed By: mferencevic

Differential Revision: https://phabricator.memgraph.io/D2158
2019-07-01 14:43:15 +02:00
Marin Tomic
71c76cfb73 [StorageV2] Make delta list double linked
Summary: this is necessary for garbage collection

Reviewers: teon.banek, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2156
2019-07-01 14:16:20 +02:00
Marin Tomic
a7fabe6109 Fix PrepareForWrite function in storage V2
Summary:
We forgot to update `modified_vertices` in the case when the vertex
has an empty version chain. It didn't manifest before because it was impossible
for a vertex to have an empty version chain without garbage collection.

Reviewers: mferencevic, teon.banek

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2154
2019-07-01 13:02:51 +02:00
Matej Ferencevic
f2b23828ab Implement PropertyValue for storage v2
Reviewers: teon.banek, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2149
2019-07-01 10:56:33 +02:00
Marin Tomic
fabfa77bd9 Fix CreateVertex function in storage v2
Summary:
We forgot to add the newly created vertex into `modified_vertices`
list.

Reviewers: teon.banek, mferencevic

Reviewed By: mferencevic

Differential Revision: https://phabricator.memgraph.io/D2153
2019-06-28 15:43:57 +02:00
Matej Ferencevic
8169513f57 Remove unnecessary constructors from storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2152
2019-06-28 14:43:35 +02:00
Marin Tomic
6ce8fae54a Implement commit log for storage v2
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2148
2019-06-27 13:52:51 +02:00
Matej Ferencevic
54e1e4e1ff Initial version of storage v2
Summary:
Initial implementation of new storage engine.  It implements snapshot isolation
for transactions.  All changes in the database are stored as deltas instead of
making full copies.  Currently, the storage supports full transaction
functionality (commit, abort, command advancement).  Also, support has been
implemented only for vertices that have only labels.

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2138
2019-06-27 10:52:38 +02:00
Matija Santl
87423eed06 Add constraint in HA
Summary:
HA should now support constraints in the same way the SM version does.
I only tested this thing manually, but I plan to add a new integration test for
this also.

Reviewers: ipaljak, vkasljevic, mferencevic

Reviewed By: ipaljak, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2083
2019-05-29 14:55:04 +02:00
Matija Santl
4ec3f62f4e Move constraints to common
Summary:
Preparing unique constraint code to be implemented in HA. To do so, I'm
moving everything to `stograge/common/` folder. I also added a new header,
`gid.hpp` which does a `ifdef` include of the correct `gid.hpp` based on the
product.

Reviewers: ipaljak, mferencevic, vkasljevic, teon.banek

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2079
2019-05-23 16:36:14 +02:00
Matija Santl
f2cc41fa59 Remove unique from label-prop index
Summary: Remove unique feature from label-property index

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: tsabolcec, pullbot

Differential Revision: https://phabricator.memgraph.io/D2045
2019-05-23 12:39:41 +02:00
Vinko Kasljevic
1f93edaa88 Add checks to GlobalAddress in order to prevent misuse
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2074
2019-05-21 15:37:03 +02:00
Matej Ferencevic
3c48e612ef Make move operations noexcept
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1991
2019-05-21 10:46:25 +02:00
Matija Santl
37c68f0508 Add multiple properties unique constraint
Summary:
Unique constraint now support multiple properties

Depends on D2043

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2044
2019-05-20 15:18:05 +02:00
Matija Santl
d70792f1ce Remove existence constraint
Summary: Removing existence constraint

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2043
2019-05-15 09:48:51 +02:00
Matej Ferencevic
5c244c1ad4 Remove Cap'n Proto
Summary:
There will be a lot of leftover files, execute the following commands inside
`src/` to remove them:
```
git clean -xf
rm -r rpc/ storage/single_node_ha/rpc/
```

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2011
2019-05-08 10:51:10 +02:00
Tonko Sabolcec
0849937aea Add less than operator for property value
Summary: This change will allow comparison of sets/maps containing `PropertyValue`s.

Reviewers: teon.banek, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2008
2019-05-07 15:14:22 +02:00
Matej Ferencevic
d678e45c10 Migrate RPC to SLK
Summary:
Migrate all RPCs
Simplify Raft InstallSnapshot RPC
Add missing Load and Save for `char`

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2001
2019-05-06 14:27:57 +02:00
Matej Ferencevic
129c6c0242 Finish SLK implementation
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1999
2019-05-02 15:47:38 +02:00
Matej Ferencevic
9291a5fc4d Migrate to C++17
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1974
2019-04-23 14:46:44 +02:00
Matej Ferencevic
45f2a06a8b Fix various compiler warnings and remove unused test
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1963
2019-04-17 15:48:52 +02:00
Matija Santl
a7ee70e365 Add KVStore::CompactRange to free disk space in HA
Summary:
Manually call `CompactRange` on KVStore (rocksDB) to free disk space
upon Raft log compaction.

Raft log takes surprisingly large amount of disk space (1MB for 1000 entries) so
we need to free disk space as we compact the log into snapshot.

When `log_size_snapshot_threshold` set to `5000` the disk usage of the
`durability_dir` goes from `4.8 MB` to `240 KB`.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1939
2019-04-10 15:29:59 +02:00
Matija Santl
ad0d9e1f87 Fix distributed snapshot recovery failure
Summary:
This is still in progress.

As reported in
https://app.asana.com/0/743890251333732/971034572323525/f
memgraph distributed fails when the snapshot that it tries to recover from is
invalid. Instead of inheritance of `StorageGc` this diff adds composition and we
don't try to register a rpc server after the initialization.

Reviewers: ipaljak, vkasljevic, mtomic

Reviewed By: vkasljevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1930
2019-04-08 14:47:44 +02:00
Vinko Kasljevic
47a1e302a0 UniqueLabelPropertiesConstraint implementation and tests
Summary:
Same as `UniqueLabelPropertyConstratint` except it works with multiple properties.
Because of that it is a bit more complex and slower.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1926
2019-04-08 10:14:47 +02:00
Vinko Kasljevic
2a63e2f12d Constraint code cleanup
Summary: Refactor tests, remove some test cases, refactor some methods in `GraphDbAccessor`.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1933
2019-04-03 08:58:33 +02:00
Vinko Kasljevic
21bd8b492f Add recovery and GraphDbAccessor interface for UniqueLabelPropertyConstraint
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1925
2019-03-27 12:33:56 +01:00
Teon Banek
a14c24e71f Use const_cast in RecordAccessor
Summary:
It seems that older clang versions erroneously accept modification of
constant pointers, this is a quick hack which resolves the issue on new
clang versions.

None of the const methods are really const and should not be marked as
such. This whole accessor thing is just a steaming pile of crap and
should be rewritten from scratch.

Reviewers: vkasljevic, msantl

Reviewed By: vkasljevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1922
2019-03-18 11:31:05 +01:00
Vinko Kasljevic
1c768287e8 Switch from Cache to LruCache
Summary: This diff contains refactor and various fixes needed for LruCache.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1889
2019-03-12 16:11:03 +01:00
Vinko Kasljevic
94afbd9f56 ExistenceConstraints performance improvement
Summary:
Case with existence constraints is about 8-9% slower then case without
existence constraints. Before this diff that difference was about 15-16%.

Reviewers: msantl, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1917
2019-03-12 14:36:27 +01:00
Vinko Kasljevic
8a5842ef63 UniqueLabelPropertyConstraint implementation
Summary:
UniqueLabelPropertyConstraint defines label + property restriction on vertices.
Label + Property + PropertyValue for that property must be unique at any given moment.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1884
2019-03-12 10:02:26 +01:00
Vinko Kasljevic
6cc8d8f5b1 Add wal for ExistenceConstraints
Summary: Depends on D1797

Reviewers: msantl, ipaljak, mferencevic

Reviewed By: msantl, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1862
2019-03-06 08:52:00 +01:00
Vinko Kasljevic
bbb69a1c00 Edges iterator implementation
Summary:
`EdgesIterable` is used for iterating over Edges in distributed memgraph. Because of lru cache there is a possibility of data getting evicted as someone iterates over it.
To prevent that `EdgesIterable` will lock that data and release it when it's deconstructed.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1868
2019-02-26 09:44:22 +01:00
Vinko Kasljevic
3ad5c8069c Add CachedDataLock throughout the code
Summary:
CachedDataLock is necessary for lru cache as remote data is no longer
persistent. Most methods internally handle this, but for methods that
return pointers or references to remote data, we need to manually
lock data.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1869
2019-02-21 15:27:40 +01:00
Vinko Kasljevic
a14c4f1864 Existence constraints for vertex for single node
Summary:
Existence constraint ensures that all nodes with certain label have a certain property.
`ExistenceRule` defines label -> properties rule and `ExistenceConstraints` manages all
constraints.

Reviewers: msantl, ipaljak, teon.banek, mferencevic

Reviewed By: msantl, teon.banek, mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1797
2019-02-15 13:14:31 +01:00
Teon Banek
5af47c6df5 Don't inherit TotalOrdering for RecordAccessor
Summary:
RecordAccessor doesn't implement `operator<`, so it doesn't make sense
to have it inherit TotalOrdering.

Reviewers: mferencevic, msantl, vkasljevic

Reviewed By: vkasljevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1866
2019-02-15 11:10:31 +01:00
Vinko Kasljevic
d1eeaa8de0 Remove Impl from RecordAccessor
Reviewers: msantl, ipaljak, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1859
2019-02-14 17:15:45 +01:00