Commit Graph

1461 Commits

Author SHA1 Message Date
Matej Ferencevic
e5a4a84de2 Fix MonotonicBufferResource buffer growth
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2483
2019-10-15 15:04:35 +02:00
Lovro Lugovic
5c5c926515 An aggregate element's input expression can be NULL
Summary: For example, the aggregate element produced for `COUNT(*)` has its `value` set to `NULL`.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2463
2019-10-03 10:07:24 +02:00
Matej Ferencevic
3adedc0679 Implement snapshots for storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2400
2019-10-01 13:42:37 +02:00
Matej Ferencevic
3756534490 Don't crash the database when using utils::InputFile
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2460
2019-10-01 10:15:42 +02:00
Matej Ferencevic
9ad49698e9 Add support for disabling properties on edges
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2447
2019-09-25 10:17:19 +02:00
Matej Ferencevic
8b6ae08682 Create/drop label indices explicitly in storage v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2432
2019-09-24 13:12:50 +02:00
Matej Ferencevic
3079522f47 Make storage v2 config hierarchical
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2434
2019-09-23 16:03:53 +02:00
Matej Ferencevic
9d39ecab7b Implement encoder/decoder for durability v2
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2393
2019-09-18 16:32:32 +02:00
Matej Ferencevic
9652f8e265 Add clear method to SkipList
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2385
2019-09-13 15:24:14 +02:00
Matej Ferencevic
55d8b98aeb Prepare storage v2 config for more options
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2382
2019-09-13 12:42:23 +02:00
Matej Ferencevic
51fc4df87a Add utils::InputFile wrapper
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2377
2019-09-12 16:30:11 +02:00
Teon Banek
7bd45f8714 Make query execution work with storage_v2
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2203
2019-09-12 10:22:00 +02:00
Matej Ferencevic
68f19df305 Migrate LDAP integration to auth module
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2368
2019-09-11 17:15:08 +02:00
Matej Ferencevic
73379bc57e Add functions for listing indices and constraints
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2371
2019-09-11 15:45:14 +02:00
Ivan Paljak
deee3b8ab7 Remove Raft's dependency on transaction id
Reviewers: buda, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2338
2019-09-10 18:02:05 +02:00
Teon Banek
1000a5e544 Use BasicResult instead of optional for v2 Commit
Summary: Depends on D2365

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2366
2019-09-10 13:32:54 +02:00
Teon Banek
f02617f2bd Replace query::GraphView with storage::View
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2356
2019-09-06 14:58:27 +02:00
Teon Banek
2e51ef9f66 Use storage/v2/view in glue/communication
Summary:
Switching to Storage V2 API will require passing storage::View when
serializing VertexAccessor and EdgeAccessor, so this is just the first
step in adapting the code.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2352
2019-09-05 10:04:44 +02:00
Teon Banek
7213bec886 Make Gid a full on class wrapper around uint64_t
Summary:
This makes Gid the same as the one in storage/v2. Before they can be
merge into one implementation, we probably want to have a similar
transition for remaining ID types.

Depends on D2346

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2347
2019-09-03 15:14:21 +02:00
Teon Banek
cdfaf0d4a6 Move gid::Gid to storage/common/types
Summary:
It never made sense that a global ID is its own namespace in the storage
directory tree.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2346
2019-09-03 14:57:44 +02:00
Ivan Paljak
58be850d5c Fix HA log compaction test
Summary: The test now checks if the cluster is alive at the end of the test.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2343
2019-09-03 08:45:53 +02:00
Matej Ferencevic
60367a287e Implement wrappers for STL containers with our allocator
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2342
2019-09-02 17:04:15 +02:00
Teon Banek
c8340ad120 Merge old and new PropertyValue implementations
Summary:
This effectively replaces the old PropertyValue implementation from the
one in storage/v2

Depends on D2333

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2335
2019-09-02 16:43:50 +02:00
Teon Banek
32ae2b4d23 Correctly release PoolResource memory
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2341
2019-09-02 15:49:57 +02:00
Teon Banek
ed802073f6 Make PropertyValue constructors explicit
Summary: Depends on D2332

Reviewers: ipaljak, mferencevic

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2333
2019-09-02 12:31:53 +02:00
Matej Ferencevic
d3b141f3ba Allocate initial execution memory
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2316
2019-09-02 11:40:38 +02:00
Teon Banek
9f4a7dcddf Remove PropertyValue::Null
Reviewers: ipaljak, mferencevic

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2332
2019-08-29 09:02:51 +02:00
Teon Banek
023538c19c Add PoolResource benchmarking in execution and skiplist
Summary:
With a pool allocator, lookups in STL set and map are up to 50% faster.
This is probably due to contiguous memory of pooled objects, i.e. nodes
of those containers. In some cases, the lookup outperforms the SkipList.
Insertions are also faster, though not as dramatically, up to 30%. This
does make a significant difference when the STL containers are used in a
single thread as they outperform the SkipList significantly.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2326
2019-08-27 09:32:01 +02:00
Teon Banek
1fa9d7752c Fix a bug in Pool::Deallocate
Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2325
2019-08-27 09:32:00 +02:00
Marin Tomic
7e741b8d25 Implement vertex count estimation
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2306
2019-08-22 17:34:00 +02:00
Teon Banek
97b20a9384 Use non-template Value invocation
Reviewers: mtomic, mferencevic

Reviewed By: mtomic, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2317
2019-08-22 16:20:31 +02:00
Marin Tomic
21d461e77d Skip deleted vertices when building index
Summary:
There's a possible race condition where we add a deleted vertex into
index and garbage collection removes it from main storage before indices are
cleaned-up.

Reviewers: mferencevic, teon.banek

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2314
2019-08-22 10:25:44 +02:00
Teon Banek
4f93632fb8 storage/v2: Add (In|Out)Degree to VertexAccessor
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2309
2019-08-21 12:58:00 +02:00
Marin Tomic
1ab0b8e0ff Implement existence constraints
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2307
2019-08-20 16:27:36 +02:00
Teon Banek
ce9e2614fa Invert the result of SetProperty and add documentation
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2297
2019-08-13 16:21:33 +02:00
Matija Santl
b684e697fd Adjust raft constants for long running tests
Summary:
Based on the recent builds (https://apollo.memgraph.io/runs/642045/,
https://apollo.memgraph.io/runs/642700/ https://apollo.memgraph.io/runs/640896/
https://apollo.memgraph.io/runs/640538/) `ha` stress tests fail.

Every tests that failed, has a re-election in progress. I'm bumping election
timeout constatns in order to give the cluster more time to do what it needs to
do before going in the state where it fails.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2292
2019-08-13 10:40:24 +02:00
Teon Banek
f7575eb112 Add utilities for type checking awesome MG functions
Reviewers: mtomic, msantl

Reviewed By: mtomic, msantl

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D2259
2019-08-12 16:24:24 +02:00
Matej Ferencevic
fb6dc83ef4 Replace bswap with proper endian functions
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2290
2019-08-07 15:45:03 +02:00
Marin Tomic
35587cddbd [StorageV2] Implement label and label-property indices
Reviewers: mferencevic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2225
2019-07-26 12:23:18 +02:00
Teon Banek
867f30d9b3 Add VerticesIterable to storage/v2
Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2229
2019-07-23 13:04:16 +02:00
Marin Tomic
5a3e98badd Implement Synchronized<T> utility
Reviewers: mferencevic, teon.banek

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2222
2019-07-22 14:55:21 +02:00
Matej Ferencevic
d7d0963434 Implement explicit types in storage v2
Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2224
2019-07-22 14:16:02 +02:00
Matej Ferencevic
e9a7623288 Implement name to id mapper in storage v2
Reviewers: teon.banek, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2217
2019-07-19 11:24:27 +02:00
Ivan Paljak
fa4c303af6 Add HA normal operation long running test to daily build
Summary: Currently set to run for 2 hours.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2206
2019-07-18 16:53:07 +02:00
Matej Ferencevic
111dd8bf19 Remove distributed
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2213
2019-07-17 15:23:42 +02:00
Teon Banek
59b3f84eb9 Set [[nodiscard]] attribute on storage::Result
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2214
2019-07-17 15:11:29 +02:00
Matej Ferencevic
4e1c7c9e34 Rename Return to Value in storage v2 Result
Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2199
2019-07-17 12:33:39 +02:00
Matej Ferencevic
b365037e23 Cleanup locks
Summary: Move RWLock and replace exceptions with `CHECK`s

Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2205
2019-07-17 10:33:44 +02:00
Ivan Paljak
4acad5795b Expose the status of transaction within Raft
Summary:
For proper client interaction, we need to expose the (term_id, log_index)
pair for the transaction that's about to be replicated and we need to be able
to retrieve the status of a transaction defined by that pair. Transaction
status can be one of the following:

  1) REPLICATED (self-explanatory)
  2) WAITING (waiting for replication)
  3) ABORTED (self-explanatory)
  4) INVALID (received request with either invalid term_id or invalid log_index)

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2201
2019-07-16 13:53:16 +02:00
Matej Ferencevic
d5106c7233 Handle zero increment in counters
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2208
2019-07-16 12:35:35 +02:00
Marin Tomic
e7f897984f Fix wrong settings in gc bench
Reviewers: mferencevic, teon.banek

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2207
2019-07-15 17:53:03 +02:00
Teon Banek
bf4f69f2e5 storage/v2: Reduce the (In|Out)Edges tuple to only EdgeAccessor
Summary:
The first 2 tuple elements are redundant as they are available through
EdgeAccessor and they needlessly complicate the usage of the API.

Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2200
2019-07-12 15:30:24 +02:00
Teon Banek
c4c6febbc4 Move Parsing utils from query/common to frontend/parsing
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2195
2019-07-11 12:12:59 +02:00
Marin Tomic
8414479abe [StorageV2] Implement GC
Summary:
Here are some numbers from the benchmark:

```
(TOOLCHAIN) mtomic@poso:~/memgraph/build_release$ tests/benchmark/storage_v2_gc --num-threads 8
Config: NoGc, Time: 25.9836
Config: OnFinishGc, Time: 49.012
Config: 100msPeriodicGc, Time: 45.9856
Config: 1000msPeriodicGc, Time: 40.3094
```

```
(TOOLCHAIN) mtomic@poso:~/memgraph/build_release$ tests/benchmark/storage_v2_gc --num-threads 7
Config: NoGc, Time: 20.4256
Config: OnFinishGc, Time: 39.6669
Config: 100msPeriodicGc, Time: 30.7956
Config: 1000msPeriodicGc, Time: 35.128
```

It is not that bad if there is a core dedicated to doing garbage collection.

Reviewers: mferencevic, teon.banek

Reviewed By: mferencevic, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2168
2019-07-10 16:12:08 +02:00
Teon Banek
5bc0be50c5 Remove GraphDbAccessor parameter from MakeCursor
Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2189
2019-07-09 14:22:55 +02:00
Teon Banek
3a34d6fcdd Support all power of 2 alignments in MonotonicBufferResource
Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2188
2019-07-09 14:22:05 +02:00
Ivan Paljak
2936312612 Add memgraph ha normal operation long running test
Summary:
The functionality of the test is the same as for single node Memgraph.
Locally, it seems to work fine. I'll update the apollo related files when I feel
a bit more certain that everything works locally.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2135
2019-07-09 13:59:56 +02:00
Teon Banek
76d8020169 Implement a PoolResource for allocations
Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2147
2019-07-09 09:31:16 +02:00
Teon Banek
a417ef36f1 Use MemoryResource in SkipList
Summary:
This is a preparation step in case we want to have a custom allocator in
SkipList. For example, pool based allocator for SkipListNode.
Introduction of MemoryResource and removal of `calloc` has reduced the
performance a bit according to micro benchmarks. This performance hit is
not visible on benchmarks which do more concurrent operations.

Reviewers: mferencevic, mtomic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2140
2019-07-09 09:31:15 +02:00
Matej Ferencevic
b1f7bbf051 Implement properties for edges in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2187
2019-07-08 16:55:47 +02:00
Matej Ferencevic
5b8e7ff432 Implement edges in storage v2
Summary:
This change implements full edges support in storage v2. Edges can be created
and deleted. Support for detach-deleting vertices is added and regular vertex
deletion verifies existance of edges.

Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2180
2019-07-08 16:06:57 +02:00
Matej Ferencevic
a2c5b5eac6 Temporarily disable LDBC test
Reviewers: mtomic, buda

Reviewed By: buda

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D2178
2019-07-05 14:04:38 +02:00
Matej Ferencevic
069000a92a Fix CommitLog namespace in storage v2
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2179
2019-07-05 13:53:32 +02:00
Matej Ferencevic
26f14443c2 Move vertex deletion to storage accessor
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2174
2019-07-04 14:47:14 +02:00
Matej Ferencevic
6e6dff81e0 Split storage and vertex accessor into cpp in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2172
2019-07-04 12:56:06 +02:00
Matej Ferencevic
9e223cdabd Use only local workers for builds on Apollo
Reviewers: mtomic, buda

Reviewed By: mtomic, buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2170
2019-07-03 22:30:55 +02:00
Matej Ferencevic
db72ef05f8 Add support for vertex properties to storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2163
2019-07-03 11:07:57 +02:00
Matej Ferencevic
83a99aaf7d Implement vertex deletion in storage v2
Reviewers: mtomic, teon.banek

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2157
2019-07-01 16:54:34 +02:00
Matej Ferencevic
f2b23828ab Implement PropertyValue for storage v2
Reviewers: teon.banek, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2149
2019-07-01 10:56:33 +02:00
Marin Tomic
6ce8fae54a Implement commit log for storage v2
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2148
2019-06-27 13:52:51 +02:00
Matej Ferencevic
54e1e4e1ff Initial version of storage v2
Summary:
Initial implementation of new storage engine.  It implements snapshot isolation
for transactions.  All changes in the database are stored as deltas instead of
making full copies.  Currently, the storage supports full transaction
functionality (commit, abort, command advancement).  Also, support has been
implemented only for vertices that have only labels.

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2138
2019-06-27 10:52:38 +02:00
Teon Banek
775930ef4e Use stack for initial memory of MonotonicBufferResource in Pull
Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2139
2019-06-26 10:26:53 +02:00
Teon Banek
5adb4aafe0 Use execution memory for Frame allocations
Summary:
Micro benchmarks show some minor variations compared to the previous
commit. Smaller cases are a bit worse while larger data cases are a bit
better.

Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2136
2019-06-24 15:05:44 +02:00
Teon Banek
6086257204 Use per_pull and whole execution allocators in Cursors
Reviewers: mtomic, mferencevic, msantl

Reviewed By: mtomic, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2134
2019-06-24 15:05:43 +02:00
Tonko Sabolcec
1253319710 Bugfix: use indices when dumping edges
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2142
2019-06-21 16:06:23 +02:00
Teon Banek
3fd14e2d5f Use custom allocator in Evaluator through context
Summary:
Micro benchmarks show improvements in performance of MapLiteral from 5%
to 40% depending on the size of the input. On the other hand, a sequence
of AdditionOperators behaves the same with both allocation schemes.

Reviewers: mtomic, mferencevic, msantl

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2132
2019-06-17 10:53:25 +02:00
Teon Banek
57d967786c Make TypedValue constructor explicit for non-primitive types
Reviewers: mtomic, mferencevic, msantl

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2128
2019-06-12 14:55:58 +02:00
Teon Banek
b3bc4d6809 Make TypedValue constructor explicit for primitive types
Reviewers: mtomic, mferencevic, msantl

Reviewed By: mtomic, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2127
2019-06-12 14:55:57 +02:00
Teon Banek
6d3e3ac4aa Remove TypedValue::Null
Summary:
The global variable may hide the fact that it uses the default
utils::NewDeleteResource() for allocations.

Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2121
2019-06-12 09:55:18 +02:00
Teon Banek
df90184b50 Use utils::Allocator in query::Path
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mtomic, llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2117
2019-06-12 09:55:18 +02:00
Teon Banek
96505b7fb4 Use Allocator for map in TypedValue
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2116
2019-06-12 09:55:17 +02:00
Lovro Lugovic
dbd226d05d LCP: Minor improvements
Summary:
- Add a unified `generate_lcp` target
- Simplify SLK-ERROR and CLONE-ERROR
- Rework WITH-VARS
- Expect CPP-CLASS, not CPP-TYPE
- Fix CPP-TYPE-REFERENCE-P
- Add CPP-GENSYM
- Add util.lisp to LCP's source files in CMake
- Make the CMake variable `lcp_src_files` a cache (persistent) variable
- Add `lcp_src_files` as a dependency to `test_lcp`
- Rename `lcp_compile` to `compile-lcp`
- Fetch docstrings for CPP-NAME-* functions at run-time
- Remove existing C++ entities on redefinition

Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2094
2019-06-12 09:38:02 +02:00
Matej Ferencevic
18698ac7e6 Disable distributed test
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2126
2019-06-11 12:17:12 +02:00
Tonko Sabolcec
485592eb48 Add support for dumping unqiue constraints
Summary: Dumps unique constraints, tests included

Reviewers: teon.banek, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2112
2019-06-10 16:50:46 +02:00
Tonko Sabolcec
9cc2224ac3 Handle dump database query
Summary:
Stream queries to the output table.
For effective output streaming, a simple operator, `OutputTableStream` is implemented which fetches and
produces a single row on each Pull.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2099
2019-06-10 13:22:14 +02:00
Teon Banek
fa62ea1920 Use ValueMap instead of Value<>
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2114
2019-06-10 10:35:04 +02:00
Teon Banek
6bb72d14a1 Use Allocator for string in TypedValue
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mtomic, llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2104
2019-06-10 10:35:04 +02:00
Teon Banek
16555eeb9a Add special construct for std::pair in Allocator
Summary:
This is unfortunately needed in the C++17 standard, so that the
allocator is correctly propagated to elements of pair which respect the
"Uses Allocator" protocol. C++20 standard resolves this issue, but we
still have a long way before it is released and implemented by the
compiler and standard library vendors.

Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2107
2019-06-06 13:44:56 +02:00
Ivan Paljak
716a80edf0 Include HA client in index integration test
Summary:
The test now uses `ha_client`. Logging is also modified to output 1-indexed
worker ids.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2113
2019-06-05 16:07:30 +02:00
Ivan Paljak
9d932797ab Include HA client in HA basic integration test
Summary:
- Included HA client
- Fixed log messages to be 1-indexed
- Added id properties to created nodes for easier debugging
- Create and check steps are now executed 20 times each

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2111
2019-06-04 16:12:51 +02:00
Ivan Paljak
45413c0612 Improve large log entries integration test
Summary:
- Server ids are now 1-indexed in logs
- All created nodes have distinct is properties which helps with debugging

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2109
2019-06-04 15:21:05 +02:00
Ivan Paljak
c775688aa7 Add HA integration test for leader election
Summary:
This tests checks the correctness of a leader election process when its
decoupled from log replication. In other words, in this test we do not change
the state of the database, i. e., the Raft log remains empty.

The test proceeds as follows for clusters of size 3 and 5:
  1. Start a random subset of workers in the cluster
  2. Check if the leader has been elected
  3. Kill all living workers
  4. GOTO 1 and repeat 10 times

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2105
2019-06-04 12:48:35 +02:00
Teon Banek
07188fff7f Add utils::Substr which can use different allocator
Reviewers: mtomic, msantl, mferencevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2103
2019-06-03 17:07:50 +02:00
Teon Banek
38c6625e2c utils: Support std::basic_string with allocators
Reviewers: mtomic, mferencevic, msantl

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2102
2019-06-03 17:07:49 +02:00
Teon Banek
c0dc37fe34 Use ValueString instead of Value<>
Reviewers: mtomic, llugovic, msantl, mferencevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2101
2019-06-03 17:07:49 +02:00
Teon Banek
bb2dbc290f Use Allocator for vector in TypedValue
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2086
2019-06-03 17:07:49 +02:00
Teon Banek
0152ac2dfe Add MemoryResource to TypedValue
Summary:
This diff only introduces a MemoryResource member to TypedValue and correctly
propagates through various constructors and assignments. At the moment,
MemoryResource is not used to actually allocate anything.

Reviewers: mtomic, llugovic, mferencevic, msantl

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2085
2019-06-03 17:07:48 +02:00
Tonko Sabolcec
1bcb3cbd86 Add grammar for dump query
Summary:
Defined Antlr4 grammar for `DUMP DATABASE` query.
Defined DUMP permission in auth module.

Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2077
2019-06-03 10:44:18 +02:00
Ivan Paljak
24ae8a759b Update constants in large log entries test to prevent test timeouts
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2100
2019-05-29 15:40:46 +02:00
Matija Santl
87423eed06 Add constraint in HA
Summary:
HA should now support constraints in the same way the SM version does.
I only tested this thing manually, but I plan to add a new integration test for
this also.

Reviewers: ipaljak, vkasljevic, mferencevic

Reviewed By: ipaljak, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2083
2019-05-29 14:55:04 +02:00
Ivan Paljak
d7acf75a78 Add an integration test which replicates large log entries
Summary:
At the moment, this test will fail. There are currently two issues with it:

  - Cap'n Proto limit exception might occur
  - `db.Reset()` method can hang (deadlock)

The first issue should be resolved by migration to SLK and we are currently
working on the second one. The test will land after those are fixed.

Reviewers: msantl, mferencevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2006
2019-05-29 11:00:55 +02:00
Teon Banek
f6264ab2ae Use ValueList instead of Value<>
Summary:
This change will make the transition to allocator backed TypedValue
smoother.

Reviewers: mtomic, llugovic, mferencevic, msantl

Reviewed By: mtomic, llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2084
2019-05-28 09:41:31 +02:00
Teon Banek
18cf877a47 Respect std::uses_allocator trait in utils::Allocator
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2081
2019-05-28 09:41:31 +02:00
Matija Santl
e4fd49a530 Remove unique index from query execution
Summary: Remove leftover unique index code from query execution.

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2078
2019-05-28 08:16:35 +02:00
Tonko Sabolcec
b57e50865e Bugfix in database dump tests
Summary:
Edge creation in tests used to happen in different
transaction than vertex creation which caused unexpected
behaviour. In this change, both vertices and edges are
created in the same transaction.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2089
2019-05-27 15:43:35 +02:00
dlozic
81a7a7b600 Add Raft term updates integration test
Reviewers: ipaljak, msantl, mferencevic

Reviewed By: ipaljak, msantl, mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D2002
2019-05-27 10:33:40 +02:00
Matija Santl
4ec3f62f4e Move constraints to common
Summary:
Preparing unique constraint code to be implemented in HA. To do so, I'm
moving everything to `stograge/common/` folder. I also added a new header,
`gid.hpp` which does a `ifdef` include of the correct `gid.hpp` based on the
product.

Reviewers: ipaljak, mferencevic, vkasljevic, teon.banek

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2079
2019-05-23 16:36:14 +02:00
Matija Santl
f2cc41fa59 Remove unique from label-prop index
Summary: Remove unique feature from label-property index

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: tsabolcec, pullbot

Differential Revision: https://phabricator.memgraph.io/D2045
2019-05-23 12:39:41 +02:00
Matija Santl
5d5dfbb6f7 Fix how HA handles leader change during commit
Summary:
During it's leadership, one peer can receive RPC messages from other peers that his reign is over.
The problem is when this happens during a transaction commit.

This is handled in the following way.
If we're the current leader and we want to commit a transaction, we need to make sure the Raft Log is replicated before we can tell the client that the transaction is committed.
During that wait, we can only notice that the replication takes too long, and we report that with `LOG(WARNING)` messages.

If we change the Raft mode during the wait, our Raft implementation will internally commit this transaction, but won't be able to acquire the Raft lock because the `db.Reset` has been called.
This is why there is an manual lock acquire. If we pick up that the `db.Reset` has been called, we throw an `UnexpectedLeaderChangeException` exception to the client.

Another thing with long running transactions, if someone decides to kill a `memgraph_ha` instance during the commit, the transaction will have `abort` hint set. This will cause the `src/query/operator.cpp` to throw a `HintedAbortError`. We need to catch this during the shutdown, because the `memgraph_ha` isn't dead from the user perspective, and the transaction wasn't aborted because it took too long, but we can differentiate between those two.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic, ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1956
2019-05-20 16:39:44 +02:00
Matija Santl
37c68f0508 Add multiple properties unique constraint
Summary:
Unique constraint now support multiple properties

Depends on D2043

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2044
2019-05-20 15:18:05 +02:00
Matej Ferencevic
6c49e6de02 Add SSL support to HA RPC
Reviewers: msantl, teon.banek

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2055
2019-05-20 12:53:52 +02:00
Teon Banek
4f4837392e Use MemoryResource in UnwindCursor
Summary:
Micro benchmarks show no change compared to global new & delete. This is
to be expected, because Unwind relies only on `std::vector` which ought
to reserve the memory in reasonable chunks.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2064
2019-05-17 15:28:54 +02:00
Teon Banek
8459fd90fc Use MemoryResource in OrderByCursor
Summary:
Micro benchmarks show an improvement to performance of about 10%
compared to global new & delete.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2061
2019-05-17 15:28:53 +02:00
Teon Banek
0075eee58b Use MemoryResource in AggregationCursor
Summary: Micro benchmarks show some improvement, but unfortunately not much.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2058
2019-05-17 15:28:53 +02:00
Matej Ferencevic
d4b2d76a35 Reimplement counter openCypher function
Reviewers: teon.banek, msantl

Reviewed By: teon.banek, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2047
2019-05-16 11:09:02 +02:00
Teon Banek
1cc71c6ce8 Use MemoryResource in AccumulateCursor
Summary:
Micro benchmarks show that MonotonicBufferResource improves performance
by a factor of 1.5.

Reviewers: mtomic, mferencevic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2048
2019-05-15 16:13:35 +02:00
Teon Banek
78830b6ed8 Use MemoryResource in STShortestPathCursor
Summary:
Benchmarks show minor improvements. Perhaps it makes sense at some later date
to use another allocator for things lasting only in a single `Pull`.

Reviewers: mferencevic, mtomic, llugovic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2018
2019-05-15 16:13:23 +02:00
Teon Banek
801bdb3a91 Use MemoryResource for members of ExpandVariableCursor
Summary:
Unfortunately, the written micro benchmark only reports minor
improvements compared to default allocator. The results are in some
cases even a tiny bit worse.

Reviewers: mtomic, mferencevic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2039
2019-05-15 16:07:27 +02:00
Tonko Sabolcec
b09b21b832 Dump indices in CypherDumpGenerator
Summary:
This change introduces dumping of indices keys. During the dump process,
an internal label is assigned to each vertex and index on vertex's
internal property id is created for faster matching during edge creation.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: msantl, pullbot

Differential Revision: https://phabricator.memgraph.io/D2046
2019-05-15 11:05:16 +02:00
Matija Santl
d70792f1ce Remove existence constraint
Summary: Removing existence constraint

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2043
2019-05-15 09:48:51 +02:00
Matej Ferencevic
d37460105a Implement HA Bolt proxy server
Reviewers: msantl, ipaljak, teon.banek

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2030
2019-05-14 17:20:21 +02:00
Tonko Sabolcec
5f24342502 Split a single dump query into multiple queries
Summary:
Prior to this change, a huge query was returned by DumpGenerator that
dumped the entire graph. This change split the single query to multiple
queries, each dumping a single vertex/edge. For easier vertex matching
when dumping edge, an internal property id is assigned to each vertex and
removed after the whole graph is dumped.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2038
2019-05-14 14:22:28 +02:00
Teon Banek
7a586b3686 Allocate Cursor through utils::MemoryResource
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2026
2019-05-13 11:36:40 +02:00
Teon Banek
2909ef63d2 Use utils::MonotonicBufferResource in query execution
Reviewers: mferencevic, mtomic, msantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2016
2019-05-13 11:36:19 +02:00
Lovro Lugovic
390b6c1557 LCP: Small test fixes
Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1947
2019-05-10 16:10:17 +02:00
Teon Banek
394039a05e Use and bench custom allocator in Distinct
Summary:
According to the written benchmark, using MonotonicBufferResource yields
significant improvements to performance of Distinct. The setup fills the
database with vertices depending on the benchmark state. No edges are
created. Then we run DISTINCT on that. Since each vertex is unique, we
will store everything in the `DistinctCursor::seen_rows_`, which is
backed by a MemoryResource. This setup, on my machine, yields 10 times
better performance when run with MonotonicBufferResource.

Reviewers: mferencevic, mtomic, msantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1894
2019-05-09 15:45:49 +02:00
Tonko Sabolcec
4d8f8be1ab Add function that dumps vertices and edges of the graph.
Summary: Fix lint errors

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1996
2019-05-08 10:52:52 +02:00
Matej Ferencevic
5c244c1ad4 Remove Cap'n Proto
Summary:
There will be a lot of leftover files, execute the following commands inside
`src/` to remove them:
```
git clean -xf
rm -r rpc/ storage/single_node_ha/rpc/
```

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2011
2019-05-08 10:51:10 +02:00
Teon Banek
54247fb266 Implement MonotonicBufferResource
Reviewers: mtomic, mferencevic, msantl

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1997
2019-05-06 16:02:30 +02:00
Tonko Sabolcec
18796c788d Add new PrintIterable to utils algorithm
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2005
2019-05-06 14:51:32 +02:00
Matej Ferencevic
d678e45c10 Migrate RPC to SLK
Summary:
Migrate all RPCs
Simplify Raft InstallSnapshot RPC
Add missing Load and Save for `char`

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2001
2019-05-06 14:27:57 +02:00
Matej Ferencevic
54882be045 Rewrite distributed_serialization test to use SLK
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2004
2019-05-06 10:51:32 +02:00
Matej Ferencevic
129c6c0242 Finish SLK implementation
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1999
2019-05-02 15:47:38 +02:00
dlozic
05e4985fd5 Replace Client with HAClient in feature benchmark
Reviewers: msantl, ipaljak, mferencevic

Reviewed By: msantl, mferencevic

Subscribers: mferencevic, teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1969
2019-05-02 10:40:31 +02:00
Teon Banek
e3fbe56588 Accept utils::MemoryResource in MakeCursor
Summary:
This API change is needed in order to propagate the memory allocation
scheme for the execution of LogicalOperator::Cursor

Depends on D1990

Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1980
2019-04-29 10:34:54 +02:00
Matej Ferencevic
04fea56e91 Remove test_results directory from build
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1987
2019-04-25 17:20:02 +02:00
Matej Ferencevic
c3cfe876c9 Make manual HA client use the new Bolt client
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1976
2019-04-25 15:45:03 +02:00
Matej Ferencevic
2f066b3a83 Remove leftover distributed fields from snapshot explorer
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1981
2019-04-25 10:11:29 +02:00
Teon Banek
4a3df704b6 Fix iterator invalidation when appending to vector
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1979
2019-04-24 17:17:56 +02:00
Teon Banek
e128a9f80f Move distributed query stuff to new folder
Summary:
The new distributed directory is inside the query, and mirrors the query
structure. This groups all of the distributed (query) source code
together, which should make the potential directory extraction easier.

Reviewers: mferencevic, llugovic, mtomic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1923
2019-04-24 09:34:53 +02:00
Matej Ferencevic
9291a5fc4d Migrate to C++17
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1974
2019-04-23 14:46:44 +02:00
Matej Ferencevic
45f2a06a8b Fix various compiler warnings and remove unused test
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1963
2019-04-17 15:48:52 +02:00
Matej Ferencevic
026c796e06 Move GraphDbAccessor to stack in SN and HA
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1944
2019-04-17 11:29:43 +02:00
Lovro Lugovic
6b62145a59 Fix CypherMainVisitor tests relying on traversal order
Summary:
The tests `RelationshipPatternNoDetails` and `PatternPartBraces` in
`memgraph__unit__cypher_main_visitor` checked for the names of the anonymous
identifiers and therefore implicitly relied on the order of the traversal of the
tree.

This "bug" surfaced when Memgraph was compiled with GCC (tested on >= 6.3.0).

Reviewers: mtomic, teon.banek, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1945
2019-04-15 16:02:18 +02:00
Matej Ferencevic
7cf9138308 Fix syntax error in jail faker
Reviewers: msantl, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1957
2019-04-12 12:56:58 +02:00
Marko Budiselic
2f4ca7d3f7 Improve error handling related to the dynamic graph partitioning
Summary:
Fixing bugs in error handling in the following cases (DGP ON):
  * SIGKILL master
  * SIGKILL worker
  * SIGTERM master

Reviewers: mferencevic, msantl, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1896
2019-04-11 16:10:06 +02:00
Marin Tomic
c269e2f468 Add syntax for node key constraints
Reviewers: msantl, teon.banek, llugovic, vkasljevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1953
2019-04-11 10:23:37 +02:00
dlozic
c7684c7e93 Add backpacking tutorial
Summary: New tutorial for backpacking through europe

Reviewers: dsantl, msantl, buda

Reviewed By: dsantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1954
2019-04-11 09:48:56 +02:00
Matija Santl
67ff1cdce3 Add distributed snapshot recovery test
Summary:
Test to check that the recovery works even if the snapshot is corrupted
in distributed.

Depends on D1930

Reviewers: vkasljevic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1950
2019-04-09 12:58:59 +02:00
Matej Ferencevic
63aeed0624 Add SHOW RAFT INFO query to HA
Reviewers: msantl, mtomic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1946
2019-04-08 13:55:55 +02:00
Vinko Kasljevic
47a1e302a0 UniqueLabelPropertiesConstraint implementation and tests
Summary:
Same as `UniqueLabelPropertyConstratint` except it works with multiple properties.
Because of that it is a bit more complex and slower.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1926
2019-04-08 10:14:47 +02:00
Vinko Kasljevic
2a63e2f12d Constraint code cleanup
Summary: Refactor tests, remove some test cases, refactor some methods in `GraphDbAccessor`.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1933
2019-04-03 08:58:33 +02:00
Marin Tomic
dc231fe4e7 Fix bug in SymbolTable
Summary:
This is a bugfix for D1836. It made `SymbolTable` return references to vector
elements, which then get invalidated and weird stuff happens.

This made a `DCHECK` in `rule_based_planner.hpp` trigger, and it was noticed by
@ipaljak 2 months later. All `DCHECK`s in `rule_based_planner.hpp` are now
changed to `CHECK`s.

Also, hash function for `Symbol` was wrong, because it also took
`user_declared` field into consideration, and `==` operator doesn't do that.

Reviewers: ipaljak, teon.banek, mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot, ipaljak

Differential Revision: https://phabricator.memgraph.io/D1938
2019-04-02 16:13:48 +02:00
Matej Ferencevic
3436094df6 Add network functions to jail faker
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot, ipaljak

Differential Revision: https://phabricator.memgraph.io/D1937
2019-04-02 09:31:25 +02:00
Marin Tomic
3965d2341b Add grammar for unique constraints
Summary:
This should be the same as Neo4J grammar. Please check:
https://neo4j.com/docs/cypher-refcard/current/

Reviewers: teon.banek, msantl, llugovic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1936
2019-03-28 16:48:16 +01:00
Vinko Kasljevic
21bd8b492f Add recovery and GraphDbAccessor interface for UniqueLabelPropertyConstraint
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1925
2019-03-27 12:33:56 +01:00
Teon Banek
49b552408b Use label+property index with regex matching
Reviewers: mtomic, llugovic, msantl

Reviewed By: mtomic

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1890
2019-03-19 16:15:32 +01:00
Matija Santl
d387bac544 Fail HA benchmark on non-zero exit status
Summary:
For HA benchmarks, if one of the executables exits with a status other
than zero, the benchmark should fail.

Also, removing `LOG(INFO)`, since failing benchmarks should flag where to look.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1921
2019-03-14 16:53:58 +01:00
Vinko Kasljevic
1c768287e8 Switch from Cache to LruCache
Summary: This diff contains refactor and various fixes needed for LruCache.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1889
2019-03-12 16:11:03 +01:00
Ivan Paljak
d3e00635c6 Measure read throughput in HA
Summary:
This macro benchmark measures read throughput in HA.
The test first creates a random graph with a given number of nodes
and edges. After that, it concurently performs the following query
for 10 seconds:

```
MATCH (n {id:$random_id})-[e]->(m) RETURN e, m;
```

In other words, it randomly picks a node and returns all its neighbours.

Locally measured results are as follows:

| nodes | edges | queries per second |
|  100  |  500  |        8900        |
| 1000  | 5000  |        2700        |
| 10000 | 50000 |        1200        |

Running the same test on Memgraph single node yields very similar results
(up to a few hundred queries).

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1916
2019-03-12 14:55:04 +01:00
Vinko Kasljevic
94afbd9f56 ExistenceConstraints performance improvement
Summary:
Case with existence constraints is about 8-9% slower then case without
existence constraints. Before this diff that difference was about 15-16%.

Reviewers: msantl, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1917
2019-03-12 14:36:27 +01:00
Vinko Kasljevic
8a5842ef63 UniqueLabelPropertyConstraint implementation
Summary:
UniqueLabelPropertyConstraint defines label + property restriction on vertices.
Label + Property + PropertyValue for that property must be unique at any given moment.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1884
2019-03-12 10:02:26 +01:00
Lovro Lugovic
c15c16c577 Prevent result accumulation within LCP's tests
Summary:
PROVE:PLAN clears the previously stored results for the current suite (which is
the suite associated with the current package) and prevents "result
accumulation" (and the accompanying huge and partly outdated reports).

Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1909
2019-03-11 13:12:39 +01:00
Matej Ferencevic
03feffc8cf Improve card fraud test startup
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1900
2019-03-06 13:38:48 +01:00
Vinko Kasljevic
6cc8d8f5b1 Add wal for ExistenceConstraints
Summary: Depends on D1797

Reviewers: msantl, ipaljak, mferencevic

Reviewed By: msantl, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1862
2019-03-06 08:52:00 +01:00
Matej Ferencevic
e07aad44a0 Improve manual bolt client
Summary: Add flag to disable printing of records.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1898
2019-03-04 17:15:07 +01:00
Matej Ferencevic
d9bc4ec476 Implement LDAP authentication
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1888
2019-02-28 09:39:30 +01:00
Matija Santl
54b23ba5b6 Add replication timeout in Raft
Summary:
Added a new config parameter, replication timeout. This parameter sets the
upper limit to the replication phase and once the timeout exceeds, the
transaction engine stops accepting new transactions.

We could experience this timeout in two cases:
 1. a network partition
 2. majority of the cluster stops working

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1893
2019-02-27 17:42:35 +01:00
Teon Banek
65b7976b78 Improve compilation times of AST unit tests
Summary: Depends on D1880

Reviewers: mtomic, mferencevic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1885
2019-02-27 13:04:49 +01:00
Teon Banek
5084123de3 Add regex matching to openCypher
Reviewers: mtomic, msantl

Reviewed By: mtomic

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1880
2019-02-27 12:54:50 +01:00
Matija Santl
4790d6458e Add index support in HA
Summary:
Added index creation and deletion handling in StateDelta.
Also included an integration test that creates an index and makes sure that it
gets replicated by killing each peer eventually causing a leader re-election.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1886
2019-02-26 12:55:57 +01:00
Matija Santl
a50abcab99 Fix typo in Raft test
Summary: lowhanging

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1887
2019-02-26 10:01:17 +01:00
Vinko Kasljevic
bbb69a1c00 Edges iterator implementation
Summary:
`EdgesIterable` is used for iterating over Edges in distributed memgraph. Because of lru cache there is a possibility of data getting evicted as someone iterates over it.
To prevent that `EdgesIterable` will lock that data and release it when it's deconstructed.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1868
2019-02-26 09:44:22 +01:00
Teon Banek
4d1d9fb15a Add syntax for managing data constraints
Reviewers: mtomic, mferencevic, buda, msantl

Reviewed By: mtomic, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1879
2019-02-25 14:31:29 +01:00
Matej Ferencevic
7be23896c2 Make auth library case insensitive
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1881
2019-02-22 15:50:51 +01:00
Vinko Kasljevic
3ad5c8069c Add CachedDataLock throughout the code
Summary:
CachedDataLock is necessary for lru cache as remote data is no longer
persistent. Most methods internally handle this, but for methods that
return pointers or references to remote data, we need to manually
lock data.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1869
2019-02-21 15:27:40 +01:00
Vinko Kasljevic
7dba861534 Add micro benchmark for existence constraint
Summary: Benchmark shows that database with `ExistenceConstraints` is around 16% slower compared to case without `ExistenceConstraints`.

Reviewers: teon.banek, msantl, ipaljak, mferencevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1876
2019-02-21 08:56:12 +01:00
Matej Ferencevic
fe361fb750 Implement audit logging
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1863
2019-02-19 15:11:30 +01:00
Teon Banek
6bba5f4cd0 Add clauses for showing DB information
Summary:
Also add STATS Privilege and Permission.
Update tests and changelog.

Reviewers: mtomic, mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1873
2019-02-19 13:32:43 +01:00
Teon Banek
8589dd124d Remove unused variables
Reviewers: mferencevic, msantl, ipaljak, vkasljevic

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1870
2019-02-18 10:45:53 +01:00
Vinko Kasljevic
a14c4f1864 Existence constraints for vertex for single node
Summary:
Existence constraint ensures that all nodes with certain label have a certain property.
`ExistenceRule` defines label -> properties rule and `ExistenceConstraints` manages all
constraints.

Reviewers: msantl, ipaljak, teon.banek, mferencevic

Reviewed By: msantl, teon.banek, mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1797
2019-02-15 13:14:31 +01:00
Matija Santl
ed75e45541 Fix Raft failure discovered in apollo run 479391
Summary:
We noticed a Raft test failure https://apollo.memgraph.io/runs/479391/

This diff should fix it.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1865
2019-02-15 10:22:52 +01:00
Matej Ferencevic
d9673698e5 Cleanup file utils
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1861
2019-02-14 12:48:37 +01:00
Vinko Kasljevic
3e739f33c9 Add LruCache implementation and tests
Reviewers: msantl, ipaljak, teon.banek

Reviewed By: msantl, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1858
2019-02-14 11:03:33 +01:00
Vinko Kasljevic
127a67ab13 Remove DistributedGraphDb
Summary: DistributedGraphDb is no longer needed.

Reviewers: msantl, teon.banek, ipaljak

Reviewed By: msantl, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1857
2019-02-14 09:38:32 +01:00
Matej Ferencevic
bb052be002 Remove serialization from utils
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1852
2019-02-13 15:41:40 +01:00
Teon Banek
cbd6f3bbe2 Move RandomGraphGenerator from utils to test
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1855
2019-02-12 16:49:13 +01:00
Matija Santl
f85095c203 Fix Raft shutdown
Summary:
During the following scenario:
 - start a HA cluster with 3 machines
 - find the leader and start sending queries
 - SIGTERM the leader but leave other 2 machines untouched

The leader would be stuck in the shutdown phase.

This was happening because during the shutdown phase of the Bolt server, a
`graph_db_accessor` would try to commit a transaction after we've already shut
down Raft server.  Raft, although not running, is still thinking it's in the
Leader mode. Tx Engine calls the `SafeToCommit` method to Commit transactions,
and ends up in an infinite loop.

Since Raft was shut down it won't handle any of the incoming RPCs and won't
change it's mode.

The fix here is to shut down the Bolt server before Raft, so we don't have any
pending commits once Raft is shut down.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1853
2019-02-12 15:12:39 +01:00
Matej Ferencevic
773acc11d6 Remove stats from main binaries
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1851
2019-02-11 14:33:41 +01:00
Matija Santl
59e11dbf9f Add Raft log compaction transfer test
Summary:
I've refactored the integration test for HA so we can reuse the common
parts like starting/stopping workers.

I've also added a test that triggers the log compaction and it checks that the
snapshot that has been transferred is the same as the origin one.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1847
2019-02-07 15:25:35 +01:00
Teon Banek
53d8f725c5 Build mg-comm-rpc library
Summary:
This change splits mg-communication into mg-communication and
mg-comm-rpc. The main reason for doing this, is to make separation of
enterprise features from community Memgraph more clear.

Reviewers: mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1844
2019-02-07 14:40:15 +01:00
Teon Banek
e461a08340 Move creating indexed lookup to a rewrite pass
Summary:
RuleBasedPlanner now generates only the regular ScanAll operations, and
Filter operations are appended as soon as possible. The newly added
Rewrite step, takes this operator tree and replaces viable Filter &
ScanAll operators with appropriate ScanAllBy<Index> operator. This
change ought to simplify the behaviour of DistributedPlanner when that
stage is moved before the indexed lookup rewrite.

Showing unoptimized plan in interactive planner is also supported.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1839
2019-02-06 14:46:19 +01:00
Marin Tomic
c03ca5f8f0 Remove UID tracking from AstStorage
Summary:
All AST nodes had a member `uid_` that was used as a key in
`SymbolTable`. It is renamed to `symbol_pos_` and it appears only in
`Identifier`, `NamedExpression` and `Aggregation`, since only those types were
used in `SymbolTable`. SymbolGenerator is now responsible for creating symbols
in `SymbolTable` and assigning positions to AST nodes.

Cloning and serialization code is now simpler since there is no need to track
UIDs.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1836
2019-02-05 14:19:48 +01:00
Matija Santl
145c81376f Add log compaction for Raft, pt. 2
Summary: Implemented snapshot replication and log compaction.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1840
2019-02-04 15:32:07 +01:00
Matija Santl
da95cbf4ec Add log compaction for Raft, pt. 1
Summary:
In this part of log compaction for raft, I've implemented snapshooting
and snapshot recovery. I've also refactored the code a bit, so `RaftServer` now
has a pointer to the `GraphDb` and it can do some things by itself.

Log compaction requires some further work. Since snapshooting isn't synchronous
between peers, and each peer can work at their own pace, once we've compacted
the log so that the next log to be sent to peer `x` isn't available anymore, we
need to send the snapshot over the wire. This means that the next part will
contain the `InstallSnapshotRPC` and then maybe one more that will implement the
logic of sending `LogEntry` or the whole snapshot.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1834
2019-01-29 14:58:57 +01:00
Marin Tomic
7542f5b0ba Include JSON serialized logical plan in summary for EXPLAIN
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1831
2019-01-25 17:01:17 +01:00
Matija Santl
4fa44c3edd Fix Raft's ReplicationLog
Summary:
`ReplicationLog` had a classic off-by-one bug. The `valid_prefix`
variable wasn't set properly.

This diff also includes a poor man's version of a HA client. This client
assumes that all the HA instances run on a single machine and that the
corresponding Bold endpoints have open ports ranging from `7687` to
`7687 + num_machines - 1`.

This should make it easeir to test certain things, ie. disk usage, P25.

This test revealed the bug with `ReplicationLog`

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1813
2019-01-23 16:27:51 +01:00
Marin Tomic
9d36f775f2 Remove GraphView from variable expansion operators
Summary:
Variable expansions cannot appear in merge patterns or after updates,
so they can only be planned with GraphState::OLD. Because of that, it makes
sense to remove GraphView parameter from them to reduce confusion.

Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1825
2019-01-22 15:35:05 +01:00
Marin Tomic
914f40411a Implement plan cloning with LCP
Summary:
Implement proper plan cloning using LCP instead of hacking it with
serialization.

depends on D1815

Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1816
2019-01-22 14:52:06 +01:00
Matija Santl
62e06d4b70 Fix re-election in Raft
Summary:
Once a leader loses it's leadership, in order to handle hanging
transactions, we reset the storage and the transaction engine.

This requires to re-apply all the commited entries from the log.

Once we add snapshot (log compaction) we would need to do that also.

One thing to have in mind is the `election_timeout_min` parameter. If it's set
too low it could trigger leader re-election too often.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1822
2019-01-22 14:51:24 +01:00