Commit Graph

1549 Commits

Author SHA1 Message Date
Teon Banek
64a05bc972 Use CHECK in TypedValueVectorCompare|Equal
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2066
2019-05-28 09:41:31 +02:00
Matija Santl
e4fd49a530 Remove unique index from query execution
Summary: Remove leftover unique index code from query execution.

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2078
2019-05-28 08:16:35 +02:00
Matej Ferencevic
42bf81021e Fix multiple raft issues
Summary:
Fix condition variable notifications
Fix vote requested invalid memory access (size off by one)
Fix blocking wait in RaftPeer while candidate
Don't copy large log entries when only the term is needed

Reviewers: ipaljak, msantl

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2088
2019-05-27 11:03:24 +02:00
Teon Banek
88778fc70f Explicitly forbid propagation of utils::Allocator
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2080
2019-05-24 15:15:00 +02:00
Matej Ferencevic
0638179a8f Fix raft voting
Reviewers: ipaljak, msantl

Reviewed By: ipaljak, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2082
2019-05-24 12:21:37 +02:00
Matija Santl
4ec3f62f4e Move constraints to common
Summary:
Preparing unique constraint code to be implemented in HA. To do so, I'm
moving everything to `stograge/common/` folder. I also added a new header,
`gid.hpp` which does a `ifdef` include of the correct `gid.hpp` based on the
product.

Reviewers: ipaljak, mferencevic, vkasljevic, teon.banek

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2079
2019-05-23 16:36:14 +02:00
Matija Santl
e1fdb85a34 Keep SnapshotMetadata in memory
Summary:
Instead of asking rocksdb for snapshot metadata, keep it in memory for
faster access.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2075
2019-05-23 14:26:12 +02:00
Matija Santl
f2cc41fa59 Remove unique from label-prop index
Summary: Remove unique feature from label-property index

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: tsabolcec, pullbot

Differential Revision: https://phabricator.memgraph.io/D2045
2019-05-23 12:39:41 +02:00
Vinko Kasljevic
1f93edaa88 Add checks to GlobalAddress in order to prevent misuse
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2074
2019-05-21 15:37:03 +02:00
Matej Ferencevic
3c48e612ef Make move operations noexcept
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1991
2019-05-21 10:46:25 +02:00
Teon Banek
ee2fa47b29 Remove inheriting TotalOrdering in TypedValue
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2070
2019-05-21 09:51:18 +02:00
Matija Santl
5d5dfbb6f7 Fix how HA handles leader change during commit
Summary:
During it's leadership, one peer can receive RPC messages from other peers that his reign is over.
The problem is when this happens during a transaction commit.

This is handled in the following way.
If we're the current leader and we want to commit a transaction, we need to make sure the Raft Log is replicated before we can tell the client that the transaction is committed.
During that wait, we can only notice that the replication takes too long, and we report that with `LOG(WARNING)` messages.

If we change the Raft mode during the wait, our Raft implementation will internally commit this transaction, but won't be able to acquire the Raft lock because the `db.Reset` has been called.
This is why there is an manual lock acquire. If we pick up that the `db.Reset` has been called, we throw an `UnexpectedLeaderChangeException` exception to the client.

Another thing with long running transactions, if someone decides to kill a `memgraph_ha` instance during the commit, the transaction will have `abort` hint set. This will cause the `src/query/operator.cpp` to throw a `HintedAbortError`. We need to catch this during the shutdown, because the `memgraph_ha` isn't dead from the user perspective, and the transaction wasn't aborted because it took too long, but we can differentiate between those two.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic, ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1956
2019-05-20 16:39:44 +02:00
Matija Santl
37c68f0508 Add multiple properties unique constraint
Summary:
Unique constraint now support multiple properties

Depends on D2043

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2044
2019-05-20 15:18:05 +02:00
Matej Ferencevic
1e79313538 Make HA heartbeat independent of other RPCs
Reviewers: msantl, ipaljak

Reviewed By: msantl, ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2073
2019-05-20 15:04:15 +02:00
Matej Ferencevic
6c49e6de02 Add SSL support to HA RPC
Reviewers: msantl, teon.banek

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2055
2019-05-20 12:53:52 +02:00
Teon Banek
4f4837392e Use MemoryResource in UnwindCursor
Summary:
Micro benchmarks show no change compared to global new & delete. This is
to be expected, because Unwind relies only on `std::vector` which ought
to reserve the memory in reasonable chunks.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2064
2019-05-17 15:28:54 +02:00
Teon Banek
880f23620d Move UnwindCursor from LCP to CPP
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2062
2019-05-17 15:28:54 +02:00
Teon Banek
8459fd90fc Use MemoryResource in OrderByCursor
Summary:
Micro benchmarks show an improvement to performance of about 10%
compared to global new & delete.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2061
2019-05-17 15:28:53 +02:00
Teon Banek
abbb57c868 Move OrderByCursor from LCP to CPP
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2060
2019-05-17 15:28:53 +02:00
Teon Banek
0075eee58b Use MemoryResource in AggregationCursor
Summary: Micro benchmarks show some improvement, but unfortunately not much.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2058
2019-05-17 15:28:53 +02:00
Matej Ferencevic
6c84092023 Add GetLeaderId to HAClient
Reviewers: msantl, dlozic, ipaljak

Reviewed By: msantl, dlozic, ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2059
2019-05-16 15:22:51 +02:00
Ivan Paljak
137e020b22 Improve sync performance in Raft
Summary: We should now exchange around O(log(n)) messages to get a follower that is n transactions behind back in sync.

Reviewers: msantl, mferencevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2051
2019-05-16 13:53:13 +02:00
Matej Ferencevic
d4b2d76a35 Reimplement counter openCypher function
Reviewers: teon.banek, msantl

Reviewed By: teon.banek, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2047
2019-05-16 11:09:02 +02:00
Teon Banek
98a853a95c Move AggregateCursor from LCP to CPP
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2056
2019-05-16 09:58:15 +02:00
Teon Banek
f9471f341d Fix move assign of MonotonicBufferResource
Summary:
I was a bit stupid and thought I can use `std::swap`, while `std::swap`
is implemented in terms of move itself.

Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2054
2019-05-15 17:20:32 +02:00
Matej Ferencevic
6082d31843 Fix communication context SSL leaks
Summary:
A `valgrind` run of a modified client that connects multiple times to the
database now shows no memory leaks, previously the context created with
`SSL_CTX_new` was leaked every time.

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek, mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2052
2019-05-15 16:47:54 +02:00
Matija Santl
52adbc6a8b Revert "Introduce shutdown mode in Raft"
Summary: This reverts commit 08b9197c79.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic, ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2053
2019-05-15 16:39:32 +02:00
Teon Banek
1cc71c6ce8 Use MemoryResource in AccumulateCursor
Summary:
Micro benchmarks show that MonotonicBufferResource improves performance
by a factor of 1.5.

Reviewers: mtomic, mferencevic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2048
2019-05-15 16:13:35 +02:00
Teon Banek
78830b6ed8 Use MemoryResource in STShortestPathCursor
Summary:
Benchmarks show minor improvements. Perhaps it makes sense at some later date
to use another allocator for things lasting only in a single `Pull`.

Reviewers: mferencevic, mtomic, llugovic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2018
2019-05-15 16:13:23 +02:00
Teon Banek
801bdb3a91 Use MemoryResource for members of ExpandVariableCursor
Summary:
Unfortunately, the written micro benchmark only reports minor
improvements compared to default allocator. The results are in some
cases even a tiny bit worse.

Reviewers: mtomic, mferencevic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2039
2019-05-15 16:07:27 +02:00
Tonko Sabolcec
b09b21b832 Dump indices in CypherDumpGenerator
Summary:
This change introduces dumping of indices keys. During the dump process,
an internal label is assigned to each vertex and index on vertex's
internal property id is created for faster matching during edge creation.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: msantl, pullbot

Differential Revision: https://phabricator.memgraph.io/D2046
2019-05-15 11:05:16 +02:00
Matija Santl
d70792f1ce Remove existence constraint
Summary: Removing existence constraint

Reviewers: ipaljak, mferencevic, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2043
2019-05-15 09:48:51 +02:00
Tonko Sabolcec
5f24342502 Split a single dump query into multiple queries
Summary:
Prior to this change, a huge query was returned by DumpGenerator that
dumped the entire graph. This change split the single query to multiple
queries, each dumping a single vertex/edge. For easier vertex matching
when dumping edge, an internal property id is assigned to each vertex and
removed after the whole graph is dumped.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2038
2019-05-14 14:22:28 +02:00
Teon Banek
7a586b3686 Allocate Cursor through utils::MemoryResource
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2026
2019-05-13 11:36:40 +02:00
Teon Banek
2909ef63d2 Use utils::MonotonicBufferResource in query execution
Reviewers: mferencevic, mtomic, msantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2016
2019-05-13 11:36:19 +02:00
Lovro Lugovic
017dec8c0e LCP: Refactor the type representation
Summary:
# Summary

## Concepts and terminology

A **namestring for <object>** is a Lisp string that names the C++ language element
`<object>` (such as a namespace, a variable, a class, etc.). Therefore we have a
"namestring for a namespace", a "namestring for a variable", etc.

A **designator for a namestring for <object>** is a Lisp object that denotes a
"namestring for <object>". These are symbols and strings themselves.

A **typestring** is a Lisp string that represents a C++ type, i.e. its
declaration.

A **typestring designator** is a Lisp object that denotes a typestring.

A typestring (and the corresponding C++ type) is said to be **supported** if it
can be parsed using `parse-cpp-type-declaration`. This concept is important and
should form the base of our design because we can't really hope to ever support
all of C++'s type declarations.

A typestring (and the corresponding C++ type) that is not supported is
**unsupported**.

A **processed typestring** is a typestring that is either fully qualified or
unqualified.

A C++ type is said to be **known** if LCP knows extra information about the type,
rather than just what kind of type it is, in which namespace it lives, etc. For
now, the only known types are those that are defined within LCP itself using
`define-class` & co.

A C++ type is **unknown** if it is not known.

**Typestring resolution** is the process of resolving a (processed) typestring
into an instance of `cpp-type` or `unsupported-cpp-type`.

**Resolving accessors** are accessors which perform typestring resolution.

## Changes

Explicitly introduce the concept of supported and known types. `cpp-type` models
supported types while `unsupported-cpp-type` models unsupported types.
Subclasses of `cpp-type` model known types. `general-cpp-type` is either a
`cpp-type` or an `unsupported-cpp-type`.

Add various type queries.

Fix `define-rpc`'s `:initarg` (remove it from the `cpp-member` struct).

Introduce namestrings and their designators in `names.lisp`.

Introduce typestrings and their designators.

Introduce **processed typestrings**. Our DSL's macros (`define-class` & co.)
convert all of the given typestrings into processed typestrings because we don't
attempt to support partially qualified names and relative name lookup yet. A
warning is signalled when a partially qualified name is treated as a fully
qualified name.

The slots of `cpp-type`, `cpp-class`, `cpp-member` and `cpp-capnp-opts` now
store processed typestrings which are lazily resolved into their corresponding
C++ types.

The only thing that instances of `unsupported-cpp-type` are good for is getting
the typestring that was used to construct them. Most of LCP's functions only
work with known C++ types, i.e. `cpp-type` instances. The only function so far
that works for both of them is `cpp-type-decl`.

Since "unsupportedness" is now explicitly part of LCP, client code is expected
to manually check whether a returned type is unsupported or not (or face
receiving an error otherwise), unless a function is documented to return only
`cpp-type` instances.

A similar thing goes for "knowness". Client code is expected to manually check
whether a returned type is known or not, unless a function is documented to
return only (instances of `cpp-type` subclasses) known types.

## TODO

Resolution still has to be done for the following slots of the
following structures:

-   `slk-opts`
    -   `save-args`
    -   `load-args`

-   `clone-opts`
    -   `args`
    -   `return-type`

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1962
2019-05-10 16:18:31 +02:00
Lovro Lugovic
e8c82e36e2 LCP: Remove Cap'n Proto
Summary: Depends on D1947

Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2034
2019-05-10 16:10:22 +02:00
Lovro Lugovic
390b6c1557 LCP: Small test fixes
Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1947
2019-05-10 16:10:17 +02:00
Lovro Lugovic
9d6e025304 LCP: SLIME debugging support when used as a tool
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1929
2019-05-10 16:09:28 +02:00
Tonko Sabolcec
60d72f489a Add DumpGenerator class
Summary: DumpGenerator dumps parts of query to stream.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2023
2019-05-10 15:57:37 +02:00
Matej Ferencevic
70ed1706b8 Add explicit common LCP target
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2033
2019-05-10 11:14:23 +02:00
Teon Banek
1c36f8ceb9 Add LCP generation of common LCP files
Summary:
This should hopefully resolve multithreading issues when multiple LCP
invocations tried to process the same file.

Reviewers: mferencevic, msantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2029
2019-05-09 17:23:48 +02:00
Ivan Paljak
358391bd22 Fix low read throughput due to active waits
Summary:
Read throughput dropped by about 50% from the last diff.
This should fix it (at least according to local measurements).

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2027
2019-05-09 16:28:39 +02:00
Teon Banek
394039a05e Use and bench custom allocator in Distinct
Summary:
According to the written benchmark, using MonotonicBufferResource yields
significant improvements to performance of Distinct. The setup fills the
database with vertices depending on the benchmark state. No edges are
created. Then we run DISTINCT on that. Since each vertex is unique, we
will store everything in the `DistinctCursor::seen_rows_`, which is
backed by a MemoryResource. This setup, on my machine, yields 10 times
better performance when run with MonotonicBufferResource.

Reviewers: mferencevic, mtomic, msantl

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1894
2019-05-09 15:45:49 +02:00
Ivan Paljak
e502b1a306 Remove SM simulation from HB issuer thread
Summary:
This allows us to decouple issuing heartbeats from the server mode
which is useful when we know we will transition to LEADER but cannot yet
change the mode due to some Raft internals.

It can now happen that a couple of HBs are sent when in FOLLOWER or CANDIDATE
mode, but this doesn't affect the correctness of the protocol.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2025
2019-05-09 13:40:09 +02:00
Ivan Paljak
08b9197c79 Introduce shutdown mode in Raft
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2019
2019-05-08 14:16:20 +02:00
Tonko Sabolcec
4d8f8be1ab Add function that dumps vertices and edges of the graph.
Summary: Fix lint errors

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1996
2019-05-08 10:52:52 +02:00
Matej Ferencevic
5c244c1ad4 Remove Cap'n Proto
Summary:
There will be a lot of leftover files, execute the following commands inside
`src/` to remove them:
```
git clean -xf
rm -r rpc/ storage/single_node_ha/rpc/
```

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2011
2019-05-08 10:51:10 +02:00
Vinko Kasljevic
16d4a7b5b6 Add nullptr check in LruList
Summary:
Clang-analyzer-core found NullDereference issue in LruList. I don't see
how that can occur, but as a precaution I'll add a CHECK.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2015
2019-05-08 08:52:17 +02:00
Tonko Sabolcec
0849937aea Add less than operator for property value
Summary: This change will allow comparison of sets/maps containing `PropertyValue`s.

Reviewers: teon.banek, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2008
2019-05-07 15:14:22 +02:00
Teon Banek
8e6317436d Add missing comma to stripped_lexer_constants
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2010
2019-05-07 09:39:05 +02:00
Teon Banek
54247fb266 Implement MonotonicBufferResource
Reviewers: mtomic, mferencevic, msantl

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1997
2019-05-06 16:02:30 +02:00
Matej Ferencevic
291f0425e2 Use SLK for Raft log
Reviewers: msantl, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2007
2019-05-06 15:29:56 +02:00
Tonko Sabolcec
18796c788d Add new PrintIterable to utils algorithm
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2005
2019-05-06 14:51:32 +02:00
Matej Ferencevic
d678e45c10 Migrate RPC to SLK
Summary:
Migrate all RPCs
Simplify Raft InstallSnapshot RPC
Add missing Load and Save for `char`

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2001
2019-05-06 14:27:57 +02:00
Ivan Paljak
5833e6cc0f Introduce new RPC for heartbeats in Raft
Summary:
Queries such as `UNWIND RANGE(1, 90000) AS x CREATE(:node{id:x});` work
now. At some point (cca 100000 state deltas) the messages become too large for
Cap'n Proto to handle and that exception kills the cluster. This is fine since
we are about to replace it anyway.

Also, the issue with leadership change during replication still remains if the
user tempers with the constants (election timeout more or less the same to hb
interval). This is unavoidable, so we need to address that issue in the future.

Also, I've removed the unnecessary `backoff_until_` variable and the logic around it.
It's a small change so I kept it within this diff (sorry).

Reviewers: msantl, mferencevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1998
2019-05-06 13:33:10 +02:00
Matej Ferencevic
129c6c0242 Finish SLK implementation
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1999
2019-05-02 15:47:38 +02:00
Teon Banek
95f4d1c3fa Generate static Save and Load methods for RPCs
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1995
2019-04-29 13:42:17 +02:00
Lovro Lugovic
5dc362d9f5 LCP: Use named-readtables
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1928
2019-04-29 12:20:06 +02:00
Matej Ferencevic
c5bf7b3c03 Refactor HA coordination
Reviewers: msantl, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1992
2019-04-29 11:26:26 +02:00
Teon Banek
bc46de7b33 Move DistinctCursor to operator.cpp
Summary: Depends on D1980

Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1986
2019-04-29 10:35:44 +02:00
Teon Banek
e3fbe56588 Accept utils::MemoryResource in MakeCursor
Summary:
This API change is needed in order to propagate the memory allocation
scheme for the execution of LogicalOperator::Cursor

Depends on D1990

Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1980
2019-04-29 10:34:54 +02:00
Teon Banek
2904e86158 Add MemoryResource and Allocator interfaces
Reviewers: mtomic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1990
2019-04-26 09:36:14 +02:00
Matej Ferencevic
8b7df83473 Abort SHOW RAFT INFO queries
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1983
2019-04-25 17:38:40 +02:00
Matej Ferencevic
6182312e3d Remove TX info from HA snapshot
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1984
2019-04-25 14:00:00 +02:00
Matej Ferencevic
872b043bed Fix snapshot recovery max TX logic
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1982
2019-04-25 10:11:43 +02:00
Ivan Paljak
c2bc5fe377 Introduce an in-memory copy of Raft log
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1978
2019-04-24 16:06:14 +02:00
Matej Ferencevic
3ffed4bf6d Refactor HA RPC clients
Reviewers: msantl, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1977
2019-04-24 15:40:27 +02:00
Teon Banek
e128a9f80f Move distributed query stuff to new folder
Summary:
The new distributed directory is inside the query, and mirrors the query
structure. This groups all of the distributed (query) source code
together, which should make the potential directory extraction easier.

Reviewers: mferencevic, llugovic, mtomic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1923
2019-04-24 09:34:53 +02:00
Matej Ferencevic
9291a5fc4d Migrate to C++17
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1974
2019-04-23 14:46:44 +02:00
Matej Ferencevic
1e7bdcec4d Initial version of HA client
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1958
2019-04-18 16:48:30 +02:00
Matej Ferencevic
8999f5fccd Extract HA config to separate file
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1964
2019-04-17 18:45:54 +02:00
Matej Ferencevic
45f2a06a8b Fix various compiler warnings and remove unused test
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1963
2019-04-17 15:48:52 +02:00
Matej Ferencevic
026c796e06 Move GraphDbAccessor to stack in SN and HA
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1944
2019-04-17 11:29:43 +02:00
Marko Budiselic
2f4ca7d3f7 Improve error handling related to the dynamic graph partitioning
Summary:
Fixing bugs in error handling in the following cases (DGP ON):
  * SIGKILL master
  * SIGKILL worker
  * SIGTERM master

Reviewers: mferencevic, msantl, vkasljevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1896
2019-04-11 16:10:06 +02:00
Matej Ferencevic
4029026c3b Add bolt verbose error
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1948
2019-04-11 14:20:40 +02:00
Marin Tomic
c269e2f468 Add syntax for node key constraints
Reviewers: msantl, teon.banek, llugovic, vkasljevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1953
2019-04-11 10:23:37 +02:00
Matija Santl
a7ee70e365 Add KVStore::CompactRange to free disk space in HA
Summary:
Manually call `CompactRange` on KVStore (rocksDB) to free disk space
upon Raft log compaction.

Raft log takes surprisingly large amount of disk space (1MB for 1000 entries) so
we need to free disk space as we compact the log into snapshot.

When `log_size_snapshot_threshold` set to `5000` the disk usage of the
`durability_dir` goes from `4.8 MB` to `240 KB`.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1939
2019-04-10 15:29:59 +02:00
Matija Santl
97e2289222 Fix term updates in Memgraph HA
Summary: Fix https://app.asana.com/0/870082256568329/1117622284146426/f

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1949
2019-04-09 12:59:37 +02:00
Matija Santl
ad0d9e1f87 Fix distributed snapshot recovery failure
Summary:
This is still in progress.

As reported in
https://app.asana.com/0/743890251333732/971034572323525/f
memgraph distributed fails when the snapshot that it tries to recover from is
invalid. Instead of inheritance of `StorageGc` this diff adds composition and we
don't try to register a rpc server after the initialization.

Reviewers: ipaljak, vkasljevic, mtomic

Reviewed By: vkasljevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1930
2019-04-08 14:47:44 +02:00
Matej Ferencevic
63aeed0624 Add SHOW RAFT INFO query to HA
Reviewers: msantl, mtomic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1946
2019-04-08 13:55:55 +02:00
Vinko Kasljevic
47a1e302a0 UniqueLabelPropertiesConstraint implementation and tests
Summary:
Same as `UniqueLabelPropertyConstratint` except it works with multiple properties.
Because of that it is a bit more complex and slower.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1926
2019-04-08 10:14:47 +02:00
Matija Santl
e1ad5cd803 Prevent snapshot recovery when exiting
Summary:
Sometimes, when the leader resigns it's leadership, a newly elected
leader would send the old leader `AppendEntriesRPC` that would cause the
snapshot recovery to happen. This diff prevents that.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1943
2019-04-04 14:41:33 +02:00
Matija Santl
6a9acb717d Refactor StateDeltaApplier for HA
Summary:
The whole `StateDeltaApplier` implementation was unnecessary. Fixing
this.

Reviewers: mferencevic, ipaljak

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1942
2019-04-04 13:57:58 +02:00
Matija Santl
1124b8d371 Fix Raft election
Summary:
Based on the failure manifested in
https://apollo.memgraph.io/runs/512803/ it seems like machines give each other
votes for the same term.

Looking at the code, `voted_for_` variable wasn't assigned on the election start
and the election starter could grant his vote to someone else but would still
count his vote to himself.

Reviewers: ipaljak, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot, vkasljevic

Differential Revision: https://phabricator.memgraph.io/D1941
2019-04-03 12:45:57 +02:00
Vinko Kasljevic
2a63e2f12d Constraint code cleanup
Summary: Refactor tests, remove some test cases, refactor some methods in `GraphDbAccessor`.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1933
2019-04-03 08:58:33 +02:00
Marin Tomic
dc231fe4e7 Fix bug in SymbolTable
Summary:
This is a bugfix for D1836. It made `SymbolTable` return references to vector
elements, which then get invalidated and weird stuff happens.

This made a `DCHECK` in `rule_based_planner.hpp` trigger, and it was noticed by
@ipaljak 2 months later. All `DCHECK`s in `rule_based_planner.hpp` are now
changed to `CHECK`s.

Also, hash function for `Symbol` was wrong, because it also took
`user_declared` field into consideration, and `==` operator doesn't do that.

Reviewers: ipaljak, teon.banek, mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot, ipaljak

Differential Revision: https://phabricator.memgraph.io/D1938
2019-04-02 16:13:48 +02:00
Marin Tomic
3965d2341b Add grammar for unique constraints
Summary:
This should be the same as Neo4J grammar. Please check:
https://neo4j.com/docs/cypher-refcard/current/

Reviewers: teon.banek, msantl, llugovic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1936
2019-03-28 16:48:16 +01:00
Ivan Paljak
750115e8ff Display server state changes to user
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1932
2019-03-27 14:24:21 +01:00
Vinko Kasljevic
21bd8b492f Add recovery and GraphDbAccessor interface for UniqueLabelPropertyConstraint
Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1925
2019-03-27 12:33:56 +01:00
Teon Banek
49b552408b Use label+property index with regex matching
Reviewers: mtomic, llugovic, msantl

Reviewed By: mtomic

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1890
2019-03-19 16:15:32 +01:00
Teon Banek
a14c24e71f Use const_cast in RecordAccessor
Summary:
It seems that older clang versions erroneously accept modification of
constant pointers, this is a quick hack which resolves the issue on new
clang versions.

None of the const methods are really const and should not be marked as
such. This whole accessor thing is just a steaming pile of crap and
should be rewritten from scratch.

Reviewers: vkasljevic, msantl

Reviewed By: vkasljevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1922
2019-03-18 11:31:05 +01:00
Teon Banek
12799f12ef Fill the EvaluationContext correctly for ProfileQuery
Reviewers: mtomic, llugovic, msantl

Reviewed By: llugovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1919
2019-03-15 13:07:44 +01:00
Matija Santl
d387bac544 Fail HA benchmark on non-zero exit status
Summary:
For HA benchmarks, if one of the executables exits with a status other
than zero, the benchmark should fail.

Also, removing `LOG(INFO)`, since failing benchmarks should flag where to look.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1921
2019-03-14 16:53:58 +01:00
Matija Santl
f97872170a Add a lock around replication timeout map in Raft
Summary:
Concurent access to the map that contains replication log timeouts
caused the HA version to often report replication log timeout errors.

Adding locks around the access prevets them from happening.

Performance on Apollo reports write speed around 8k/s.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1920
2019-03-14 10:52:03 +01:00
Vinko Kasljevic
1c768287e8 Switch from Cache to LruCache
Summary: This diff contains refactor and various fixes needed for LruCache.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1889
2019-03-12 16:11:03 +01:00
Vinko Kasljevic
94afbd9f56 ExistenceConstraints performance improvement
Summary:
Case with existence constraints is about 8-9% slower then case without
existence constraints. Before this diff that difference was about 15-16%.

Reviewers: msantl, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1917
2019-03-12 14:36:27 +01:00
Vinko Kasljevic
8a5842ef63 UniqueLabelPropertyConstraint implementation
Summary:
UniqueLabelPropertyConstraint defines label + property restriction on vertices.
Label + Property + PropertyValue for that property must be unique at any given moment.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1884
2019-03-12 10:02:26 +01:00
Lovro Lugovic
11f3cb8838 Undefine CPP types at the end of LCP's test
Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1915
2019-03-11 14:45:32 +01:00
Lovro Lugovic
0679997e8f Use the FIVEAM reporter in LCP's test
Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1914
2019-03-11 13:21:38 +01:00
Lovro Lugovic
c15c16c577 Prevent result accumulation within LCP's tests
Summary:
PROVE:PLAN clears the previously stored results for the current suite (which is
the suite associated with the current package) and prevents "result
accumulation" (and the accompanying huge and partly outdated reports).

Reviewers: mtomic, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1909
2019-03-11 13:12:39 +01:00
Matija Santl
1de34d8b92 Add proper storage stats for HA
Summary:
`SHOW STORAGE STATS` when executed in a Raft cluster should return
stats for each member of the cluster.

`StorageStats` starts a RPC server on each member of the cluster that answers
about its local storage stats.

The query can be invoked only on the current leader, the leader sends a request
to each peer and shows the results it gets. If some peers don't answer within 1
second, stats for those peers won't be shown.

The new output can be seen here: P27

Reviewers: ipaljak, mferencevic

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1907
2019-03-07 15:00:40 +01:00
Vinko Kasljevic
6cc8d8f5b1 Add wal for ExistenceConstraints
Summary: Depends on D1797

Reviewers: msantl, ipaljak, mferencevic

Reviewed By: msantl, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1862
2019-03-06 08:52:00 +01:00
Matej Ferencevic
768aeab942 Add WAL recovery logging
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1899
2019-03-04 17:14:23 +01:00
Matej Ferencevic
d9bc4ec476 Implement LDAP authentication
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1888
2019-02-28 09:39:30 +01:00
Matija Santl
54b23ba5b6 Add replication timeout in Raft
Summary:
Added a new config parameter, replication timeout. This parameter sets the
upper limit to the replication phase and once the timeout exceeds, the
transaction engine stops accepting new transactions.

We could experience this timeout in two cases:
 1. a network partition
 2. majority of the cluster stops working

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1893
2019-02-27 17:42:35 +01:00
Teon Banek
5084123de3 Add regex matching to openCypher
Reviewers: mtomic, msantl

Reviewed By: mtomic

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1880
2019-02-27 12:54:50 +01:00
Ivan Paljak
ed28ed873d Remove Kafka, auth and audit log from Memgraph HA
Summary: Removing Kafka, auth and audit log features from HA instance for the time being.

Reviewers: mferencevic, msantl

Reviewed By: mferencevic, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1882
2019-02-27 10:25:26 +01:00
Matija Santl
4790d6458e Add index support in HA
Summary:
Added index creation and deletion handling in StateDelta.
Also included an integration test that creates an index and makes sure that it
gets replicated by killing each peer eventually causing a leader re-election.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1886
2019-02-26 12:55:57 +01:00
Vinko Kasljevic
f685c08b7b Update DataRpcClient and DataRpcServer
Summary: `DataRpcClient` and `DataRpcServer` now work with both old and new record. This will be needed later when I replace `Cache` with `LruCache`.

Reviewers: msantl, teon.banek, ipaljak

Reviewed By: msantl, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1875
2019-02-26 11:18:17 +01:00
Vinko Kasljevic
bbb69a1c00 Edges iterator implementation
Summary:
`EdgesIterable` is used for iterating over Edges in distributed memgraph. Because of lru cache there is a possibility of data getting evicted as someone iterates over it.
To prevent that `EdgesIterable` will lock that data and release it when it's deconstructed.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1868
2019-02-26 09:44:22 +01:00
Teon Banek
4d1d9fb15a Add syntax for managing data constraints
Reviewers: mtomic, mferencevic, buda, msantl

Reviewed By: mtomic, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1879
2019-02-25 14:31:29 +01:00
Matej Ferencevic
7be23896c2 Make auth library case insensitive
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1881
2019-02-22 15:50:51 +01:00
Vinko Kasljevic
3ad5c8069c Add CachedDataLock throughout the code
Summary:
CachedDataLock is necessary for lru cache as remote data is no longer
persistent. Most methods internally handle this, but for methods that
return pointers or references to remote data, we need to manually
lock data.

Reviewers: msantl, ipaljak

Reviewed By: msantl

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1869
2019-02-21 15:27:40 +01:00
Matej Ferencevic
fe361fb750 Implement audit logging
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1863
2019-02-19 15:11:30 +01:00
Teon Banek
6bba5f4cd0 Add clauses for showing DB information
Summary:
Also add STATS Privilege and Permission.
Update tests and changelog.

Reviewers: mtomic, mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1873
2019-02-19 13:32:43 +01:00
Teon Banek
8589dd124d Remove unused variables
Reviewers: mferencevic, msantl, ipaljak, vkasljevic

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1870
2019-02-18 10:45:53 +01:00
Vinko Kasljevic
10d4171348 Remove DistributedAccessor
Reviewers: teon.banek, msantl, ipaljak

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1867
2019-02-15 16:55:40 +01:00
Vinko Kasljevic
a14c4f1864 Existence constraints for vertex for single node
Summary:
Existence constraint ensures that all nodes with certain label have a certain property.
`ExistenceRule` defines label -> properties rule and `ExistenceConstraints` manages all
constraints.

Reviewers: msantl, ipaljak, teon.banek, mferencevic

Reviewed By: msantl, teon.banek, mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1797
2019-02-15 13:14:31 +01:00
Teon Banek
5af47c6df5 Don't inherit TotalOrdering for RecordAccessor
Summary:
RecordAccessor doesn't implement `operator<`, so it doesn't make sense
to have it inherit TotalOrdering.

Reviewers: mferencevic, msantl, vkasljevic

Reviewed By: vkasljevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1866
2019-02-15 11:10:31 +01:00
Matija Santl
ed75e45541 Fix Raft failure discovered in apollo run 479391
Summary:
We noticed a Raft test failure https://apollo.memgraph.io/runs/479391/

This diff should fix it.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1865
2019-02-15 10:22:52 +01:00
Vinko Kasljevic
d1eeaa8de0 Remove Impl from RecordAccessor
Reviewers: msantl, ipaljak, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1859
2019-02-14 17:15:45 +01:00
Matej Ferencevic
19aabaa0d3 Fix parent build in diffs
Reviewers: teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1864
2019-02-14 15:00:02 +01:00
Matej Ferencevic
d9673698e5 Cleanup file utils
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1861
2019-02-14 12:48:37 +01:00
Vinko Kasljevic
3e739f33c9 Add LruCache implementation and tests
Reviewers: msantl, ipaljak, teon.banek

Reviewed By: msantl, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1858
2019-02-14 11:03:33 +01:00
Vinko Kasljevic
127a67ab13 Remove DistributedGraphDb
Summary: DistributedGraphDb is no longer needed.

Reviewers: msantl, teon.banek, ipaljak

Reviewed By: msantl, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1857
2019-02-14 09:38:32 +01:00
Matej Ferencevic
bb052be002 Remove serialization from utils
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1852
2019-02-13 15:41:40 +01:00
Matija Santl
68c910a083 Simplify log compaction
Summary: Teon found this nit so we might fix it.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1860
2019-02-13 15:35:14 +01:00
Teon Banek
cbd6f3bbe2 Move RandomGraphGenerator from utils to test
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1855
2019-02-12 16:49:13 +01:00
Matija Santl
f85095c203 Fix Raft shutdown
Summary:
During the following scenario:
 - start a HA cluster with 3 machines
 - find the leader and start sending queries
 - SIGTERM the leader but leave other 2 machines untouched

The leader would be stuck in the shutdown phase.

This was happening because during the shutdown phase of the Bolt server, a
`graph_db_accessor` would try to commit a transaction after we've already shut
down Raft server.  Raft, although not running, is still thinking it's in the
Leader mode. Tx Engine calls the `SafeToCommit` method to Commit transactions,
and ends up in an infinite loop.

Since Raft was shut down it won't handle any of the incoming RPCs and won't
change it's mode.

The fix here is to shut down the Bolt server before Raft, so we don't have any
pending commits once Raft is shut down.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1853
2019-02-12 15:12:39 +01:00
Matija Santl
1b11e109fa Add storageInfo awesome memgraph function
Summary:
Added a new awesome function called `storageInfo`. This function
returns a list of key-value pairs containing the following (useful?)
information:
 * number of vertices
 * number of edges
 * average degree
 * memory usage
 * disk usage

The current implementation is in `awesome_memgraph_functions` but it will end up
as a separate clause for better access control.

Reviewers: teon.banek, mtomic, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1850
2019-02-11 16:25:01 +01:00
Matej Ferencevic
773acc11d6 Remove stats from main binaries
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1851
2019-02-11 14:33:41 +01:00
Teon Banek
53d8f725c5 Build mg-comm-rpc library
Summary:
This change splits mg-communication into mg-communication and
mg-comm-rpc. The main reason for doing this, is to make separation of
enterprise features from community Memgraph more clear.

Reviewers: mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1844
2019-02-07 14:40:15 +01:00
Teon Banek
71360cba2a Extract io serialization to a library
Summary: Serialization is only needed in distributed Memgraph

Reviewers: mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1846
2019-02-07 12:46:35 +01:00
Ivan Paljak
e3cf4d0df8 Make an in-memory copy of HA persistent storage pt. 1
Summary:
In Raft, we often need to access persistent state of the server
without modifying it. In order to speed up such operations, we
keep an in-memory copy of that state.

In this diff we make a copy of all persistent state except for
the log itself. Running our feature benchmark locally, we manage
to increase the throughput for cca 750 queries/s.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1843
2019-02-06 15:25:57 +01:00
Teon Banek
e461a08340 Move creating indexed lookup to a rewrite pass
Summary:
RuleBasedPlanner now generates only the regular ScanAll operations, and
Filter operations are appended as soon as possible. The newly added
Rewrite step, takes this operator tree and replaces viable Filter &
ScanAll operators with appropriate ScanAllBy<Index> operator. This
change ought to simplify the behaviour of DistributedPlanner when that
stage is moved before the indexed lookup rewrite.

Showing unoptimized plan in interactive planner is also supported.

Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1839
2019-02-06 14:46:19 +01:00
Teon Banek
8a508a1cb6 Remove unused helper closure variable
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1845
2019-02-06 14:06:52 +01:00
Marin Tomic
c03ca5f8f0 Remove UID tracking from AstStorage
Summary:
All AST nodes had a member `uid_` that was used as a key in
`SymbolTable`. It is renamed to `symbol_pos_` and it appears only in
`Identifier`, `NamedExpression` and `Aggregation`, since only those types were
used in `SymbolTable`. SymbolGenerator is now responsible for creating symbols
in `SymbolTable` and assigning positions to AST nodes.

Cloning and serialization code is now simpler since there is no need to track
UIDs.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1836
2019-02-05 14:19:48 +01:00
Matija Santl
145c81376f Add log compaction for Raft, pt. 2
Summary: Implemented snapshot replication and log compaction.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1840
2019-02-04 15:32:07 +01:00
Matija Santl
da95cbf4ec Add log compaction for Raft, pt. 1
Summary:
In this part of log compaction for raft, I've implemented snapshooting
and snapshot recovery. I've also refactored the code a bit, so `RaftServer` now
has a pointer to the `GraphDb` and it can do some things by itself.

Log compaction requires some further work. Since snapshooting isn't synchronous
between peers, and each peer can work at their own pace, once we've compacted
the log so that the next log to be sent to peer `x` isn't available anymore, we
need to send the snapshot over the wire. This means that the next part will
contain the `InstallSnapshotRPC` and then maybe one more that will implement the
logic of sending `LogEntry` or the whole snapshot.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1834
2019-01-29 14:58:57 +01:00
Marin Tomic
7542f5b0ba Include JSON serialized logical plan in summary for EXPLAIN
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1831
2019-01-25 17:01:17 +01:00
Teon Banek
a871212675 Replace dynamic_casts in query with utils::Downcast
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1830
2019-01-25 15:31:14 +01:00
Teon Banek
f91428f23f Generate virtual GetTypeInfo with LCP
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1827
2019-01-25 14:12:50 +01:00
Matija Santl
4fa44c3edd Fix Raft's ReplicationLog
Summary:
`ReplicationLog` had a classic off-by-one bug. The `valid_prefix`
variable wasn't set properly.

This diff also includes a poor man's version of a HA client. This client
assumes that all the HA instances run on a single machine and that the
corresponding Bold endpoints have open ports ranging from `7687` to
`7687 + num_machines - 1`.

This should make it easeir to test certain things, ie. disk usage, P25.

This test revealed the bug with `ReplicationLog`

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1813
2019-01-23 16:27:51 +01:00
Teon Banek
743d82b78d Use plan rewrite hook to generate DistributedPlan
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1826
2019-01-23 10:48:03 +01:00
Teon Banek
abc9f0b0e7 Add hooks for post processing generated plans
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1824
2019-01-22 16:45:05 +01:00
Marin Tomic
9d36f775f2 Remove GraphView from variable expansion operators
Summary:
Variable expansions cannot appear in merge patterns or after updates,
so they can only be planned with GraphState::OLD. Because of that, it makes
sense to remove GraphView parameter from them to reduce confusion.

Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1825
2019-01-22 15:35:05 +01:00
Marin Tomic
914f40411a Implement plan cloning with LCP
Summary:
Implement proper plan cloning using LCP instead of hacking it with
serialization.

depends on D1815

Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1816
2019-01-22 14:52:06 +01:00
Matija Santl
62e06d4b70 Fix re-election in Raft
Summary:
Once a leader loses it's leadership, in order to handle hanging
transactions, we reset the storage and the transaction engine.

This requires to re-apply all the commited entries from the log.

Once we add snapshot (log compaction) we would need to do that also.

One thing to have in mind is the `election_timeout_min` parameter. If it's set
too low it could trigger leader re-election too often.

Reviewers: ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1822
2019-01-22 14:51:24 +01:00
Marin Tomic
f9c63987c6 Clone AST using LCP
Summary:
use newly added LCP functionality to get rid of manually written
`Clone` functions in AST.

depends on D1808

Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1815
2019-01-18 13:24:40 +01:00