Summary:
During it's leadership, one peer can receive RPC messages from other peers that his reign is over.
The problem is when this happens during a transaction commit.
This is handled in the following way.
If we're the current leader and we want to commit a transaction, we need to make sure the Raft Log is replicated before we can tell the client that the transaction is committed.
During that wait, we can only notice that the replication takes too long, and we report that with `LOG(WARNING)` messages.
If we change the Raft mode during the wait, our Raft implementation will internally commit this transaction, but won't be able to acquire the Raft lock because the `db.Reset` has been called.
This is why there is an manual lock acquire. If we pick up that the `db.Reset` has been called, we throw an `UnexpectedLeaderChangeException` exception to the client.
Another thing with long running transactions, if someone decides to kill a `memgraph_ha` instance during the commit, the transaction will have `abort` hint set. This will cause the `src/query/operator.cpp` to throw a `HintedAbortError`. We need to catch this during the shutdown, because the `memgraph_ha` isn't dead from the user perspective, and the transaction wasn't aborted because it took too long, but we can differentiate between those two.
Reviewers: mferencevic, ipaljak
Reviewed By: mferencevic, ipaljak
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1956
Summary:
Micro benchmarks show no change compared to global new & delete. This is
to be expected, because Unwind relies only on `std::vector` which ought
to reserve the memory in reasonable chunks.
Reviewers: mtomic, llugovic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2064
Summary:
Micro benchmarks show an improvement to performance of about 10%
compared to global new & delete.
Reviewers: mtomic, llugovic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2061
Summary: We should now exchange around O(log(n)) messages to get a follower that is n transactions behind back in sync.
Reviewers: msantl, mferencevic
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2051
Summary:
I was a bit stupid and thought I can use `std::swap`, while `std::swap`
is implemented in terms of move itself.
Reviewers: mtomic, mferencevic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2054
Summary:
A `valgrind` run of a modified client that connects multiple times to the
database now shows no memory leaks, previously the context created with
`SSL_CTX_new` was leaked every time.
Reviewers: teon.banek, mtomic
Reviewed By: teon.banek, mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2052
Summary:
Micro benchmarks show that MonotonicBufferResource improves performance
by a factor of 1.5.
Reviewers: mtomic, mferencevic, llugovic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2048
Summary:
Benchmarks show minor improvements. Perhaps it makes sense at some later date
to use another allocator for things lasting only in a single `Pull`.
Reviewers: mferencevic, mtomic, llugovic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2018
Summary:
Unfortunately, the written micro benchmark only reports minor
improvements compared to default allocator. The results are in some
cases even a tiny bit worse.
Reviewers: mtomic, mferencevic, llugovic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2039
Summary:
This change introduces dumping of indices keys. During the dump process,
an internal label is assigned to each vertex and index on vertex's
internal property id is created for faster matching during edge creation.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: msantl, pullbot
Differential Revision: https://phabricator.memgraph.io/D2046
Summary:
Prior to this change, a huge query was returned by DumpGenerator that
dumped the entire graph. This change split the single query to multiple
queries, each dumping a single vertex/edge. For easier vertex matching
when dumping edge, an internal property id is assigned to each vertex and
removed after the whole graph is dumped.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2038
Summary:
Rocksdb cmake configuration uses `GNUInstallDirs`, which on some
platforms sets `CMAKE_INSTALL_LIBDIR` to `lib64`. We have hardcoded the
path to the library inside `lib`. Since this is a local installation, it
doesn't really matter, so it should be OK to always install rocksdb
under `lib`.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2042
Summary:
# Summary
## Concepts and terminology
A **namestring for <object>** is a Lisp string that names the C++ language element
`<object>` (such as a namespace, a variable, a class, etc.). Therefore we have a
"namestring for a namespace", a "namestring for a variable", etc.
A **designator for a namestring for <object>** is a Lisp object that denotes a
"namestring for <object>". These are symbols and strings themselves.
A **typestring** is a Lisp string that represents a C++ type, i.e. its
declaration.
A **typestring designator** is a Lisp object that denotes a typestring.
A typestring (and the corresponding C++ type) is said to be **supported** if it
can be parsed using `parse-cpp-type-declaration`. This concept is important and
should form the base of our design because we can't really hope to ever support
all of C++'s type declarations.
A typestring (and the corresponding C++ type) that is not supported is
**unsupported**.
A **processed typestring** is a typestring that is either fully qualified or
unqualified.
A C++ type is said to be **known** if LCP knows extra information about the type,
rather than just what kind of type it is, in which namespace it lives, etc. For
now, the only known types are those that are defined within LCP itself using
`define-class` & co.
A C++ type is **unknown** if it is not known.
**Typestring resolution** is the process of resolving a (processed) typestring
into an instance of `cpp-type` or `unsupported-cpp-type`.
**Resolving accessors** are accessors which perform typestring resolution.
## Changes
Explicitly introduce the concept of supported and known types. `cpp-type` models
supported types while `unsupported-cpp-type` models unsupported types.
Subclasses of `cpp-type` model known types. `general-cpp-type` is either a
`cpp-type` or an `unsupported-cpp-type`.
Add various type queries.
Fix `define-rpc`'s `:initarg` (remove it from the `cpp-member` struct).
Introduce namestrings and their designators in `names.lisp`.
Introduce typestrings and their designators.
Introduce **processed typestrings**. Our DSL's macros (`define-class` & co.)
convert all of the given typestrings into processed typestrings because we don't
attempt to support partially qualified names and relative name lookup yet. A
warning is signalled when a partially qualified name is treated as a fully
qualified name.
The slots of `cpp-type`, `cpp-class`, `cpp-member` and `cpp-capnp-opts` now
store processed typestrings which are lazily resolved into their corresponding
C++ types.
The only thing that instances of `unsupported-cpp-type` are good for is getting
the typestring that was used to construct them. Most of LCP's functions only
work with known C++ types, i.e. `cpp-type` instances. The only function so far
that works for both of them is `cpp-type-decl`.
Since "unsupportedness" is now explicitly part of LCP, client code is expected
to manually check whether a returned type is unsupported or not (or face
receiving an error otherwise), unless a function is documented to return only
`cpp-type` instances.
A similar thing goes for "knowness". Client code is expected to manually check
whether a returned type is known or not, unless a function is documented to
return only (instances of `cpp-type` subclasses) known types.
## TODO
Resolution still has to be done for the following slots of the
following structures:
- `slk-opts`
- `save-args`
- `load-args`
- `clone-opts`
- `args`
- `return-type`
Reviewers: teon.banek, mtomic
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1962
Summary:
This should hopefully resolve multithreading issues when multiple LCP
invocations tried to process the same file.
Reviewers: mferencevic, msantl
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2029
Summary:
Read throughput dropped by about 50% from the last diff.
This should fix it (at least according to local measurements).
Reviewers: msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2027
Summary:
According to the written benchmark, using MonotonicBufferResource yields
significant improvements to performance of Distinct. The setup fills the
database with vertices depending on the benchmark state. No edges are
created. Then we run DISTINCT on that. Since each vertex is unique, we
will store everything in the `DistinctCursor::seen_rows_`, which is
backed by a MemoryResource. This setup, on my machine, yields 10 times
better performance when run with MonotonicBufferResource.
Reviewers: mferencevic, mtomic, msantl
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1894
Summary:
This allows us to decouple issuing heartbeats from the server mode
which is useful when we know we will transition to LEADER but cannot yet
change the mode due to some Raft internals.
It can now happen that a couple of HBs are sent when in FOLLOWER or CANDIDATE
mode, but this doesn't affect the correctness of the protocol.
Reviewers: msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2025
Summary:
There will be a lot of leftover files, execute the following commands inside
`src/` to remove them:
```
git clean -xf
rm -r rpc/ storage/single_node_ha/rpc/
```
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2011
Summary:
Clang-analyzer-core found NullDereference issue in LruList. I don't see
how that can occur, but as a precaution I'll add a CHECK.
Reviewers: msantl, ipaljak
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D2015
