Commit Graph

1940 Commits

Author SHA1 Message Date
Marin Tomic
632663dca5 Fix address handling in remote edge creation
Summary: Global address of `from` was stored in Edge record when created via CreateEdge RPC.

Reviewers: buda, msantl, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1387
2018-05-07 13:20:51 +02:00
Marko Culinovic
49d69560e6 Serialize cpp types with capnp
Summary: Serialize shared_ptr, unique_ptr, optional and vector.

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1364
2018-05-04 15:02:38 +02:00
Marko Culinovic
4b9970ccf8 Save and load pointers in ast capnp serialization
Summary:
Pointers in AstTreeStorage can point to same nodes. These nodes
should be serialized(saved) only once when serializing Ast tree.
Same goes for deserializing(loading).
When deserializing nodes, one should not use storage Create method,
because it will add node which isn't completely loaded to storage vector.
Therefore, one should use new Deserialize method which doesn't
add node to storage vector. Inserting node into storage vector
is defered until all node data is loaded.

Pass pointer to saved uids set, instead of storage

Delete unused set

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1344
2018-05-04 09:50:34 +02:00
Marko Culinovic
8795501890 Serialize ast using capnproto
Reviewers: teon.banek, florijan, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1306
2018-05-03 14:12:38 +02:00
Teon Banek
ab574bf84f Benchmark Cap'n Proto and boost serialization
Reviewers: florijan, mferencevic, buda, mculinovic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1273
2018-05-02 12:38:18 +02:00
Teon Banek
fc6173b4e2 Disable every extra library in rocksdb
Reviewers: buda, dgleich

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1385
2018-04-30 16:26:33 +02:00
Teon Banek
c10773522b Add Lisp C++ Preprocessing (LCP)
Summary:
In order to enhance C++ metaprogramming capabilities, a custom
preprocessing step is added before compilation. C++ code may be mixed
with Lisp code in order to generate a complete C++ source code. The
mechanism is hooked into cmake. To notify cmake of .lcp files, `add_lcp`
function in src/CMakeLists.txt needs to be invoked.

The main executable entry point is in tools/lcp, while the source code
is in src/lisp/lcp.lisp

The main goal of LCP is to auto generate class serialization code and
member variable getter functions. This should now be significantly less
error prone, since you cannot forget to serialize a member variable
through this mechanism. Future uses should be generating other repeating
code, such as `Clone` methods or perhaps some debug information.

.lcp files may contain mixed C++ code (enclosed in #>cpp ... cpp<#
blocks) with Common Lisp code.

NOTE: With great power comes great responsibility. Lisp metaprogramming
capabilities are incredibly powerful. To keep the sanity of the team
intact, use Lisp preprocessing only when *really* necessary.

Reviewers: buda, mferencevic, msantl, dgleich, ipaljak, mculinovic, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1361
2018-04-30 14:36:30 +02:00
Teon Banek
6234075983 Compile utils/file.cpp in kvstore_lib
Reviewers: buda, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1384
2018-04-30 14:34:38 +02:00
Teon Banek
43d5980ff3 Disable lz4, zstd and snappy libraries for rocksdb
Reviewers: buda, dgleich, msantl

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1383
2018-04-30 12:56:12 +02:00
Matija Santl
96c413fe45 Use git commit hashed instead of tags
Summary: git tag based checkout probably doesn't work with git 2.11.0

Reviewers: teon.banek, buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1382
2018-04-30 10:40:45 +02:00
Dominik Gleich
518f83aaa3 Optimize gc clog rpc calls
Summary:
GC called clog info on transactions which were not completed even though it
could deduce that they are not finished. By avoiding calling clog info on them
we can lower the number of rpc calls and remove a strange behaviour of gc
appearing to be hanging while recovering a worker.

Reviewers: mculinovic, mferencevic, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1377
2018-04-30 10:15:55 +02:00
Marko Budiselic
7d94878860 Add rocksdb as a lib + initial implementation of the KVStore
Reviewers: teon.banek, mferencevic, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1331
2018-04-28 13:19:09 +02:00
Matej Ferencevic
df11e8fd2c Fix Debian package generation
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1376
2018-04-24 16:49:16 +02:00
Matija Santl
609f33b086 Fix wsp docs
Summary: s/a/an/

Reviewers: ipaljak

Reviewed By: ipaljak

Differential Revision: https://phabricator.memgraph.io/D1373
2018-04-23 16:01:32 +02:00
Teon Banek
939056eac7 Add single node memgraph binary
Summary:
This binary is installed and packaged for release. This is just a quick
solution for releasing the Community 0.10 version. We still need to
setup the installation and packaging for both the Enterprise and
Community versions. Additionally, the automated build system needs to
test both binaries for correct behaviour. Obviously, some tests can only
be run on one of the 2 versions.

Reviewers: mferencevic, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1363
2018-04-23 14:05:23 +02:00
Marko Budiselic
c76170a9db Clean utils folder (namespaces, function names)
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1359
2018-04-22 09:44:32 +02:00
Marko Budiselic
44474b55e3 Add durability stress test
Reviewers: dgleich, mferencevic

Reviewed By: dgleich, mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1371
2018-04-22 08:32:44 +02:00
Dominik Gleich
261d50a02e Destroy and create storage object after failed recovery
Reviewers: buda

Reviewed By: buda

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1370
2018-04-20 16:16:52 +02:00
Matija Santl
5c7d3a908f Add two state dijkstra for wsp implementation
Summary:
Added a new markdown file for concepts and a new test to test the edge
case with upper bound.

Reviewers: teon.banek, mtomic, dgleich, buda, ipaljak

Reviewed By: teon.banek, dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1366
2018-04-20 16:10:42 +02:00
Dominik Gleich
7af80ebb8d Replace command_id_t with CommandId and transaction_id_t with TransactionId.
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1367
2018-04-20 13:55:14 +02:00
Dominik Gleich
debe5a961c Mention clog cleanup in changelog
Reviewers: buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1368
2018-04-20 11:07:23 +02:00
Dominik Gleich
3ecf839198 Fix non-atomic set of transaction owner
Summary: Remove static unlocked variable

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1365
2018-04-18 10:24:36 +02:00
Matej Ferencevic
c9d2ad845c Increase pip timeout
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1360
2018-04-17 12:58:28 +02:00
Dominik Gleich
dba81f223c Ensure workers recover appropriate snapshot
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1355
2018-04-16 16:58:54 +02:00
Dominik Gleich
e519a64c7c Fix wal retention test
Summary:
Wal retention test was flaky, this kinda fixes it.
It's hard to fix it completely since there are separate threads happening which
are not synchronized, and forcing the synchronization from the outside would
not agree to the specifications i.e. single-threadness of WAL.

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1358
2018-04-16 15:16:04 +02:00
Ivan Paljak
92613fcab8 Football demo tck tests and tech docs
Summary: Heading fix in examples.md

Reviewers: teon.banek, buda, mtomic

Reviewed By: teon.banek, buda, mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1290
2018-04-16 12:06:32 +02:00
Teon Banek
6e849d212a Document the toolchain bootstrap process on CentOS
Reviewers: mferencevic, buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1356
2018-04-13 12:49:35 +02:00
Dominik Gleich
f26a529a5b Fix transactions releasing locks
Summary: Locks should be released as early as possible

Reviewers: msantl, florijan

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1357
2018-04-12 15:33:40 +02:00
Teon Banek
79b9c0a647 Update user technical documentation
Summary: Document durability and RPM installation in user docs

Reviewers: buda, mtomic, ipaljak

Reviewed By: buda, mtomic

Differential Revision: https://phabricator.memgraph.io/D1345
2018-04-12 13:08:34 +02:00
Teon Banek
eff843bbcc Tidy up the RPM packaging
Summary:
Use /var/lib/memgraph as home dir for memgraph user.
Merge post and pre RPM scripts in RPM spec file.
Properly handle memgraph.service and directory permissions.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1350
2018-04-12 10:31:38 +02:00
Dominik Gleich
87dbe26038 Check if connection alive & fix pure virtual bug
Reviewers: mferencevic, msantl, florijan

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1353
2018-04-12 10:17:00 +02:00
Marko Budiselic
20d667f8bf Clean utils folder
Summary: The goal is to standardize utils folder.

Reviewers: teon.banek, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1354
2018-04-11 16:19:46 +02:00
Matija Santl
4cbfc800b8 Check that workers desired id is equal to the assigned id from master.
Summary:
Returns -1 from coordinations `AddWorker` method and propagate it to
worker if master can't assign the desired worker id.

Reviewers: dgleich, florijan

Reviewed By: dgleich

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1352
2018-04-11 12:36:40 +02:00
florijan
a01c26439b Clean SkipList up
Summary:
- Removed a lot of stuff that was incorrect and/or unnecessary
- Fixed const-correctness in the skiplist family

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1351
2018-04-11 10:08:20 +02:00
florijan
f91aa8f49f Document distributed, durability and storage concepts
Summary:
Adding all you guys as reviewers. Feel free to take a peek and comment
on whatever is not clear. Where needed the documentation can be extended
or concepts explained in-person.

Reviewers: buda, teon.banek, dgleich, ipaljak, msantl, mculinovic, mtomic

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1322
2018-04-10 12:50:32 +02:00
Dominik Gleich
43ce227eec DCHECK/CHECK
Summary: Debug checks should be checks because otherwise tests in release fail

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1348
2018-04-10 10:05:55 +02:00
florijan
1edeb655a6 Return PropertyValue from store by value
Summary: Do it so properties-on-disk can be implemented.

Reviewers: buda, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1343
2018-04-10 09:33:00 +02:00
Teon Banek
1c83b840f0 Bump memgraph version to 0.10
Reviewers: buda, mferencevic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1346
2018-04-10 08:57:13 +02:00
Matija Santl
753d57eb80 Fix concurent file creation
Summary:
If someone was to call `EnsureDir` concurrently it might fail because
the `fs::create_directories` returns false when the directory already exists
with the `error_code` value being `0`.

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1349
2018-04-09 17:13:55 +02:00
Dominik Gleich
29234067d9 Call engine tx cache cleanup
Summary:
After the commit log was cleared there was some transaction that tried to acquire
a lock on an object that was taken by a transaction that was not longer active on
the worker. Inquring the commit log about that transaction causes a crash since
the commit log is cleared of that transaction.

Solution is to clear the transaction cache before clearing the commit log, which
forces the transactions to release their locks and as such their ids will never be
queried through the commit log in the future.

Reviewers: florijan, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1342
2018-04-09 16:02:38 +02:00
Matija Santl
7b88e514b8 Add ClusterDiscovery RPC for distributed BFS
Summary:
Implemented cluster discovery in distributed memgraph.
When a worker registers, it sends a RPC request to master.
The master assigns that worker an id and sends the information about other
workers (pairs of <worker_id, endpoint>) to the new worker.
Master also sends the information about the new worker to all existing workers
in the process of worker registration.

After the last worker registers, all memgraph instances in the clusters should
know about every other.

Reviewers: mtomic, buda, florijan

Reviewed By: mtomic

Subscribers: teon.banek, dgleich, pullbot

Differential Revision: https://phabricator.memgraph.io/D1339
2018-04-09 14:28:22 +02:00
Matej Ferencevic
71565287b8 Remove obsolete network buffers
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1336
2018-04-06 16:26:46 +02:00
Dominik Gleich
b20e31e800 Synchronize snapshooting
Summary: Make synchronized snapshot. This invokese the snapshooter on workers on the master snapshot scheduler interval.

Reviewers: msantl, mtomic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1334
2018-04-06 10:10:01 +02:00
Dominik Gleich
23fe66e828 Update tx_id to 0
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1340
2018-04-06 09:59:37 +02:00
Dominik Gleich
a36753cf27 Update snapshot naming
Summary:
Snapshots should have the transaction from which they were created because we need this info for recovery later on.
Otherwise we wouldn't be able to tell the workers from which snapshots to recover. The whole cluster should be recovered
from the same transaction snapshot.

Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1338
2018-04-05 13:42:51 +02:00
Marin Tomic
5f53a7a759 Add tuple serialization
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1337
2018-04-05 10:58:59 +02:00
Dominik Gleich
eb30ecb6a0 Commit log gc
Summary: Adds a commit log garbage collector, which clears old transactions from the commit log

Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1310
2018-04-04 10:25:25 +02:00
Matija Santl
0bcf2edeae Two phase commit on cursor destruction
Summary:
When commiting/aborting a transaction in tx master engine, make a two
phase commit to all workers so they can stop all futures and clear
transactional cache.

Reviewers: dgleich, florijan

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1320
2018-04-03 16:20:00 +02:00
Matej Ferencevic
485e1988c3 Split signals utils into cpp and hpp
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1335
2018-04-03 11:17:11 +02:00
Dominik Gleich
3741c5b45a Fix memgraph crashing on exit
Summary:
Snapshot scheduler object was released from unique ptr and not actually freed, which
caused the snapshooter to access the tx_engine after it was destructed.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1325
2018-03-30 13:51:51 +02:00