Commit Graph

1805 Commits

Author SHA1 Message Date
Dominik Gleich
c294127065 Auto generate vertex/edge ids
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1406
2018-05-25 16:03:12 +02:00
Dominik Gleich
adda7d1200 Remote edge with gid creation
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1402
2018-05-25 09:57:55 +02:00
Dominik Gleich
63f90a453d Expose ID
Reviewers: buda, teon.banek

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1400
2018-05-23 15:04:50 +02:00
Dominik Gleich
2c49487eae Fix incorrent namespace
Summary:
It seems that streamoff is not in std::iostream namespace, but in std::
namespace. This caused problems with newer versions of libraries.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1404
2018-05-23 11:03:03 +02:00
Dominik Gleich
7d161319f0 Remote vertex with gid creation
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1401
2018-05-23 10:10:14 +02:00
Matej Ferencevic
639e68cf1d Increase init timeout
Reviewers: teon.banek, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1397
2018-05-18 14:32:41 +02:00
Marko Budiselic
68755edc5c Fix compile warning (order of database::Config members)
Reviewers: msantl, teon.banek, ipaljak

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1396
2018-05-16 14:57:43 +02:00
Matej Ferencevic
cf648a8453 Optimize Apollo build process
Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1394
2018-05-16 14:52:27 +02:00
Matija Santl
f872c93ad1 Add command id to remote produce
Summary:
Command id is necessary in remote produce to identify an ongoing pull
because a transaction can have multiple commands that all belong under
the same plan and tx id.

Reviewers: teon.banek, mtomic, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1386
2018-05-16 10:20:39 +02:00
Marin Tomic
91e38f6413 Distributed BFS
Summary: depends on D1387

Reviewers: msantl, teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1341
2018-05-15 17:38:51 +02:00
Matej Ferencevic
94ec0b23cd Fix distributed apollo runs
Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1393
2018-05-15 15:13:44 +02:00
Marin Tomic
0a6b8cdf4f Remove AS_IS from GraphView
Summary:
Removing `AS_IS` from GraphView because it doesn't seem like it is necessary for query execution and it also has weird semantics (you might get a mix of old and new records). `Unwind`, `OrderBy` and `PullRemoteOrderBy` now use `OLD` graph view.

Remove AS_IS from GraphView

Fix query_cost_estimator tests

Fix query_expression_evaluator tests

Fix query_plan_match_filter_return tests

Fix query_plan_create_set_remove_delete tests

Fix query_plan_accumulate_aggregate tests

Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1390
2018-05-15 13:10:19 +02:00
Ivan Paljak
909e42d414 Add initial version of properties on disk
Summary:
A simplified end-to-end implementation

POD interface set-up, still have bugs with HDDkeys

Version bug fix and first iterator implementation

Fixed out-of-scope reference in PVS iterator

Added PVS unit tests

Reviewers: buda, mferencevic, dgleich, teon.banek

Reviewed By: buda, dgleich

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1369
2018-05-10 17:58:38 +02:00
Dominik Gleich
797bd9e435 Notify master of worker recovery
Summary: Notify master when workers finish recovering such that master doesn't start doing stuff before they recovered

Reviewers: buda, msantl, mferencevic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1389
2018-05-10 11:09:34 +02:00
Teon Banek
30506f44f5 Use LCP to generate LogicalOperator boost serialization
Summary:
Since we are moving from boost to Capnp for serialization, it makes
sense to keep all of the LogicalOperator classes in LCP format. This
will make it easier to generate Capnp code.

Depends on D1361

Reviewers: buda, mferencevic, msantl, dgleich, ipaljak, mculinovic, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1362
2018-05-10 09:56:40 +02:00
Marin Tomic
632663dca5 Fix address handling in remote edge creation
Summary: Global address of `from` was stored in Edge record when created via CreateEdge RPC.

Reviewers: buda, msantl, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1387
2018-05-07 13:20:51 +02:00
Marko Culinovic
49d69560e6 Serialize cpp types with capnp
Summary: Serialize shared_ptr, unique_ptr, optional and vector.

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1364
2018-05-04 15:02:38 +02:00
Marko Culinovic
4b9970ccf8 Save and load pointers in ast capnp serialization
Summary:
Pointers in AstTreeStorage can point to same nodes. These nodes
should be serialized(saved) only once when serializing Ast tree.
Same goes for deserializing(loading).
When deserializing nodes, one should not use storage Create method,
because it will add node which isn't completely loaded to storage vector.
Therefore, one should use new Deserialize method which doesn't
add node to storage vector. Inserting node into storage vector
is defered until all node data is loaded.

Pass pointer to saved uids set, instead of storage

Delete unused set

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1344
2018-05-04 09:50:34 +02:00
Marko Culinovic
8795501890 Serialize ast using capnproto
Reviewers: teon.banek, florijan, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1306
2018-05-03 14:12:38 +02:00
Teon Banek
ab574bf84f Benchmark Cap'n Proto and boost serialization
Reviewers: florijan, mferencevic, buda, mculinovic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1273
2018-05-02 12:38:18 +02:00
Teon Banek
fc6173b4e2 Disable every extra library in rocksdb
Reviewers: buda, dgleich

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1385
2018-04-30 16:26:33 +02:00
Teon Banek
c10773522b Add Lisp C++ Preprocessing (LCP)
Summary:
In order to enhance C++ metaprogramming capabilities, a custom
preprocessing step is added before compilation. C++ code may be mixed
with Lisp code in order to generate a complete C++ source code. The
mechanism is hooked into cmake. To notify cmake of .lcp files, `add_lcp`
function in src/CMakeLists.txt needs to be invoked.

The main executable entry point is in tools/lcp, while the source code
is in src/lisp/lcp.lisp

The main goal of LCP is to auto generate class serialization code and
member variable getter functions. This should now be significantly less
error prone, since you cannot forget to serialize a member variable
through this mechanism. Future uses should be generating other repeating
code, such as `Clone` methods or perhaps some debug information.

.lcp files may contain mixed C++ code (enclosed in #>cpp ... cpp<#
blocks) with Common Lisp code.

NOTE: With great power comes great responsibility. Lisp metaprogramming
capabilities are incredibly powerful. To keep the sanity of the team
intact, use Lisp preprocessing only when *really* necessary.

Reviewers: buda, mferencevic, msantl, dgleich, ipaljak, mculinovic, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1361
2018-04-30 14:36:30 +02:00
Teon Banek
6234075983 Compile utils/file.cpp in kvstore_lib
Reviewers: buda, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1384
2018-04-30 14:34:38 +02:00
Teon Banek
43d5980ff3 Disable lz4, zstd and snappy libraries for rocksdb
Reviewers: buda, dgleich, msantl

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1383
2018-04-30 12:56:12 +02:00
Matija Santl
96c413fe45 Use git commit hashed instead of tags
Summary: git tag based checkout probably doesn't work with git 2.11.0

Reviewers: teon.banek, buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1382
2018-04-30 10:40:45 +02:00
Dominik Gleich
518f83aaa3 Optimize gc clog rpc calls
Summary:
GC called clog info on transactions which were not completed even though it
could deduce that they are not finished. By avoiding calling clog info on them
we can lower the number of rpc calls and remove a strange behaviour of gc
appearing to be hanging while recovering a worker.

Reviewers: mculinovic, mferencevic, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1377
2018-04-30 10:15:55 +02:00
Marko Budiselic
7d94878860 Add rocksdb as a lib + initial implementation of the KVStore
Reviewers: teon.banek, mferencevic, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1331
2018-04-28 13:19:09 +02:00
Matej Ferencevic
df11e8fd2c Fix Debian package generation
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1376
2018-04-24 16:49:16 +02:00
Matija Santl
609f33b086 Fix wsp docs
Summary: s/a/an/

Reviewers: ipaljak

Reviewed By: ipaljak

Differential Revision: https://phabricator.memgraph.io/D1373
2018-04-23 16:01:32 +02:00
Teon Banek
939056eac7 Add single node memgraph binary
Summary:
This binary is installed and packaged for release. This is just a quick
solution for releasing the Community 0.10 version. We still need to
setup the installation and packaging for both the Enterprise and
Community versions. Additionally, the automated build system needs to
test both binaries for correct behaviour. Obviously, some tests can only
be run on one of the 2 versions.

Reviewers: mferencevic, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1363
2018-04-23 14:05:23 +02:00
Marko Budiselic
c76170a9db Clean utils folder (namespaces, function names)
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1359
2018-04-22 09:44:32 +02:00
Marko Budiselic
44474b55e3 Add durability stress test
Reviewers: dgleich, mferencevic

Reviewed By: dgleich, mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1371
2018-04-22 08:32:44 +02:00
Dominik Gleich
261d50a02e Destroy and create storage object after failed recovery
Reviewers: buda

Reviewed By: buda

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1370
2018-04-20 16:16:52 +02:00
Matija Santl
5c7d3a908f Add two state dijkstra for wsp implementation
Summary:
Added a new markdown file for concepts and a new test to test the edge
case with upper bound.

Reviewers: teon.banek, mtomic, dgleich, buda, ipaljak

Reviewed By: teon.banek, dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1366
2018-04-20 16:10:42 +02:00
Dominik Gleich
7af80ebb8d Replace command_id_t with CommandId and transaction_id_t with TransactionId.
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1367
2018-04-20 13:55:14 +02:00
Dominik Gleich
debe5a961c Mention clog cleanup in changelog
Reviewers: buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1368
2018-04-20 11:07:23 +02:00
Dominik Gleich
3ecf839198 Fix non-atomic set of transaction owner
Summary: Remove static unlocked variable

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1365
2018-04-18 10:24:36 +02:00
Matej Ferencevic
c9d2ad845c Increase pip timeout
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1360
2018-04-17 12:58:28 +02:00
Dominik Gleich
dba81f223c Ensure workers recover appropriate snapshot
Reviewers: msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1355
2018-04-16 16:58:54 +02:00
Dominik Gleich
e519a64c7c Fix wal retention test
Summary:
Wal retention test was flaky, this kinda fixes it.
It's hard to fix it completely since there are separate threads happening which
are not synchronized, and forcing the synchronization from the outside would
not agree to the specifications i.e. single-threadness of WAL.

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1358
2018-04-16 15:16:04 +02:00
Ivan Paljak
92613fcab8 Football demo tck tests and tech docs
Summary: Heading fix in examples.md

Reviewers: teon.banek, buda, mtomic

Reviewed By: teon.banek, buda, mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1290
2018-04-16 12:06:32 +02:00
Teon Banek
6e849d212a Document the toolchain bootstrap process on CentOS
Reviewers: mferencevic, buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1356
2018-04-13 12:49:35 +02:00
Dominik Gleich
f26a529a5b Fix transactions releasing locks
Summary: Locks should be released as early as possible

Reviewers: msantl, florijan

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1357
2018-04-12 15:33:40 +02:00
Teon Banek
79b9c0a647 Update user technical documentation
Summary: Document durability and RPM installation in user docs

Reviewers: buda, mtomic, ipaljak

Reviewed By: buda, mtomic

Differential Revision: https://phabricator.memgraph.io/D1345
2018-04-12 13:08:34 +02:00
Teon Banek
eff843bbcc Tidy up the RPM packaging
Summary:
Use /var/lib/memgraph as home dir for memgraph user.
Merge post and pre RPM scripts in RPM spec file.
Properly handle memgraph.service and directory permissions.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1350
2018-04-12 10:31:38 +02:00
Dominik Gleich
87dbe26038 Check if connection alive & fix pure virtual bug
Reviewers: mferencevic, msantl, florijan

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1353
2018-04-12 10:17:00 +02:00
Marko Budiselic
20d667f8bf Clean utils folder
Summary: The goal is to standardize utils folder.

Reviewers: teon.banek, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1354
2018-04-11 16:19:46 +02:00
Matija Santl
4cbfc800b8 Check that workers desired id is equal to the assigned id from master.
Summary:
Returns -1 from coordinations `AddWorker` method and propagate it to
worker if master can't assign the desired worker id.

Reviewers: dgleich, florijan

Reviewed By: dgleich

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1352
2018-04-11 12:36:40 +02:00
florijan
a01c26439b Clean SkipList up
Summary:
- Removed a lot of stuff that was incorrect and/or unnecessary
- Fixed const-correctness in the skiplist family

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1351
2018-04-11 10:08:20 +02:00
florijan
f91aa8f49f Document distributed, durability and storage concepts
Summary:
Adding all you guys as reviewers. Feel free to take a peek and comment
on whatever is not clear. Where needed the documentation can be extended
or concepts explained in-person.

Reviewers: buda, teon.banek, dgleich, ipaljak, msantl, mculinovic, mtomic

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1322
2018-04-10 12:50:32 +02:00