Commit Graph

944 Commits

Author SHA1 Message Date
Marko Budiselic
888c6a4bca Add support for the id function
Reviewers: dgleich, teon.banek, mferencevic

Reviewed By: mferencevic

Subscribers: dgleich, mferencevic, teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1462
2018-07-04 22:14:13 +02:00
Marin Tomic
2b04d8213d Fix e function in openCypher
Summary:
For reasons unknown to man, antlr didn't want to parse calls
to functions whose name is a single hex letter properly. Now it should work.

Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1464
2018-07-02 16:37:26 +02:00
Marin Tomic
b934d194ca Implement subscripting operator for vertex and edge
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1461
2018-07-02 11:28:27 +02:00
Marin Tomic
18d8129b99 Implement string functions
Summary:
Added missing string functions.
I also cleaned up error messages a bit in effort to make them uniform.

Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1458
2018-07-02 10:39:11 +02:00
Teon Banek
843aa4f92a Handle indexed ScanAll in distributed Cartesian
Reviewers: mtomic, msantl, buda

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1435
2018-06-29 10:06:11 +02:00
Marin Tomic
86a00b00fa Implement extract function
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1455
2018-06-28 17:45:20 +02:00
Marko Budiselic
232ef77e43 Add a plan for integration of publicly available benchmarks
Reviewers: mculinovic

Reviewed By: mculinovic

Differential Revision: https://phabricator.memgraph.io/D1420
2018-06-28 10:06:40 +02:00
Marin Tomic
cd07664564 Add timestamp function
Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1452
2018-06-27 16:06:54 +02:00
Matej Ferencevic
8cdd6a906a Increase durability large test timeout
Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1454
2018-06-27 12:09:37 +02:00
Matej Ferencevic
1d448d40ca Implement SSL support for servers and clients
Summary:
This diff implements OpenSSL support in the network stack.
Currently SSL support is only enabled for Bolt connections,
support for RPC connections will be added in another diff.

Reviewers: buda, teon.banek

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1328
2018-06-20 17:56:47 +02:00
Matej Ferencevic
44821a918c Initial implementation of telemetry
Summary:
Add telemetry to main memgraph binary

Add resource usage collector

Add telemetry flag

Change telemetry collector logic

Fix utils compilation

Add timestamp

Add first version of interactive test

Started working on test runner

Implement all tests

Flake8 on runner.py

Integrate test with Apollo

Add TODO

Reviewers: buda, teon.banek

Reviewed By: buda, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1419
2018-06-20 14:49:07 +02:00
Teon Banek
4b97747c14 Allow planning Cartesian after Produce
Summary:
Hopefully, the mechanism of generating Cartesian is general enough, so
this simple change should work correctly in all cases.

Planner tests have been modified to use a FakeDbAccessor in order to
speed them up and potentially allow extracting planning into a library.

Reviewers: msantl, mtomic, buda

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1431
2018-06-20 12:55:56 +02:00
Teon Banek
2fbf2c7ff4 Handle dependent branches in basic Cartesian
Summary:
This change should correctly plan Cartesian which have dependent Filter
or Expand operators. Tests have been added for those cases. Other cases
are not yet supported and should throw an exception.

Reviewers: msantl, mtomic, buda

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1426
2018-06-19 13:06:34 +02:00
Ivan Paljak
4f28b4b6fe Remove 'using namespace' from header files
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1429
2018-06-15 15:28:31 +02:00
Ivan Paljak
d3bdca5ca3 Use temporary durability directory in PVS unit test
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1427
2018-06-15 15:27:36 +02:00
Marin Tomic
054c127175 Implement toString function
Summary: https://neo4j.com/docs/developer-manual/3.4/cypher/functions/string/#functions-tostring

Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1430
2018-06-15 13:18:21 +02:00
Marin Tomic
b017283bfb Add BFS and Dijkstra example
Reviewers: buda, msantl, teon.banek, ipaljak

Reviewed By: buda, ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1423
2018-06-15 10:45:14 +02:00
Marin Tomic
b9be394cb2 Add parsing and planning of basic user management queries
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek, mferencevic

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1398
2018-06-14 16:51:22 +02:00
Marin Tomic
3948cea83c Rename AstTreeStorage to AstStorage
Summary: happiness

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1403
2018-06-14 13:39:03 +02:00
Dominik Gleich
34e2a4f0ff Add tests
Reviewers: buda, msantl

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1425
2018-06-13 16:03:45 +02:00
Ivan Paljak
035540c598 POD serialization, rocksdb integration and Gleich's optimization
Reviewers: buda, dgleich, mferencevic, msantl, teon.banek

Reviewed By: buda, dgleich, teon.banek

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1399
2018-06-12 13:54:41 +02:00
Dominik Gleich
46034baeba Remove sleep from tests
Reviewers: buda, mferencevic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1421
2018-06-12 10:15:42 +02:00
Matej Ferencevic
23b91b929f Remove temporary allocations from bolt::Decoder
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1374
2018-06-05 13:00:24 +02:00
Marko Budiselic
a1e4dc0268 Add stackoverflow answers TCK test suite
Reviewers: teon.banek, mferencevic, mculinovic, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1412
2018-06-05 00:06:45 +01:00
Marko Culinovic
366cd1fffc Add ASAN, TSAN and UBSAN as build options
Summary:
Clangs Address, Thread and Undefined Behaviour sanitizers are now available as build options.
Only one of those can be set ON in the same build.

Reviewers: teon.banek, buda

Reviewed By: buda

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1380
2018-06-04 14:06:25 +02:00
Teon Banek
e0474a8e92 Replace boost with capnp in RPC
Summary:
Converts the RPC stack to use Cap'n Proto for serialization instead of
boost. There are still some traces of boost in other places in the code,
but most of it is removed. A future diff should cleanup boost for good.

The RPC API is now changed to be more flexible with regards to how
serialize data. This makes the simplest cases a bit more verbose, but
allows complex serialization code to be correctly written instead of
relying on hacks. (For reference, look for the old serialization of
`PullRpc` which had a nasty pointer hacks to inject accessors in
`TypedValue`.)

Since RPC messages were uselessly modeled via inheritance of Message
base class, that class is now removed. Furthermore, that approach
doesn't really work with Cap'n Proto. Instead, each message type is
required to have some type information. This can be automated, so
`define-rpc` has been added to LCP, which hopefully simplifies defining
new RPC request and response messages.

Specify Cap'n Proto schema ID in cmake

This preserves Cap'n Proto generated typeIds across multiple generations
of capnp schemas through LCP. It is imperative that typeId stays the
same to ensure that different compilations of Memgraph may communicate
via RPC in a distributed cluster.

Use CLOS for meta information on C++ types in LCP

Since some structure slots and functions have started to repeat
themselves, it makes sense to model C++ meta information via Common Lisp
Object System.

Depends on D1391

Reviewers: buda, dgleich, mferencevic, mtomic, mculinovic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1407
2018-06-04 10:45:12 +02:00
Teon Banek
e56ed0acce Add generating Capnp schema in LCP
Summary:
Add additional structs and functions for handling C++ meta information.
Add capnp-file and capnp-id arguments to lcp:process-file.
Generate cpp along with hpp and capnp in lcp.
Wrap LogicalOperator base class in lcp:define-class.
Modify logical operators for capnp serialization.
Add query/common.capnp.

Reviewers: mculinovic, buda, mtomic, msantl, ipaljak, dgleich, mferencevic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1391
2018-05-30 16:14:54 +02:00
Teon Banek
c7b6cae526 Extract io/network into mg-io library
Reviewers: buda, dgleich, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1411
2018-05-30 14:58:41 +02:00
Dominik Gleich
5b7947cc18 Revert "Remote vertex with gid creation"
Summary: This reverts commit 7d161319f0.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1410
2018-05-30 14:44:50 +02:00
Dominik Gleich
dd04aa7631 Revert "Remote edge with gid creation"
Summary: This reverts commit adda7d1200.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1409
2018-05-30 13:41:30 +02:00
Teon Banek
ce29517998 Extract utils into mg-utils static library and explicitly list tests
Summary:
Utils source files are now moved to a standalone mg-utils library.

Unit and manual tests are no longer collected using glob recursion in
cmake, but are explicitly listed. This allows us to set only required
dependencies of those tests.

Both of these changes should improve compilation and link times, as well
as lower the disk usage.

Additional improvement would be to cleanup utils header files to be
split in .hpp and .cpp as well as merging threading into utils. Other
potential library extractions that shouldn't be difficult are:

  * data_structures
  * io/network
  * communication

Reviewers: buda, mferencevic, dgleich, ipaljak, mculinovic, mtomic, msantl

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1408
2018-05-30 09:41:56 +02:00
Dominik Gleich
1875be1e34 Create Dynamic Graph Partitioner,
Summary:
Count labels in DGP

Add worker_id getter

Replace current_worker_id with worker_id

Add spinner

Add rpc calls for worker counts

Check worker capacity

Migrate to new worker

Fix moving two connected vertices

Token sharing algorithm

Reviewers: msantl, buda

Reviewed By: buda

Subscribers: msantl, pullbot

Differential Revision: https://phabricator.memgraph.io/D1392
2018-05-29 11:32:39 +02:00
Dominik Gleich
c294127065 Auto generate vertex/edge ids
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1406
2018-05-25 16:03:12 +02:00
Dominik Gleich
adda7d1200 Remote edge with gid creation
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1402
2018-05-25 09:57:55 +02:00
Dominik Gleich
63f90a453d Expose ID
Reviewers: buda, teon.banek

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1400
2018-05-23 15:04:50 +02:00
Dominik Gleich
7d161319f0 Remote vertex with gid creation
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1401
2018-05-23 10:10:14 +02:00
Matej Ferencevic
cf648a8453 Optimize Apollo build process
Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1394
2018-05-16 14:52:27 +02:00
Matija Santl
f872c93ad1 Add command id to remote produce
Summary:
Command id is necessary in remote produce to identify an ongoing pull
because a transaction can have multiple commands that all belong under
the same plan and tx id.

Reviewers: teon.banek, mtomic, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1386
2018-05-16 10:20:39 +02:00
Marin Tomic
91e38f6413 Distributed BFS
Summary: depends on D1387

Reviewers: msantl, teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1341
2018-05-15 17:38:51 +02:00
Matej Ferencevic
94ec0b23cd Fix distributed apollo runs
Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1393
2018-05-15 15:13:44 +02:00
Marin Tomic
0a6b8cdf4f Remove AS_IS from GraphView
Summary:
Removing `AS_IS` from GraphView because it doesn't seem like it is necessary for query execution and it also has weird semantics (you might get a mix of old and new records). `Unwind`, `OrderBy` and `PullRemoteOrderBy` now use `OLD` graph view.

Remove AS_IS from GraphView

Fix query_cost_estimator tests

Fix query_expression_evaluator tests

Fix query_plan_match_filter_return tests

Fix query_plan_create_set_remove_delete tests

Fix query_plan_accumulate_aggregate tests

Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1390
2018-05-15 13:10:19 +02:00
Ivan Paljak
909e42d414 Add initial version of properties on disk
Summary:
A simplified end-to-end implementation

POD interface set-up, still have bugs with HDDkeys

Version bug fix and first iterator implementation

Fixed out-of-scope reference in PVS iterator

Added PVS unit tests

Reviewers: buda, mferencevic, dgleich, teon.banek

Reviewed By: buda, dgleich

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1369
2018-05-10 17:58:38 +02:00
Dominik Gleich
797bd9e435 Notify master of worker recovery
Summary: Notify master when workers finish recovering such that master doesn't start doing stuff before they recovered

Reviewers: buda, msantl, mferencevic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1389
2018-05-10 11:09:34 +02:00
Marin Tomic
632663dca5 Fix address handling in remote edge creation
Summary: Global address of `from` was stored in Edge record when created via CreateEdge RPC.

Reviewers: buda, msantl, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1387
2018-05-07 13:20:51 +02:00
Marko Culinovic
49d69560e6 Serialize cpp types with capnp
Summary: Serialize shared_ptr, unique_ptr, optional and vector.

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1364
2018-05-04 15:02:38 +02:00
Marko Culinovic
4b9970ccf8 Save and load pointers in ast capnp serialization
Summary:
Pointers in AstTreeStorage can point to same nodes. These nodes
should be serialized(saved) only once when serializing Ast tree.
Same goes for deserializing(loading).
When deserializing nodes, one should not use storage Create method,
because it will add node which isn't completely loaded to storage vector.
Therefore, one should use new Deserialize method which doesn't
add node to storage vector. Inserting node into storage vector
is defered until all node data is loaded.

Pass pointer to saved uids set, instead of storage

Delete unused set

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1344
2018-05-04 09:50:34 +02:00
Marko Culinovic
8795501890 Serialize ast using capnproto
Reviewers: teon.banek, florijan, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1306
2018-05-03 14:12:38 +02:00
Teon Banek
ab574bf84f Benchmark Cap'n Proto and boost serialization
Reviewers: florijan, mferencevic, buda, mculinovic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1273
2018-05-02 12:38:18 +02:00
Marko Budiselic
7d94878860 Add rocksdb as a lib + initial implementation of the KVStore
Reviewers: teon.banek, mferencevic, ipaljak

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1331
2018-04-28 13:19:09 +02:00
Marko Budiselic
c76170a9db Clean utils folder (namespaces, function names)
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1359
2018-04-22 09:44:32 +02:00