Commit Graph

509 Commits

Author SHA1 Message Date
florijan
bfb3a0d9b1 Resolve global address to local
Summary:
It is possible that we have a global address to resolve, for a graph
element that's local. Consider W1 expanding, getting data from W2,
expanding from there and getting data that is on W1. We then don't want
to do RPC from W1 to W1, but do a lookup directly.

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1145
2018-01-26 16:09:19 +01:00
Dominik Gleich
24857cc1cf Support distributed (label, property) indexes
Summary:
Call workers buildindex

Merge branch 'master' into setup_distributed_index

Use ExecuteOnWorkers api

Merge branch 'master' into setup_distributed_index

Improve test

Merge branch 'master' into setup_distributed_index

Finish test

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1134
2018-01-25 18:08:04 +01:00
florijan
ccefae4002 Add distributed execution RPC
Summary:
NOTE: This diff is still in progress. Many TODOs, lacking documentation
etc. But the main logic is there (some could be different), and it tests
out OK.

Reviewers: teon.banek, msantl, buda

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1138
2018-01-25 14:50:28 +01:00
Matej Ferencevic
e2f2cd5722 Improve network performance
Summary:
With this patch the number of packets for a simple RPC call is lowered
from 22 to 12 (45% reduction). The number of packets for the Bolt protocol
is lowered from 26 to 18 (30% reduction).
Impact on the Bolt protocol will be a constant of ~ 8 packets less per
connection, while the impact on the RPC protocol will be approximately
a 45% reduction overall.

Reviewers: buda, teon.banek

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1141
2018-01-25 14:04:02 +01:00
Marin Tomic
6fa06f7ea5 Add rpc client pool
Summary: See above. The unit test creates two clients on demand so I guess it works.

Reviewers: mferencevic, florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1136
2018-01-25 12:48:10 +01:00
Matej Ferencevic
010ed52622 Implement RPC call abort
Reviewers: teon.banek, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1140
2018-01-24 22:02:26 +01:00
Matej Ferencevic
fc20ddcd25 RPC refactor
Summary:
Start removal of old logic
Remove more obsolete classes
Move Message class to RPC
Remove client logic from system
Remove messaging namespace
Move protocol from messaging to rpc
Move System from messaging to rpc
Remove unnecessary namespace
Remove System from RPC Client
Split Client and Server into separate files
Start implementing new client logic
First semi-working state
Changed network protocol layout
Rewrite client
Fix client receive bug
Cleanup code of debug lines
Migrate to accessors
Migrate back to binary boost archives
Remove debug logging from server
Disable timeout test
Reduce message_id from uint64_t to uint32_t
Add multiple workers to server
Fix compiler warnings
Apply clang-format

Reviewers: teon.banek, florijan, dgleich, buda, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1129
2018-01-24 15:27:40 +01:00
Matija Santl
31aded2dae Add test for PlanDispatcher and PlanConsumer
Summary:
Added test for `PlanDispatcher` and `PlanConsumer`.
This diff also contains a fix for the async rpc call on all clients.h

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot, dgleich

Differential Revision: https://phabricator.memgraph.io/D1135
2018-01-24 12:38:52 +01:00
Dominik Gleich
ca9fac8adc Expose worker ids
Summary: Add worker ids expose to rpc_worker_clients

Reviewers: msantl

Reviewed By: msantl

Differential Revision: https://phabricator.memgraph.io/D1128
2018-01-22 17:09:12 +01:00
Teon Banek
1d6ac3d23d Add ProduceRemote and PullRemote operator stubs
Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1124
2018-01-22 15:08:32 +01:00
florijan
e1e4a70714 Implement graph element rpc
Summary:
- End to end distributed GraphDb testing
- Refactors as necessary
- Basic RemoteCache for storing remote data
- RemoteDataRpc

As we are on a tight schedule, please let's focus on the essentials:
functionality and proper testing.

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1121
2018-01-22 14:20:41 +01:00
Teon Banek
252018ab22 Serialize query plan operators
Summary:
With the added support for serialization, we should be able to transfer
plans across distributed workers.

The planner tests has been extended to test serialization. Operators
should be mostly tested, but the expression they contain aren't
completely. The quick solution is to use typeid for rudimentary
expression equality testing.  The more involved solution of comparing
the expression tree for equality is the correct choice. It should be
done in the near future.

Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1122
2018-01-22 12:48:05 +01:00
Teon Banek
2a8c64882f Serialize SymbolTable
Summary:
Other than the plan operators and the frame, we will need to pass the
generated symbol table to distributed workers.

Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1123
2018-01-22 12:47:51 +01:00
Dominik Gleich
68a75eeca2 Add database master test
Summary: Seperate rpc server names

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1118
2018-01-19 14:52:05 +01:00
florijan
9361d79c6d Implement GraphDbAccessor creation for running transaction
Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1119
2018-01-19 13:21:54 +01:00
Dominik Gleich
2a130e784e Worker id in snapshot/wal
Summary:
Adds worker id to snapshot and wal filename.
Adds a new worker_id flag to be used for recovering a worker with a distributed snapshot.
Adds worker_id field to snapshot to check for consistency.

Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1096
2018-01-18 11:46:47 +01:00
Teon Banek
93de41e717 Throw RecordDeletedError if updating a deleted record
Summary:
Previously, we would have a `DCHECK` which crashes the application. This
was evident when testing a queries, such as:

    MATCH (n) DELETE n SET n.prop = 42

Since the argument to update clauses is evaluated during execution, it
makes it very difficult to prevent such errors during semantic analysis.
For example:

    MATCH (n)--(m) WITH collect(n) as ns, m
    DETACH DELETE ns[m.prop] SET head(ns).prop = 42

Test query updates on deleted graph elements

Reviewers: florijan, dgleich

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1114
2018-01-17 14:30:57 +01:00
florijan
813d37e939 Migrate db::types to storage::
Reviewers: teon.banek, dgleich

Reviewed By: teon.banek, dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1110
2018-01-17 10:35:12 +01:00
Teon Banek
98cbd2b500 Cleanup utils::File API
Summary:
Close the file descriptor in File destructor. This will prevent
accidental crashes during unexpected destructor calls. For example, if
an exception is thrown before the file is closed. File now takes
ownership of the descriptor. These changes now honor RAII idiom, which
should handle most of the peculiarities of C++.

Use optional value for TryOpenFile function, instead of returning a File
without a descriptor. It makes the failure state more semantically clear
to the API user.

Merge utils/filesystem with utils/file

The files aren't that big, and the naming is a bit confusing because
functions aren't really grouped for file and filesystem distinction.

Reviewers: mferencevic, mtomic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1111
2018-01-16 15:38:12 +01:00
Marin Tomic
9e42ebbb67 Implement simple log file storage for raft
Summary:
Added wrappers for some Unix system calls in utils/filesystem.hpp and implemented
a simple log storage interface for Raft. It is not very efficient, we will need
something more sophisticated later, but this is good enough for testing.

Reviewers: mferencevic, mislav.bradac, buda, mculinovic

Reviewed By: mferencevic

Subscribers: teon.banek, dgleich, pullbot

Differential Revision: https://phabricator.memgraph.io/D1091
2018-01-15 18:07:45 +01:00
Teon Banek
a166c613ec Inline test data in the benchmark tests itself
Summary:
The directory was never actually copied on apollo, so tests weren't even
doing anything...

Also remove fswatcher unit test, it should be rewritten correctly.

Reviewers: mislav.bradac, mferencevic, buda

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1108
2018-01-15 16:12:30 +01:00
Dominik Gleich
5418dfb19e Rename NetworkEndpoint
Summary:
Rename redunant port str

Add endpoint << operator

Migrate everything to endpoint

Reviewers: mferencevic, florijan

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1100
2018-01-15 15:47:37 +01:00
Matej Ferencevic
41679b6ec5 Integrate code coverage with Apollo
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1050
2018-01-15 12:20:11 +01:00
florijan
80c5fd48e1 Reduce durability test flakyness
Summary:
Reduce flakyness by ensuring the first database gets destructed before
taking wal-file count.

Reviewers: teon.banek, mferencevic, dgleich

Reviewed By: mferencevic, dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1105
2018-01-15 11:12:05 +01:00
florijan
6fc6a27288 Refactor GraphDb
Summary:
GraphDb is refactored to become an API exposing different parts
necessary for the database to function. These different parts can have
different implementations in SingleNode or distributed Master/Server
GraphDb implementations.

Interally GraphDb is implemented using two class heirarchies. One
contains all the members and correct wiring for each situation. The
other takes care of initialization and shutdown. This architecture is
practical because it can guarantee that the initialization of the
object structure is complete, before initializing state.

Reviewers: buda, mislav.bradac, dgleich, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1093
2018-01-12 16:47:24 +01:00
florijan
c4327b26f4 Extract tx::SingleNodeEngine from tx::MasterEngine
Summary:
No logic changes, just split `tx::MasterEngine` into
`tx::SingleNodeEngine` and `tx::MasterEngine`. This gives better
responsibility separation and is more appropriate now there is no
Start/Shutdown.

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1099
2018-01-11 15:44:42 +01:00
Dominik Gleich
007a7f1a6d Change network design from Start/Shutdown to constructor/destructor
Summary:
Make ServerT start on constructor

Remove shutdown from MasterCoordinator

Distributed system remove Shutdown

Rcp server start and shutdown removed

Reviewers: florijan, mferencevic

Reviewed By: mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1097
2018-01-10 14:58:57 +01:00
florijan
87f8c4e256 Fix durability test flakyness (hopefully)
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1098
2018-01-10 14:24:40 +01:00
florijan
3cf81380b8 Implement vertex/edge serialization
Summary:
Serialization of vertices and edges for distributed. Based on Boost
serialization. Threrefore moved TypedValue serialization from AST to
utils.

Reviewers: buda, dgleich, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1088
2018-01-10 13:32:18 +01:00
florijan
4a0345e1c5 Prepare counter for distributed
Reviewers: dgleich, teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1090
2018-01-10 13:28:41 +01:00
florijan
99000c6ec1 Remove PropertyValueStore templatization
Summary:
A PropertyValueStore is not a generic data structure, but only ever used
to store properties in a Vertex/Edge. It has behaviours specific to it.
So, the templatization was not necessary.

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1089
2018-01-02 16:18:55 +01:00
Dominik Gleich
503381549e Change gid bit size
Summary:
Change gid methods

Rename GidGenerator and add tests

Fix tools broken by gid changes

Reviewers: dgleich, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1044
2017-12-28 11:04:52 +01:00
Mislav Bradac
d3623585e7 Migrate cereal to boost_serialization
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1077
2017-12-27 13:44:46 +01:00
florijan
9b878c91eb Make interpretation pullable
Reviewers: mislav.bradac, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1079
2017-12-22 15:19:05 +01:00
Marin Tomic
e999207b2f Remove reactors
Summary: kill reactors

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Differential Revision: https://phabricator.memgraph.io/D1075
2017-12-20 17:56:57 +01:00
Dominik Gleich
1556d78d15 Update snapshot format
Summary:
Set vertex/edge generator id from recovery

Add tests

Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1061
2017-12-20 16:57:42 +01:00
Marin Tomic
20aa8d563a Implement RaftNetworkInterface via RPC
Summary:
Add missing fields in RequestVote RPC

Unnest Raft RPC requests and responses

Make RPC request and reply serializable

Refactor Raft network interface

Implement RPC retrying with backoff period

Add cerealization functions for optional

Add RPC network interface for raft

Add some comments and documentation

Add manual test for RPC network interface

Remove raft reactor experiments

Reviewers: mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot, mculinovic

Differential Revision: https://phabricator.memgraph.io/D1066
2017-12-20 16:48:46 +01:00
Teon Banek
5f7837d613 Serialize Ast classes
Summary:
Although the first solution used cereal, the final implementation uses
boost. Since the cereal is still used in the codebase, compilation has
been modified to support multithreaded cereal.

In addition to serializing Ast classes, the following also needed to be
serialized:

  * GraphDbTypes
  * Symbol
  * TypedValue

TypedValue is treated specially, by inlining the serialization code in
the Ast class, concretely PrimitiveLiteral.

Another special case was the Function Ast class, which now stores a
function name which is resolved to a concrete std::function on
construction.

Tests have been added for serialized Ast in
tests/unit/cypher_main_visitor

Reviewers: mferencevic, mislav.bradac, florijan

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1067
2017-12-20 13:28:34 +01:00
florijan
3ae45e0d19 Add master/worker flags, main functions and coordination
Reviewers: dgleich, mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1060
2017-12-19 16:05:57 +01:00
florijan
37722e54b3 Add RPC to concurrent ID mapper
Summary:
The distributed ID mapper is not yet utilised in GraphDb as those
changes are in D1060. Depending on landing order it will be added.

Reviewers: dgleich, mislav.bradac

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1064
2017-12-19 09:11:16 +01:00
Dominik Gleich
0eb30db8db Remove executor not run option
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1062
2017-12-18 14:36:15 +01:00
Marin Tomic
5b7cb4602c Add simple client facing API for RaftMember
Summary:
Implement AddComand method on RaftMember

Move RPCType out of rpc request and reply

Add unit test for AddCommand

Reviewers: mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot, mculinovic

Differential Revision: https://phabricator.memgraph.io/D1042
2017-12-18 07:32:39 +01:00
Dominik Gleich
0f1e8aa763 Add rpc server local thread
Summary: Update tests for new rpc server

Reviewers: mislav.bradac, florijan, mferencevic

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1057
2017-12-14 13:37:32 +01:00
florijan
52b3941c8a Fix tx::WorkerEngine flakyness. Change Endpoint::port type.
Reviewers: mislav.bradac, dgleich

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1054
2017-12-14 10:36:24 +01:00
Matija Santl
2758b35c08 Fix semicolon in column name
Summary:
Special casing the last token when adding it to the named expression list.
Added a test to check this case.

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1052
2017-12-14 09:44:21 +01:00
Dominik Gleich
a8488ac497 Rpc client locking
Summary:
Rpc client wasn't thread safe and required a lock before each rpc call.
The locking functionality is now incorporated in Rpc client.

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Differential Revision: https://phabricator.memgraph.io/D1056
2017-12-13 17:37:32 +01:00
Matija Santl
4da8312abe Fix query returning a map
Summary: Add curvy braces handling in `QueryStripper` and an accompanying test for it.

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1051
2017-12-13 15:38:20 +01:00
Mislav Bradac
cf7d9070d8 Fix flakyness in rpc/messaging tests
Summary: In this diff I just wanted to fix tests' flakyness. We can discuss if we want to always pass endpoint as an argument and never pass address:port pair explicitly. However if we decide that, I will do that change in another diff.

Reviewers: dgleich, florijan

Reviewed By: dgleich, florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1049
2017-12-13 13:15:02 +01:00
Matija Santl
9ad7d82a54 Implement UNION query combinator
Summary:
Union query combinator implementation consists of:
 * adjustments to the AST and `cypher_main_visitor`
 * enabling `QueryStripper` to parse multiple `return` statements (not stopping after first)
 * symbol generation for union results
 * union logical operator
 * query plan generator adjustments

Reviewers: teon.banek, mislav.bradac

Reviewed By: teon.banek

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1038
2017-12-13 09:54:00 +01:00
Mislav Bradac
eb272f0b67 Assert on endpoint failures
Summary: .

Reviewers: mferencevic, florijan

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1045
2017-12-12 14:27:24 +01:00