Commit Graph

129 Commits

Author SHA1 Message Date
Matija Santl
a66351c3f4 Make RemotePull operator async
Reviewers: teon.banek, florijan

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1168
2018-02-06 13:42:11 +01:00
florijan
f84b0b0787 Fix address serialization
Summary:
Defined a new test based on reported bug, for multiple remote expansion.
Fixed the bug. Introduced minor refactors in distributed unit testing.

Reviewers: mculinovic, dgleich

Reviewed By: mculinovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1173
2018-02-06 11:01:06 +01:00
florijan
f808252142 Prepare RecordAccessor for distributed, part one
Summary:
This diff consolidates local and remote update handling. It ensures and
tests that updates for remote elements are visible locally (on the
updating worker).

The next part will be accumulating remote updates and applying them on
the owner.

Also extracted a common testing fixture.

Reviewers: dgleich, buda, mtomic

Reviewed By: mtomic

Subscribers: mtomic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1169
2018-02-05 09:48:50 +01:00
florijan
e5035cf477 Support graph elements in remote pull rpc
Reviewers: teon.banek, dgleich, msantl, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1157
2018-02-01 10:53:41 +01:00
Matija Santl
78afaa07a3 Use RPC ClientPool instead of Client
Summary: Use RPC `ClientPool` instead of `Client`

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot, mtomic

Differential Revision: https://phabricator.memgraph.io/D1153
2018-02-01 10:32:05 +01:00
florijan
3f30a6fad9 Add async remote pull RPC
Summary:
Remote pulls can now be async. Note that
`RemotePullRpcClients::RemotePull` takes references to data structures
which should not be temporary in the caller. Still, maybe safer to make
copies?

Changed `RpcWorkerClients` API to make that possible.

Reviewers: dgleich, msantl, teon.banek

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1150
2018-01-30 10:32:20 +01:00
Dominik Gleich
c37bb87ed8 Support snapshot creation and recovery in distributed
Summary:
Add custom encoder/decoder

Update snapshot recovery

Reviewers: florijan, teon.banek, mferencevic, mculinovic

Reviewed By: florijan

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1142
2018-01-29 19:16:13 +01:00
Teon Banek
760c6246d8 Make and dispatch worker plans on distributed master
Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1144
2018-01-29 10:55:57 +01:00
Matija Santl
a73a4c3762 Implement PullRemote logical operator
Summary: PullRemoteCursor will pull all clients in a RoundRobin fashion until all clients are exhausted and there are no more results to return.

Reviewers: teon.banek, florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1147
2018-01-26 17:12:45 +01:00
Dominik Gleich
24857cc1cf Support distributed (label, property) indexes
Summary:
Call workers buildindex

Merge branch 'master' into setup_distributed_index

Use ExecuteOnWorkers api

Merge branch 'master' into setup_distributed_index

Improve test

Merge branch 'master' into setup_distributed_index

Finish test

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1134
2018-01-25 18:08:04 +01:00
florijan
ccefae4002 Add distributed execution RPC
Summary:
NOTE: This diff is still in progress. Many TODOs, lacking documentation
etc. But the main logic is there (some could be different), and it tests
out OK.

Reviewers: teon.banek, msantl, buda

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1138
2018-01-25 14:50:28 +01:00
Matej Ferencevic
fc20ddcd25 RPC refactor
Summary:
Start removal of old logic
Remove more obsolete classes
Move Message class to RPC
Remove client logic from system
Remove messaging namespace
Move protocol from messaging to rpc
Move System from messaging to rpc
Remove unnecessary namespace
Remove System from RPC Client
Split Client and Server into separate files
Start implementing new client logic
First semi-working state
Changed network protocol layout
Rewrite client
Fix client receive bug
Cleanup code of debug lines
Migrate to accessors
Migrate back to binary boost archives
Remove debug logging from server
Disable timeout test
Reduce message_id from uint64_t to uint32_t
Add multiple workers to server
Fix compiler warnings
Apply clang-format

Reviewers: teon.banek, florijan, dgleich, buda, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1129
2018-01-24 15:27:40 +01:00
Matija Santl
31aded2dae Add test for PlanDispatcher and PlanConsumer
Summary:
Added test for `PlanDispatcher` and `PlanConsumer`.
This diff also contains a fix for the async rpc call on all clients.h

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot, dgleich

Differential Revision: https://phabricator.memgraph.io/D1135
2018-01-24 12:38:52 +01:00
Matija Santl
62323965e3 Implement distributed plan dispatcher/consumer methods
Summary:
Implementations of `DistributePlanRpc`.
I'll add tests afterwards #promise

Reviewers: teon.banek, florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1131
2018-01-24 10:50:27 +01:00
Dominik Gleich
ca32538f63 Fix set of clients
Reviewers: florijan

Reviewed By: florijan

Differential Revision: https://phabricator.memgraph.io/D1133
2018-01-23 13:08:52 +01:00
Dominik Gleich
5ad39a8735 Fix ExecuteOnWorkers
Reviewers: florijan

Reviewed By: florijan

Differential Revision: https://phabricator.memgraph.io/D1132
2018-01-23 11:29:39 +01:00
Dominik Gleich
5a698444bf Add ExecuteOnWorkers
Reviewers: florijan

Reviewed By: florijan

Differential Revision: https://phabricator.memgraph.io/D1130
2018-01-23 10:56:52 +01:00
Matija Santl
3bee31d8f3 Add PlanDispatcher and PlanConsumer stubs
Reviewers: teon.banek, florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1126
2018-01-23 10:15:24 +01:00
Dominik Gleich
ca9fac8adc Expose worker ids
Summary: Add worker ids expose to rpc_worker_clients

Reviewers: msantl

Reviewed By: msantl

Differential Revision: https://phabricator.memgraph.io/D1128
2018-01-22 17:09:12 +01:00
florijan
912d178391 Extract RpcWorkerClients
Summary: Extracting the RPC client-per-worker functionality we'll commonly need.

Reviewers: teon.banek, msantl, dgleich

Reviewed By: dgleich

Differential Revision: https://phabricator.memgraph.io/D1127
2018-01-22 16:27:05 +01:00
florijan
35197d6c4b Add distributed record TODOs
Summary:
It seems that RecordAccessor &co are ready for read-only distributed
execution. In read-only there is no command advancement and the implied
cache invalidation, `SwitchOld` and `SwitchNew` perform default
switching and `Reconstruct` uses the `RemoteCache` which is implemented.

I just added a few TODOs for proper CRUD.

Reviewers: dgleich

Reviewed By: dgleich

Differential Revision: https://phabricator.memgraph.io/D1125
2018-01-22 14:55:45 +01:00
florijan
e1e4a70714 Implement graph element rpc
Summary:
- End to end distributed GraphDb testing
- Refactors as necessary
- Basic RemoteCache for storing remote data
- RemoteDataRpc

As we are on a tight schedule, please let's focus on the essentials:
functionality and proper testing.

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1121
2018-01-22 14:20:41 +01:00
florijan
813d37e939 Migrate db::types to storage::
Reviewers: teon.banek, dgleich

Reviewed By: teon.banek, dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1110
2018-01-17 10:35:12 +01:00
Dominik Gleich
5418dfb19e Rename NetworkEndpoint
Summary:
Rename redunant port str

Add endpoint << operator

Migrate everything to endpoint

Reviewers: mferencevic, florijan

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1100
2018-01-15 15:47:37 +01:00
florijan
6fc6a27288 Refactor GraphDb
Summary:
GraphDb is refactored to become an API exposing different parts
necessary for the database to function. These different parts can have
different implementations in SingleNode or distributed Master/Server
GraphDb implementations.

Interally GraphDb is implemented using two class heirarchies. One
contains all the members and correct wiring for each situation. The
other takes care of initialization and shutdown. This architecture is
practical because it can guarantee that the initialization of the
object structure is complete, before initializing state.

Reviewers: buda, mislav.bradac, dgleich, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1093
2018-01-12 16:47:24 +01:00
Dominik Gleich
007a7f1a6d Change network design from Start/Shutdown to constructor/destructor
Summary:
Make ServerT start on constructor

Remove shutdown from MasterCoordinator

Distributed system remove Shutdown

Rcp server start and shutdown removed

Reviewers: florijan, mferencevic

Reviewed By: mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1097
2018-01-10 14:58:57 +01:00
florijan
3cf81380b8 Implement vertex/edge serialization
Summary:
Serialization of vertices and edges for distributed. Based on Boost
serialization. Threrefore moved TypedValue serialization from AST to
utils.

Reviewers: buda, dgleich, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1088
2018-01-10 13:32:18 +01:00
Mislav Bradac
d3623585e7 Migrate cereal to boost_serialization
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1077
2017-12-27 13:44:46 +01:00
florijan
3ae45e0d19 Add master/worker flags, main functions and coordination
Reviewers: dgleich, mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1060
2017-12-19 16:05:57 +01:00