Commit Graph

954 Commits

Author SHA1 Message Date
florijan
44aefe775e Fix flakyness in tests/unit/distributed_coord
Summary:
Instead of waiting for a fix period for the coordinations to start and
coordinate with the master, wait for each of them individually to report
being done.

Also: rename `WorkerInThread` to `WorkerCoordinationInThread`.

Reviewers: dgleich, teon.banek, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1288
2018-03-12 12:30:59 +01:00
florijan
67092ae4d7 Fix ScanAll cursor family w.r.t. empty collections
Reviewers: teon.banek, dgleich, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1291
2018-03-12 11:18:07 +01:00
florijan
bdc7ec9b71 Create a local distributed test based on the credit card fraud demo.
Reviewers: msantl, mtomic

Reviewed By: msantl

Differential Revision: https://phabricator.memgraph.io/D1289
2018-03-09 13:52:42 +01:00
florijan
42ca81eb01 Use custom future that waits on destruct
Reviewers: teon.banek, dgleich

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1286
2018-03-09 10:16:17 +01:00
florijan
848749cf25 Fix and rename REPL
Summary: Fix and rename REPL

Reviewers: teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1284
2018-03-08 10:58:43 +01:00
Marin Tomic
68c68655cc Fix transaction id offset
Summary: ^^

Reviewers: mferencevic

Reviewed By: mferencevic

Differential Revision: https://phabricator.memgraph.io/D1283
2018-03-07 16:49:55 +01:00
Marin Tomic
0013c80881 Change transaction id distribution strategy
Summary: this will make delete query more efficient

Reviewers: mferencevic, mculinovic

Reviewed By: mferencevic

Differential Revision: https://phabricator.memgraph.io/D1282
2018-03-07 14:47:19 +01:00
Matija Santl
b8685509aa Add distributed cartesian test
Summary: Cartesian test

Reviewers: teon.banek, florijan

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1279
2018-03-07 11:17:19 +01:00
Marin Tomic
5637703317 add strata v1 scenario
Reviewers: mculinovic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1266
2018-03-06 11:20:39 +01:00
Marko Culinovic
89cf8ca077 Add connected_frauds to pos properties
Summary: Property connected_frauds added and set to zero for strata scenario.

Reviewers: mtomic

Reviewed By: mtomic

Differential Revision: https://phabricator.memgraph.io/D1275
2018-03-06 11:02:09 +01:00
florijan
23efdf27d0 Optimize card_fraud_generate_snapshot
Summary:
Reduce memory usage and execution time in card fraud snapshot
generation.

Reviewers: mculinovic

Reviewed By: mculinovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1270
2018-03-05 16:36:00 +01:00
Teon Banek
9c95d4c381 Allow replacing Accumulate with Synchronize in all cases
Summary:
This fixes a bug where the planning would raise NotYetImplemented error,
due to preventing plan splitting while the operator is already on
master. For example, `MATCH (a), (b) CREATE (a)-[e:r]->(b) RETURN e`
would split the plan after Cartesian and before Create. Thus, the rest
of the plan would be on master, including the Accumulate before Produce.
We still can (and must) replace Accumulate with Synchronize but this
would fail due to unneeded check.

Reviewers: florijan, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1272
2018-03-05 15:35:36 +01:00
florijan
43c0e91057 Enable transaction killer on worker
Summary:
I have not thoroughly thought this through, especially the worker
destruction (is it legit to abort all running tx?), but it's tested to
abort during remote pull, what we need.

Also I improved error handling for vertex deletion failure during
remote pull (@dgleich).

Reviewers: teon.banek, msantl, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1263
2018-03-05 12:45:42 +01:00
Dominik Gleich
d5b9a11e87 DetachRemoveVertex remote
Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1264
2018-03-02 10:51:26 +01:00
Matija Santl
1ca98826af Use the same ClientPools in distributed
Summary:
Instead of passing `coordination`, pass `rpc_worker_clients` that
holds a map of worker_id->clientPool. By having only one instance of
`RpcWorkerClients` that is owned by `GraphDB` and passing it by refference
we'll share the same client pools for rpc clients.

Reviewers: teon.banek, florijan, dgleich, mferencevic

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1261
2018-03-01 17:14:59 +01:00
Matej Ferencevic
5839dbc401 Fix distributed tests
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1262
2018-03-01 13:19:18 +01:00
florijan
e61bb8ab1e Support distributed edge removal
Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1256
2018-02-28 14:35:40 +01:00
Dominik Gleich
99375a4b47 Vertex removal using rpcs
Summary:
Remove vertex remote

Add tests

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1230
2018-02-28 11:35:44 +01:00
Matej Ferencevic
7e945c6667 Integrate distributed card fraud test with Apollo
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1247
2018-02-28 11:09:18 +01:00
Matej Ferencevic
19abd0ce02 Move TED talk examples to another suite in QA
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1249
2018-02-27 19:19:42 +01:00
Matej Ferencevic
a826f41444 Temporarily disable parallel macro benchmarks
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1248
2018-02-27 11:05:13 +01:00
florijan
bb62f463f8 Refactor distributed transactional cache GC
Summary:
Release of per-transaction data in distributed Memgraph refactored. The
master node no longer releases each time a transaction is done, thus
offloading some work from the engine.

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1235
2018-02-27 10:47:58 +01:00
Matej Ferencevic
7b262599e6 Fix Bolt timeout check
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1242
2018-02-26 18:18:01 +01:00
Dominik Gleich
ef6cfc4c0e Refactor state deltas call
Summary:
Add label test

Index gets updated after remote update

Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1246
2018-02-26 15:44:50 +01:00
florijan
c8dc07ad0e Remove tx::Engine::GlobalIsActive
Summary:
Remove a method in tx::Engine whose results can be obtained from commit
log info (also guaranteed to be globally correct in distributed).

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1240
2018-02-26 14:01:49 +01:00
florijan
dc21511e6e Fix remote edge create
Summary:
Difficult to test properly as the problem was an implicit conversion of
`gid::Gid` to `mvcc::VersionList<> *`. The only proper way to defend
against this would be to make `gid::Gid` a type.

The `CHECK` added in this diff does not help, but should be there, so...

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1244
2018-02-26 14:00:45 +01:00
Teon Banek
a44d9a0fac Always plan Synchronize if the plan needs it
Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1241
2018-02-26 13:16:08 +01:00
Teon Banek
30d2cfb9db Plan PullRemoteOrderBy
Summary:
During distributed execution, OrderBy is split across workers and the
master gets to merge those results via PullRemoteOrderBy. Since this
operator may be an input to almost any other operator, virtual accessors
to `input` have been added in LogicalOperator.

Depends on D1221

Reviewers: florijan, msantl, buda

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1232
2018-02-26 11:15:46 +01:00
Matej Ferencevic
2a68543f3e Change Apollo slave groups
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1238
2018-02-25 23:02:31 +01:00
Matija Santl
b9d61a0127 Implement distributed aware OrderBy operator
Summary:
Extracted `TypedValueVectorCompare` and `RemotePuller` from operators
so it can be reused.
The new `PullRemoteOrerBy` operator pulls one result from each worker and one
from master, relies on the fact that workers/master returned sorted results,
returns the next one in order, and pulls the source of that result to get the
next one.

Depends on D1215 that (at the moment) is still in review.

Reviewers: florijan, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1221
2018-02-23 17:44:21 +01:00
florijan
cbc9420c17 Support distributed query::plan::CreateExpand
Summary:
- The expansion vertex gets created on the origin's worker
- The edge automatically gets created wherever necessary
- Vertex creation logic reuse
- End to end test

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1227
2018-02-23 17:34:01 +01:00
florijan
c83078e7d9 Suppress logging in tests/manual
Reviewers: teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1237
2018-02-23 16:36:49 +01:00
Matej Ferencevic
d99724647a Add thread names
Summary:
To see the thread names in htop press <F2> to enter setup and under
'Display options' choose 'Show custom thread names'.

Reviewers: buda, teon.banek, florijan

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1234
2018-02-23 16:04:49 +01:00
florijan
80b86f0314 Add test/manual/distributed_console
Summary:
- Add the said test
- Add `workerid` function
- Add random wait (enabled with flag) to `rpc::Client::Call` for network
  latency simulation

Reviewers: buda, mferencevic, teon.banek

Reviewed By: buda, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1222
2018-02-23 15:23:26 +01:00
Matej Ferencevic
c877c87bb4 Refactor RPC
Summary:
Previously, the RPC stack used the network stack only to receive messages. The
messages were then added to a separate queue that was processed by different
thread pools. This design was inefficient because there was a lock when
inserting and getting messages from the common queue.

This diff removes the need for separate thread pools by utilising the new
network stack design. This is possible because the new network stack allows
full processing of the network request without blocking the whole queue.

Reviewers: buda, florijan, teon.banek, dgleich, mislav.bradac

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1229
2018-02-23 12:07:22 +01:00
Marin Tomic
604ebf9d1e Add tx creation scenario
Summary: ^^

Reviewers: buda, mculinovic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1225
2018-02-22 17:43:32 +01:00
Teon Banek
452ce6a30c Use std::async for fetching remote edges in Expand
Summary:
This may improve the query execution workload, because the Expand will
yield local results while it waits for remote ones. Note that we rely on
the fact that walking the graph produces results without any predetermined
order. Therefore, we can yield paths as we see fit. Enforcing the order
is done through OrderBy operator, and this change shouldn't affect that.

Reviewers: florijan, msantl, buda

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1215
2018-02-22 17:11:53 +01:00
Dominik Gleich
753aa07cdf Tests and fix remote update already deleted bug
Summary:
Updating a record locally while there is an remote update waiting to be applied caused
the operation to return as already deleted, instead of applying it

Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1226
2018-02-22 16:36:22 +01:00
Matej Ferencevic
017e8004e8 Refactor network stack
Summary:
Previously, the network stack `communication::Server` accepted connections and
assigned them statically in a round-robin fashion to `communication::Worker`.
That meant that if two compute intensive connections were assigned to the same
worker they would block each other while the other workers would do nothing.

This implementation replaces `communication::Worker` with
`communication::Listener` which holds all accepted connections in one pool and
ensures that all workers execute all connections.

Reviewers: buda, florijan, teon.banek

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1220
2018-02-22 16:29:17 +01:00
Teon Banek
c7eaf4711a Pass a flag for distributed CreateNode
Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1218
2018-02-22 14:25:46 +01:00
Marin Tomic
5ff1ca4a5d Add client side RPC stats
Summary: ^^

Reviewers: mferencevic, buda

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1224
2018-02-22 12:34:48 +01:00
Dominik Gleich
46b8a91a2f Add rpc_worker_clients tests
Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1214
2018-02-21 13:54:28 +01:00
florijan
ae95e9480c Use RemoteCreateVertex in CreateNode operator
Summary:
- Add `database::GraphDb::GetWorkerIds()`
- Change `CreateNode` constructor API
- Make `CreateNode` distribute nodes uniformly over workers

Did not yet modify `CreateExpand`, coming in the next diff.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1216
2018-02-21 12:17:43 +01:00
Marin Tomic
f1cdae0617 Add teps and pointwise lookup scenarios
Reviewers: mculinovic

Reviewed By: mculinovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1211
2018-02-21 11:44:57 +01:00
florijan
0c40c67ac2 Implement remote create (storage, RPC, not operator)
Summary:
Implementation of remote vertex and edge creation. This diff addresses
the creation API (`GraphDbAccessor::InsertEdge`,
`GraphDbAccessor::InsertRemoteVertex`) and the necessary RPC and
`RemoteCache` stuff.

What is missing for full remote creation support are
`query::plan::operator` changes that are expected to minor. Pushing this
diff as it's large enough, operator and end to end tests in the next.

Also, the naming of existing structures and files is confusing (update
refering to both updates and created, `results` used too often etc.). I
will address this too, but feel free to comment on bad naming.

Reviewers: dgleich, teon.banek, msantl

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1210
2018-02-21 09:17:48 +01:00
Teon Banek
2055176139 Plan basic distributed Cartesian without any checks
Summary:
This is still very much in progress.  No advanced checks are done to
prevent planning unimplemented things.  Basic Cartesian product should
work, for example `MATCH (a), (b) CREATE (a)-[:r]->(c)-[:r]->(b)`. But
anything more advanced may lead to undefined behaviour of the planner
and therefore execution. Use at your own risk!

Add ModifiedSymbols method to LogicalOperator

For planning Cartesian, we need information on which symbols are filled
by operator sub-trees.  Currently, this is used to set symbols which
should be transferred over network. Later, they should be used to detect
whether filter expressions use symbols modified from Cartesian branches.
Then we will be able to ensure correct dependency of filters and their
behaviour.

Prepare DistributedPlan for multiple worker plans

Since Cartesian branches need to be split and handled by each worker, we
now dispatch multiple plans to workers.

Reviewers: florijan, msantl, buda

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1208
2018-02-20 13:02:08 +01:00
Teon Banek
8856682c7f Always plan Synchronize for write queries
Reviewers: florijan, msantl

Reviewed By: florijan, msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1179
2018-02-20 09:55:42 +01:00
Dominik Gleich
0639c44ed7 Fix flaky concurrent list test
Summary:
Rename and fix concurrent list test.,
This tests a fix for the concurrent list.I was not able to reproduce the flakyness, but this could be it.
If it happens again we'll know that this is not the solution to the problem, and look further.
Change memory ordering

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: buda, pullbot

Differential Revision: https://phabricator.memgraph.io/D1188
2018-02-19 12:49:38 +01:00
Matej Ferencevic
611e3cd149 Increase concurrent tests timeout
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1207
2018-02-19 11:08:59 +01:00
Marko Culinovic
61967f2e06 Fix statsd and benchmark client
Summary:
Stats server wasn't connecting to the right service on statsd.
Also, benchmark client stats now have prefix `client` instead
of machine name to be consistent with memgraphs stats naming
which starts with `master` or `worker`.

Reviewers: mtomic, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1209
2018-02-17 17:06:25 +01:00
florijan
9721ccf61c Cleanup per-transaction caches in distributed
Summary:
On the master cleanups are hooked directly into the transaction engine.
This is beneficial because the master might have bigger caches and we
want to clear them as soon as possible.

On the workers there is a periodic RPC call to the master about living
transactions, which takes care of releasing local caches. This is
suboptimal because long transactions will prevent cache GC (like with
data GC). It is however fairly simple.

Note that all cleanup is not done automatically and `RemotePull` has
been reduced accordingly. @msantl, please verify correctness and
consider if the code can be additionally simplified.

Reviewers: teon.banek, msantl

Reviewed By: msantl

Subscribers: pullbot, msantl

Differential Revision: https://phabricator.memgraph.io/D1202
2018-02-16 15:30:05 +01:00
Matija Santl
84e7aec27b Implement Cartesian Cursor
Summary:
Pulls left op cursor and keeps the result, and then for each pull of
the right op cursor, adds all the left op results to produce a cartesian
product.

Reviewers: teon.banek, florijan

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1201
2018-02-16 13:43:12 +01:00
Matej Ferencevic
763bdaac0f Add support for large messages to RPC
Reviewers: buda, teon.banek, mculinovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1196
2018-02-16 12:38:31 +01:00
Marin Tomic
d15464e181 Add some metric types and basic RPC server stats
Summary:
Get rid of client class

Fix cppcheck errors

Add documentation to metrics.hpp

Add documentation to stats.hpp

Remove stats from global namespace

Fix build failures

Refactor a bit

Refactor stopwatch into a function

Add rpc execution time stats

Fix segmentation fault

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1200
2018-02-16 08:33:15 +01:00
florijan
b2d7f95568 Extract address types
Summary:
We have been using `Edges::VertexAddress` and `Edges::EdgeAddress` a lot
in other parts of the codebase because it's cleaner to write then
`Address<mvcc::VersionList<Edge>>`, especially in code what should not
really be MVCC-aware. However, a lot of that code should not really be
`Edges` aware either, as that's a storage datastructure that should not
be exposed.

This became annoying, so I extracted these addresses into a type-file. I
don't really like this approach, it might be better to have
`Vertex::Address` and `Edge::Address`, but that means we'd have to
import those headers and we'd get circular dependencies.

“The horror! The horror!”
   - Joseph Conrad, Heart of Darkness

Reviewers: teon.banek, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1204
2018-02-15 17:31:10 +01:00
Matija Santl
f83cc31779 Add openCypher 09 tck tests
Summary:
Added latest tck openCypher tests. Made sure the file is parseable and that
it returns coverage.

Also bumped continuous integration configuration to use the 09 version.

Reviewers: mferencevic, teon.banek, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1197
2018-02-15 16:57:41 +01:00
florijan
796946ad1b Implement sync operator
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1192
2018-02-14 13:03:18 +01:00
Matija Santl
b1a8e48dc4 Add ExpandWeightedShortestPathCursor
Summary: Implement Dijkstras algorithm to find weighted shortest path.

Reviewers: teon.banek, florijan, buda

Reviewed By: teon.banek

Subscribers: pullbot, mtomic

Differential Revision: https://phabricator.memgraph.io/D1182
2018-02-14 09:32:41 +01:00
Teon Banek
4326847ab3 Add SINGLE function to openCypher
Reviewers: florijan, msantl, buda

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1193
2018-02-13 17:30:44 +01:00
Teon Banek
e8608f4050 Request correct remote symbols for non-associative aggregation
Reviewers: florijan, msantl, mculinovic

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1191
2018-02-13 15:39:09 +01:00
Marko Culinovic
b99436910f Implement manage script for card fraud demo
Summary:
Script provides helper actions for demo:
1) Copy memgraph to cluster
2) Copy durability directories to cluster
3) Start master and workers
4) Stop memgraph cluster
5) Clean memgraph directories
6) Clean durability directories

Cluster config should be provided in separate json file.

Reviewers: buda, mtomic

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1180
2018-02-12 13:29:12 +01:00
Teon Banek
c1e4676316 Add optional total_weight variable to wShortest grammar
Reviewers: florijan, msantl, buda

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1187
2018-02-12 13:23:17 +01:00
Teon Banek
db407142ce Add wShortest expansion to openCypher
Reviewers: florijan, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1181
2018-02-09 11:02:41 +01:00
Matija Santl
013ee34f90 Fix RemotePull infinite loop when all workers don't have results
Reviewers: buda, teon.banek, florijan

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1184
2018-02-08 16:29:34 +01:00
florijan
45da645a4b Fix remote pull
Summary:
- Extracted TypedValue reconstruction into utils (will need that soon)
- Added more error handling in RemotePull

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1183
2018-02-08 15:00:40 +01:00
Marko Budiselic
e2f5bcf96d Stats from card fraud benchmark
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1177
2018-02-07 19:57:25 +01:00
florijan
81e2e8f64f Add remote updates RPC
Summary:
Updates are supported, insertions and removals not in this diff. The
test is a bit overdesigned, it happens.

Reviewers: teon.banek, dgleich, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1176
2018-02-07 15:29:57 +01:00
Teon Banek
fa55c130ba Add REDUCE function to openCypher
Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1175
2018-02-07 12:48:20 +01:00
Matija Santl
a66351c3f4 Make RemotePull operator async
Reviewers: teon.banek, florijan

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1168
2018-02-06 13:42:11 +01:00
florijan
f84b0b0787 Fix address serialization
Summary:
Defined a new test based on reported bug, for multiple remote expansion.
Fixed the bug. Introduced minor refactors in distributed unit testing.

Reviewers: mculinovic, dgleich

Reviewed By: mculinovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1173
2018-02-06 11:01:06 +01:00
Teon Banek
abe419984a Plan Unwind and write operations in distributed
Summary:
This is a basic take on planning distributed writes. Main logic is in handling
the Accumulate operator, which requires Synchronize operation if the results
are used after modification.

Tests for planning distributed writes have been added.

Reviewers: florijan, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1172
2018-02-06 10:13:53 +01:00
Matej Ferencevic
b95e2084ad Fix daily LDBC crash
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1171
2018-02-05 19:02:49 +01:00
Marko Culinovic
523d54392b Fix pass by values
Summary: Also, some other minor changes

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1170
2018-02-05 14:31:37 +01:00
florijan
f808252142 Prepare RecordAccessor for distributed, part one
Summary:
This diff consolidates local and remote update handling. It ensures and
tests that updates for remote elements are visible locally (on the
updating worker).

The next part will be accumulating remote updates and applying them on
the owner.

Also extracted a common testing fixture.

Reviewers: dgleich, buda, mtomic

Reviewed By: mtomic

Subscribers: mtomic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1169
2018-02-05 09:48:50 +01:00
Marin Tomic
dca1e9eebc Add initial version of statsd
Reviewers: mferencevic, teon.banek, buda, mculinovic

Reviewed By: mferencevic

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1152
2018-02-02 17:02:37 +01:00
Marin Tomic
5a162213e5 Refactor pokec, add card fraud
Reviewers: buda, mculinovic, mferencevic

Reviewed By: mculinovic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1143
2018-02-02 13:45:18 +01:00
florijan
1d5d67aeac Refactor database::StateDelta
Summary:
Refactor in two ways. First, expose members without getters as we will
need most of them in distributed. And this was always the sensible thing
to do. Second, add storage type values to deltas. This is also a
sensible thing to do, and it will be very beneficial in distributed. We
didn't do it before because name<->value type mappings aren't guaranteed
to be the same after recovery. A task has been added to address this
(preserve mappings in durability).

Reviewers: dgleich, buda

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1167
2018-02-02 11:44:03 +01:00
Teon Banek
583f6f1794 Add Synchronize operator stub
Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1166
2018-02-02 11:21:15 +01:00
Marko Culinovic
fe556e3128 Add hop edges info in summary log
Reviewers: florijan

Reviewed By: florijan

Differential Revision: https://phabricator.memgraph.io/D1165
2018-02-01 15:00:56 +01:00
Marko Culinovic
25dae811c9 Generate distributed snapshot for card fraud demo
Reviewers: mtomic, buda, dgleich

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1146
2018-02-01 13:07:40 +01:00
Teon Banek
27a90a5c25 Support distributed CreateIndex plan
Reviewers: florijan, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1162
2018-02-01 13:05:11 +01:00
florijan
b97b48b365 Support worker transaction begin/advance/commit/abort
Reviewers: dgleich, buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1161
2018-02-01 12:12:20 +01:00
florijan
e5035cf477 Support graph elements in remote pull rpc
Reviewers: teon.banek, dgleich, msantl, buda

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1157
2018-02-01 10:53:41 +01:00
Teon Banek
abdd7d8e9d Add basic tests for distributed query plans
Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1156
2018-01-31 12:56:55 +01:00
Teon Banek
471d63ad86 Don't distribute single execution queries
Summary:
Queries such as `RETURN 1` should only be run on a single machine. This
change assumes that a query should only be distributed if it contains at
least one `ScanAll` operator, i.e. a `MATCH` clause.

Reviewers: florijan, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1154
2018-01-30 11:07:26 +01:00
florijan
3f30a6fad9 Add async remote pull RPC
Summary:
Remote pulls can now be async. Note that
`RemotePullRpcClients::RemotePull` takes references to data structures
which should not be temporary in the caller. Still, maybe safer to make
copies?

Changed `RpcWorkerClients` API to make that possible.

Reviewers: dgleich, msantl, teon.banek

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1150
2018-01-30 10:32:20 +01:00
Dominik Gleich
c37bb87ed8 Support snapshot creation and recovery in distributed
Summary:
Add custom encoder/decoder

Update snapshot recovery

Reviewers: florijan, teon.banek, mferencevic, mculinovic

Reviewed By: florijan

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1142
2018-01-29 19:16:13 +01:00
florijan
bfb3a0d9b1 Resolve global address to local
Summary:
It is possible that we have a global address to resolve, for a graph
element that's local. Consider W1 expanding, getting data from W2,
expanding from there and getting data that is on W1. We then don't want
to do RPC from W1 to W1, but do a lookup directly.

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1145
2018-01-26 16:09:19 +01:00
Dominik Gleich
24857cc1cf Support distributed (label, property) indexes
Summary:
Call workers buildindex

Merge branch 'master' into setup_distributed_index

Use ExecuteOnWorkers api

Merge branch 'master' into setup_distributed_index

Improve test

Merge branch 'master' into setup_distributed_index

Finish test

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1134
2018-01-25 18:08:04 +01:00
florijan
ccefae4002 Add distributed execution RPC
Summary:
NOTE: This diff is still in progress. Many TODOs, lacking documentation
etc. But the main logic is there (some could be different), and it tests
out OK.

Reviewers: teon.banek, msantl, buda

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1138
2018-01-25 14:50:28 +01:00
Teon Banek
dbc9c995c9 Plan distributed execution of basic read operators
Summary:
Remove AdvanceCommand operator declaration.
Add query/plan/distributed source files.
Add hacked cloning of LogicalOperator via serialization.
Add virtual Default methods to CompositeVisitor.

Use BOOST_CLASS_EXPORT_KEY in ast.hpp and operator.hpp.

This is needed in a single binary to correctly serialize polymorphic
classes. The previous implementation worked because the memgraph_lib was
linked with test binaries, but nobody was actually serializing things
inside the single memgraph binary itself.

Print PullRemote symbols in tests/manual/query_planner

Add names to implicitly created aggregation symbols

Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1137
2018-01-25 14:20:51 +01:00
Matej Ferencevic
e2f2cd5722 Improve network performance
Summary:
With this patch the number of packets for a simple RPC call is lowered
from 22 to 12 (45% reduction). The number of packets for the Bolt protocol
is lowered from 26 to 18 (30% reduction).
Impact on the Bolt protocol will be a constant of ~ 8 packets less per
connection, while the impact on the RPC protocol will be approximately
a 45% reduction overall.

Reviewers: buda, teon.banek

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1141
2018-01-25 14:04:02 +01:00
Marin Tomic
6fa06f7ea5 Add rpc client pool
Summary: See above. The unit test creates two clients on demand so I guess it works.

Reviewers: mferencevic, florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1136
2018-01-25 12:48:10 +01:00
Matej Ferencevic
010ed52622 Implement RPC call abort
Reviewers: teon.banek, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1140
2018-01-24 22:02:26 +01:00
Matej Ferencevic
fc20ddcd25 RPC refactor
Summary:
Start removal of old logic
Remove more obsolete classes
Move Message class to RPC
Remove client logic from system
Remove messaging namespace
Move protocol from messaging to rpc
Move System from messaging to rpc
Remove unnecessary namespace
Remove System from RPC Client
Split Client and Server into separate files
Start implementing new client logic
First semi-working state
Changed network protocol layout
Rewrite client
Fix client receive bug
Cleanup code of debug lines
Migrate to accessors
Migrate back to binary boost archives
Remove debug logging from server
Disable timeout test
Reduce message_id from uint64_t to uint32_t
Add multiple workers to server
Fix compiler warnings
Apply clang-format

Reviewers: teon.banek, florijan, dgleich, buda, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1129
2018-01-24 15:27:40 +01:00
Matija Santl
31aded2dae Add test for PlanDispatcher and PlanConsumer
Summary:
Added test for `PlanDispatcher` and `PlanConsumer`.
This diff also contains a fix for the async rpc call on all clients.h

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot, dgleich

Differential Revision: https://phabricator.memgraph.io/D1135
2018-01-24 12:38:52 +01:00
Marko Budiselic
142b1f42b1 Add run_pokec script and minimum refactor
Summary:
  * add run_pokec script because more than one step is required
  * refactor of plot_throughput script
  * move all plot scripts under tools/plot

Reviewers: mferencevic, teon.banek, mislav.bradac

Reviewed By: mferencevic

Subscribers: florijan, pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1106
2018-01-23 16:17:09 +01:00
Dominik Gleich
ca9fac8adc Expose worker ids
Summary: Add worker ids expose to rpc_worker_clients

Reviewers: msantl

Reviewed By: msantl

Differential Revision: https://phabricator.memgraph.io/D1128
2018-01-22 17:09:12 +01:00
Teon Banek
1d6ac3d23d Add ProduceRemote and PullRemote operator stubs
Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1124
2018-01-22 15:08:32 +01:00
florijan
e1e4a70714 Implement graph element rpc
Summary:
- End to end distributed GraphDb testing
- Refactors as necessary
- Basic RemoteCache for storing remote data
- RemoteDataRpc

As we are on a tight schedule, please let's focus on the essentials:
functionality and proper testing.

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1121
2018-01-22 14:20:41 +01:00
Teon Banek
252018ab22 Serialize query plan operators
Summary:
With the added support for serialization, we should be able to transfer
plans across distributed workers.

The planner tests has been extended to test serialization. Operators
should be mostly tested, but the expression they contain aren't
completely. The quick solution is to use typeid for rudimentary
expression equality testing.  The more involved solution of comparing
the expression tree for equality is the correct choice. It should be
done in the near future.

Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1122
2018-01-22 12:48:05 +01:00
Teon Banek
2a8c64882f Serialize SymbolTable
Summary:
Other than the plan operators and the frame, we will need to pass the
generated symbol table to distributed workers.

Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1123
2018-01-22 12:47:51 +01:00
Dominik Gleich
68a75eeca2 Add database master test
Summary: Seperate rpc server names

Reviewers: florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1118
2018-01-19 14:52:05 +01:00
florijan
9361d79c6d Implement GraphDbAccessor creation for running transaction
Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1119
2018-01-19 13:21:54 +01:00
Dominik Gleich
2a130e784e Worker id in snapshot/wal
Summary:
Adds worker id to snapshot and wal filename.
Adds a new worker_id flag to be used for recovering a worker with a distributed snapshot.
Adds worker_id field to snapshot to check for consistency.

Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1096
2018-01-18 11:46:47 +01:00
Marin Tomic
189fd75369 Change manual/raft_rpc to use persistent storage
Summary: see title

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1113
2018-01-17 18:41:01 +01:00
Teon Banek
93de41e717 Throw RecordDeletedError if updating a deleted record
Summary:
Previously, we would have a `DCHECK` which crashes the application. This
was evident when testing a queries, such as:

    MATCH (n) DELETE n SET n.prop = 42

Since the argument to update clauses is evaluated during execution, it
makes it very difficult to prevent such errors during semantic analysis.
For example:

    MATCH (n)--(m) WITH collect(n) as ns, m
    DETACH DELETE ns[m.prop] SET head(ns).prop = 42

Test query updates on deleted graph elements

Reviewers: florijan, dgleich

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1114
2018-01-17 14:30:57 +01:00
florijan
813d37e939 Migrate db::types to storage::
Reviewers: teon.banek, dgleich

Reviewed By: teon.banek, dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1110
2018-01-17 10:35:12 +01:00
Teon Banek
98cbd2b500 Cleanup utils::File API
Summary:
Close the file descriptor in File destructor. This will prevent
accidental crashes during unexpected destructor calls. For example, if
an exception is thrown before the file is closed. File now takes
ownership of the descriptor. These changes now honor RAII idiom, which
should handle most of the peculiarities of C++.

Use optional value for TryOpenFile function, instead of returning a File
without a descriptor. It makes the failure state more semantically clear
to the API user.

Merge utils/filesystem with utils/file

The files aren't that big, and the naming is a bit confusing because
functions aren't really grouped for file and filesystem distinction.

Reviewers: mferencevic, mtomic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1111
2018-01-16 15:38:12 +01:00
Marin Tomic
9e42ebbb67 Implement simple log file storage for raft
Summary:
Added wrappers for some Unix system calls in utils/filesystem.hpp and implemented
a simple log storage interface for Raft. It is not very efficient, we will need
something more sophisticated later, but this is good enough for testing.

Reviewers: mferencevic, mislav.bradac, buda, mculinovic

Reviewed By: mferencevic

Subscribers: teon.banek, dgleich, pullbot

Differential Revision: https://phabricator.memgraph.io/D1091
2018-01-15 18:07:45 +01:00
Teon Banek
a166c613ec Inline test data in the benchmark tests itself
Summary:
The directory was never actually copied on apollo, so tests weren't even
doing anything...

Also remove fswatcher unit test, it should be rewritten correctly.

Reviewers: mislav.bradac, mferencevic, buda

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1108
2018-01-15 16:12:30 +01:00
Dominik Gleich
5418dfb19e Rename NetworkEndpoint
Summary:
Rename redunant port str

Add endpoint << operator

Migrate everything to endpoint

Reviewers: mferencevic, florijan

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1100
2018-01-15 15:47:37 +01:00
Matej Ferencevic
41679b6ec5 Integrate code coverage with Apollo
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1050
2018-01-15 12:20:11 +01:00
florijan
80c5fd48e1 Reduce durability test flakyness
Summary:
Reduce flakyness by ensuring the first database gets destructed before
taking wal-file count.

Reviewers: teon.banek, mferencevic, dgleich

Reviewed By: mferencevic, dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1105
2018-01-15 11:12:05 +01:00
Marko Budiselic
4f61e6db7e Delete very old data files
Summary: Cleanup of very old data files from the tests/data folder.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1107
2018-01-14 12:55:14 +01:00
florijan
6fc6a27288 Refactor GraphDb
Summary:
GraphDb is refactored to become an API exposing different parts
necessary for the database to function. These different parts can have
different implementations in SingleNode or distributed Master/Server
GraphDb implementations.

Interally GraphDb is implemented using two class heirarchies. One
contains all the members and correct wiring for each situation. The
other takes care of initialization and shutdown. This architecture is
practical because it can guarantee that the initialization of the
object structure is complete, before initializing state.

Reviewers: buda, mislav.bradac, dgleich, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1093
2018-01-12 16:47:24 +01:00
florijan
c4327b26f4 Extract tx::SingleNodeEngine from tx::MasterEngine
Summary:
No logic changes, just split `tx::MasterEngine` into
`tx::SingleNodeEngine` and `tx::MasterEngine`. This gives better
responsibility separation and is more appropriate now there is no
Start/Shutdown.

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1099
2018-01-11 15:44:42 +01:00
Dominik Gleich
007a7f1a6d Change network design from Start/Shutdown to constructor/destructor
Summary:
Make ServerT start on constructor

Remove shutdown from MasterCoordinator

Distributed system remove Shutdown

Rcp server start and shutdown removed

Reviewers: florijan, mferencevic

Reviewed By: mferencevic

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1097
2018-01-10 14:58:57 +01:00
Matej Ferencevic
499ad3ba15 Add initial version of Apollo config files
Summary:
Improve Apollo config files

Add name to apollo_build

Remove old generate script from build

Add build_release symlink to release build

Rename 'args' to 'arguments'

Add run definition for cppcheck

Host doxygen documentation

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1095
2018-01-10 14:46:10 +01:00
florijan
87f8c4e256 Fix durability test flakyness (hopefully)
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1098
2018-01-10 14:24:40 +01:00
florijan
3cf81380b8 Implement vertex/edge serialization
Summary:
Serialization of vertices and edges for distributed. Based on Boost
serialization. Threrefore moved TypedValue serialization from AST to
utils.

Reviewers: buda, dgleich, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1088
2018-01-10 13:32:18 +01:00
florijan
4a0345e1c5 Prepare counter for distributed
Reviewers: dgleich, teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1090
2018-01-10 13:28:41 +01:00
Marko Culinovic
7a06040a76 Setup RAFT test infrastructure
Summary:
`JailService` exposes jail API and knows how to
start and stop binaries. Every test should be
defined as python module with exposed
`run` method. Script `master.py` is used
for running tests and takes test module
name as argument. Machine IP addresses are
defined in environment variables. To run test
locally use `local_runner` script.

Reviewers: mislav.bradac, mferencevic, mtomic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1065
2018-01-05 09:59:42 +01:00
florijan
99000c6ec1 Remove PropertyValueStore templatization
Summary:
A PropertyValueStore is not a generic data structure, but only ever used
to store properties in a Vertex/Edge. It has behaviours specific to it.
So, the templatization was not necessary.

Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1089
2018-01-02 16:18:55 +01:00
Matej Ferencevic
82bede9a97 Implement measurements collection in tests
Summary:
Implement measurements collection in QA

Implement measurements collection in LDBC

Implement measurements collection in stress tests

Reviewers: mislav.bradac, buda, florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1087
2018-01-01 17:49:37 +01:00
Matej Ferencevic
859641cb0c Changed macro benchmark summary format
Reviewers: buda, mtomic, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: florijan, pullbot

Differential Revision: https://phabricator.memgraph.io/D972
2017-12-28 16:35:45 +01:00
Dominik Gleich
503381549e Change gid bit size
Summary:
Change gid methods

Rename GidGenerator and add tests

Fix tools broken by gid changes

Reviewers: dgleich, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1044
2017-12-28 11:04:52 +01:00
Mislav Bradac
d3623585e7 Migrate cereal to boost_serialization
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1077
2017-12-27 13:44:46 +01:00
florijan
9b878c91eb Make interpretation pullable
Reviewers: mislav.bradac, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1079
2017-12-22 15:19:05 +01:00
Dominik Gleich
caaab41108 Add new snapshot format support
Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1082
2017-12-22 15:16:53 +01:00
Marin Tomic
e999207b2f Remove reactors
Summary: kill reactors

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Differential Revision: https://phabricator.memgraph.io/D1075
2017-12-20 17:56:57 +01:00
Dominik Gleich
1556d78d15 Update snapshot format
Summary:
Set vertex/edge generator id from recovery

Add tests

Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1061
2017-12-20 16:57:42 +01:00
Marin Tomic
20aa8d563a Implement RaftNetworkInterface via RPC
Summary:
Add missing fields in RequestVote RPC

Unnest Raft RPC requests and responses

Make RPC request and reply serializable

Refactor Raft network interface

Implement RPC retrying with backoff period

Add cerealization functions for optional

Add RPC network interface for raft

Add some comments and documentation

Add manual test for RPC network interface

Remove raft reactor experiments

Reviewers: mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot, mculinovic

Differential Revision: https://phabricator.memgraph.io/D1066
2017-12-20 16:48:46 +01:00
Teon Banek
5f7837d613 Serialize Ast classes
Summary:
Although the first solution used cereal, the final implementation uses
boost. Since the cereal is still used in the codebase, compilation has
been modified to support multithreaded cereal.

In addition to serializing Ast classes, the following also needed to be
serialized:

  * GraphDbTypes
  * Symbol
  * TypedValue

TypedValue is treated specially, by inlining the serialization code in
the Ast class, concretely PrimitiveLiteral.

Another special case was the Function Ast class, which now stores a
function name which is resolved to a concrete std::function on
construction.

Tests have been added for serialized Ast in
tests/unit/cypher_main_visitor

Reviewers: mferencevic, mislav.bradac, florijan

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1067
2017-12-20 13:28:34 +01:00
florijan
3ae45e0d19 Add master/worker flags, main functions and coordination
Reviewers: dgleich, mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1060
2017-12-19 16:05:57 +01:00
florijan
37722e54b3 Add RPC to concurrent ID mapper
Summary:
The distributed ID mapper is not yet utilised in GraphDb as those
changes are in D1060. Depending on landing order it will be added.

Reviewers: dgleich, mislav.bradac

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1064
2017-12-19 09:11:16 +01:00
Dominik Gleich
0eb30db8db Remove executor not run option
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1062
2017-12-18 14:36:15 +01:00
Marin Tomic
5b7cb4602c Add simple client facing API for RaftMember
Summary:
Implement AddComand method on RaftMember

Move RPCType out of rpc request and reply

Add unit test for AddCommand

Reviewers: mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot, mculinovic

Differential Revision: https://phabricator.memgraph.io/D1042
2017-12-18 07:32:39 +01:00
Marko Culinovic
0cc1ac92b0 Add TED Talk example to quick start
Reviewers: teon.banek, florijan, mtomic, buda

Reviewed By: teon.banek, buda

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1022
2017-12-14 13:48:07 +01:00
Dominik Gleich
0f1e8aa763 Add rpc server local thread
Summary: Update tests for new rpc server

Reviewers: mislav.bradac, florijan, mferencevic

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1057
2017-12-14 13:37:32 +01:00
florijan
52b3941c8a Fix tx::WorkerEngine flakyness. Change Endpoint::port type.
Reviewers: mislav.bradac, dgleich

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1054
2017-12-14 10:36:24 +01:00
Matija Santl
2758b35c08 Fix semicolon in column name
Summary:
Special casing the last token when adding it to the named expression list.
Added a test to check this case.

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1052
2017-12-14 09:44:21 +01:00
Dominik Gleich
a8488ac497 Rpc client locking
Summary:
Rpc client wasn't thread safe and required a lock before each rpc call.
The locking functionality is now incorporated in Rpc client.

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Differential Revision: https://phabricator.memgraph.io/D1056
2017-12-13 17:37:32 +01:00
Matija Santl
4da8312abe Fix query returning a map
Summary: Add curvy braces handling in `QueryStripper` and an accompanying test for it.

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1051
2017-12-13 15:38:20 +01:00
Mislav Bradac
cf7d9070d8 Fix flakyness in rpc/messaging tests
Summary: In this diff I just wanted to fix tests' flakyness. We can discuss if we want to always pass endpoint as an argument and never pass address:port pair explicitly. However if we decide that, I will do that change in another diff.

Reviewers: dgleich, florijan

Reviewed By: dgleich, florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1049
2017-12-13 13:15:02 +01:00
Matija Santl
9ad7d82a54 Implement UNION query combinator
Summary:
Union query combinator implementation consists of:
 * adjustments to the AST and `cypher_main_visitor`
 * enabling `QueryStripper` to parse multiple `return` statements (not stopping after first)
 * symbol generation for union results
 * union logical operator
 * query plan generator adjustments

Reviewers: teon.banek, mislav.bradac

Reviewed By: teon.banek

Subscribers: pullbot, buda

Differential Revision: https://phabricator.memgraph.io/D1038
2017-12-13 09:54:00 +01:00
Mislav Bradac
eb272f0b67 Assert on endpoint failures
Summary: .

Reviewers: mferencevic, florijan

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1045
2017-12-12 14:27:24 +01:00
Marin Tomic
6b4a2cafc5 Fix OnRequestVote RPC callback
Summary: RPC recipient should update its term even if it is rejecting the request.

Reviewers: mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1039
2017-12-11 12:53:17 +01:00
florijan
4982d45e27 Add RPC to distributed tx::Engine
Reviewers: mislav.bradac, dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1035
2017-12-11 11:05:01 +01:00
Marin Tomic
0bf692f8ec Refactor Raft leader election to use multiple threads
Summary:
Implement log replication

Rebase and fix src/CMakeLists.txt

Some style fixes

Changed shared_ptr to unique_ptr for RaftPeerState

Change Id and Leader to const

Move implementation to separate class

Fix raft_experiments.cpp

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1033
2017-12-08 16:30:57 +01:00