Commit Graph

2101 Commits

Author SHA1 Message Date
Matej Ferencevic
125502cbb5 Reduce telemetry test flakiness
Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1666
2018-10-18 14:29:13 +02:00
Matej Ferencevic
9a7d25f6a5 Fix distributed startup and QA test
Reviewers: msantl, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1674
2018-10-18 10:34:05 +02:00
Marko Culinovic
93267531be Improve mg_client
Summary:
 - Set query time to the milisecond precision.
 - Print 'Bye' when `EOF` is received in interactive mode.
 - In non-interactive mode output exit on first failed query and
   print the query along with the error message.
 - Don't quote stings when they are alone

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1670
2018-10-18 10:09:26 +02:00
Matej Ferencevic
f7d1050a9d Remove ConcurrentMap from RPC
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1662
2018-10-18 10:04:55 +02:00
Teon Banek
0829b2bb90 Copy accumulated elements in Synchronize
Summary:
The row elements must not be moved, because the following pulls may
still use them.

Reviewers: mtomic, llugovic, mferencevic

Reviewed By: llugovic, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1672
2018-10-18 10:03:25 +02:00
Matej Ferencevic
18eee6337c Fix include in wal.cpp
Summary: easy

Reviewers: ipaljak, vkasljevic

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1671
2018-10-17 17:02:13 +02:00
Teon Banek
bdc2e8d647 Bump version to 0.13 and update changelog
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1668
2018-10-17 10:56:08 +02:00
Matej Ferencevic
f1bae7be44 Reduce mg_import_csv memory usage
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1663
2018-10-16 14:22:14 +02:00
Matija Santl
fb6284cb99 Add unique constraint to label property index
Summary:
LabelPropertyIndex now has the ability to enforce unique constraint.

This doesn't lock the tx engine.

Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot, vkasljevic, buda

Differential Revision: https://phabricator.memgraph.io/D1660
2018-10-16 14:20:47 +02:00
Matej Ferencevic
cdaf7581bf Add explicit start to servers
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1661
2018-10-16 11:39:42 +02:00
Marin Tomic
285e02d5ec Remove root tracking from AST storage
Summary: Up till now, `AstStorage` also took care of tracking the root of the `Query` and loading of cloning of `Query` nodes would change that root. This felt out of place because sometimes `AstStorage` is used only for storing expressions, and we don't even have an entire query in the storage. This diff removes that feature from `AstStorage`. Now its only functionality is owning AST nodes and assigning unique IDs to them.

Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1646
2018-10-16 10:22:21 +02:00
Matej Ferencevic
baae40fcc6 Move RPC server to Coordination
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1658
2018-10-16 09:12:37 +02:00
Marko Culinovic
c553a309d2 New client features: non-interactive shell, shell commands, csv format
Summary:
If the `STDIN_FILENO` is a `TTY`, then an interactive command prompt
is shown. Otherwise, client runs in non-interactive mode.
Shell commands start with a colon sign, and end at the end of
line.
Csv and tabular output formats are supported.

Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1633
2018-10-16 09:04:34 +02:00
Matej Ferencevic
abef91616d Remove leftover traces of user docs
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1650
2018-10-15 18:40:47 +02:00
Lovro Lugovic
a6a621e08b Add LCP tests
Summary: Create an ASDF system for LCP. Add LCP tests.

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1645
2018-10-15 11:52:20 +02:00
Teon Banek
8c2ca44cb9 Split manual/query_planner for distributed and single
Reviewers: mtomic, llugovic, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1644
2018-10-12 14:02:47 +02:00
Teon Banek
1b6a6c15e5 Add syntax for creating unique indexes
Reviewers: mtomic, llugovic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1653
2018-10-12 12:50:58 +02:00
Matej Ferencevic
b527b2b4e4 Refactor PropertyValue
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1654
2018-10-12 11:10:29 +02:00
Lovro Lugovic
4b5c0d3426 Implement coalesce as a special operator
Reviewers: teon.banek, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1652
2018-10-11 16:25:26 +02:00
Matija Santl
cbe100ef5a Fix Edges filter by vertex
Summary:
This diff fixes the storage part of the bug that happen for the
following queries if all nodes end up on the non-master machine.
```
CREATE (a:T{c: True})-[:X{x: 2.5}]->(:A:B)-[:Y]->()-[:Z{r: 1}]->(a)

MATCH (:T{c: True})-[a:X{x: 2.5}]->(node:A:B)-[:Y]->()-[:Z{r: 1}]->() RETURN a AS node, node AS a
```

The current state of the code doesn't work completely, it's still missing the
query execution fix.

Reviewers: teon.banek, ipaljak, vkasljevic

Reviewed By: teon.banek

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D1629
2018-10-11 15:29:31 +02:00
Lovro Lugovic
467e46c302 Add inDegree and outDegree functions
Reviewers: teon.banek, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1651
2018-10-11 15:05:39 +02:00
Matej Ferencevic
6525451489 Remove distributed logic from single node
Summary:
Start removing `is_remote` from `Address`
Remove `GlobalAddress`
Remove `GlobalizedAddress`
Remove bitmasks from `Address`
Remove `is_local` from `Address`
Remove `is_local` from `RecordAccessor`
Remove `worker_id` from `Address`
Remove `worker_id` from `GidGenerator`
Unfriend `IndexRpcServer` from `Storage`
Remove `LocalizedAddressIfPossible`
Make member private
Remove `worker_id` from `Storage`
Copy function to ease removal of distributed logic
Remove `worker_id` from `WriteAheadLog`
Remove `worker_id` from `GraphDb`
Remove `worker_id` from durability
Remove nonexistant function
Remove `gid` from `Address`
Remove usage of `Address`
Remove `Address`
Remove `VertexAddress` and `EdgeAddress`
Fix Id test
Remove `cypher_id` from `VersionList`
Remove `cypher_id` from durability
Remove `cypher_id` member from `VersionList`
Remove `cypher_id` from database
Fix recovery (revert D1142)
Remove unnecessary functions from `GraphDbAccessor`
Revert `InsertEdge` implementation to the way it was in/before D1142
Remove leftover `VertexAddress` from `Edge`
Remove `PostCreateIndex` and `PopulateIndexFromBuildIndex`
Split durability paths into single node and distributed
Fix `TransactionIdFromWalFilename` implementation
Fix tests
Remove `cypher_id` from `snapshooter` and `durability` test

Reviewers: msantl, teon.banek

Reviewed By: msantl

Subscribers: msantl, pullbot

Differential Revision: https://phabricator.memgraph.io/D1647
2018-10-11 13:08:22 +02:00
Matej Ferencevic
4c03cb615e Remove user technical docs from daily build
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1648
2018-10-10 16:28:15 +02:00
Marin Tomic
695129f401 Separate query types in openCypher grammars
Summary: depends on D1640

Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1643
2018-10-10 15:08:48 +02:00
Teon Banek
ca2106bb91 Handle transitive dependencies for Cartesian
Reviewers: mtomic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1641
2018-10-10 12:55:05 +02:00
Marin Tomic
ce6085fa06 Fix pointer tracking in AST serialization
Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1640
2018-10-10 11:24:39 +02:00
Teon Banek
e4726f54ec Move distributed tests from query_planner to distributed_query_plan
Reviewers: mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1638
2018-10-09 17:17:01 +02:00
Lovro Lugovic
5bdb04a693 Add move constructor and assignment operator to TypedValue
Summary:
`heaptrack` shows a miniscule decrease in memory usage during query
execution.

Running the below query on the TEDTalk database 100 times gives the
following results:

- number of allocations: from 642647 to 642589
- bytes allocated in total: from 48.79 MiB to 48.78 MiB

```
MATCH (t:Tag)<-[:HasTag]-(n:Talk)
RETURN t.name AS Tag, COUNT(n) AS TalksCount
ORDER BY TalksCount DESC, Tag LIMIT 20;
```

Regarding `TypedValue`'s destructor: we're allowed to manually destruct the
union memebers that we constructed using placement-new. However, it is undefined
behavior to call the destructor after an object's lifetime has ended. Calling
`TypedValue`'s own destructor within its assignment operator counts as ending
its lifetime, which means that the next call to its destructor will invoke
undefined behavior.

See the C++ Draft N4140, clauses 12.4/15 and 3.8/1.3.

Reviewers: teon.banek, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1630
2018-10-09 14:49:09 +02:00
Matej Ferencevic
4e5fe37dd6 Remove virtual and pimpl from single node
Summary:
This diff removes: `SingleNodeRecoveryTransactions`, `TypemapPack`
It also removes virtual and/or pimpl from: `SingleNodeCounters`,
`StorageGcSingleNode`, `SingleNodeConcurrentIdMapper`,
accessors (revert D1510), transaction engine, `GraphDbAccessor`, `GraphDb`

Reviewers: msantl, teon.banek

Reviewed By: msantl, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1639
2018-10-09 11:48:30 +02:00
Marko Budiselic
7b70d126f6 Remove docs/user_technical
Summary:
The new source of truth is https://github.com/memgraph/docs
because content writers and community members will write
most of the content.

Reviewers: ipaljak, teon.banek

Reviewed By: ipaljak

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1634
2018-10-09 11:32:17 +02:00
Matej Ferencevic
5097c10ba8 Separate distributed from single node transaction engine
Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1637
2018-10-05 14:11:39 +02:00
Matej Ferencevic
ade2593b51 Separate distributed from single node GraphDb
Summary:
To clean the working directory after this diff you should execute:

```
rm src/database/counters_rpc_messages.capnp
rm src/database/counters_rpc_messages.hpp
rm src/database/serialization.capnp
rm src/database/serialization.hpp
```

Reviewers: teon.banek, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1636
2018-10-05 13:32:05 +02:00
Matija Santl
fdf609b9d8 Move distributed.md to feature_spec folder
Summary:
The diff where I added `distributed.md` was a bit old. When landing
this, I didn't notice the `feature_specs` folder was renamed to `feature_spec`.
This fixes this.

Reviewers: ipaljak

Reviewed By: ipaljak

Differential Revision: https://phabricator.memgraph.io/D1635
2018-10-05 10:33:49 +02:00
Matija Santl
56a5c61732 Distributed durability feature spec
Summary: Notes from the meeting with @mferencevic.

Reviewers: mferencevic, buda

Reviewed By: buda

Subscribers: buda, mferencevic

Differential Revision: https://phabricator.memgraph.io/D1585
2018-10-05 10:25:04 +02:00
Marin Tomic
343fe54f2c Move 'explain' to Cypher.g4 and add it to keyword list
Summary: Move explain to Cypher.g4 so MemgraphCypher contains only enterprise features. Also added `EXPLAIN` to keyword list because query `MATCH (explain) RETURN explain` wasn't working.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1632
2018-10-05 10:09:50 +02:00
Teon Banek
9071552ce1 Plan Cartesian after a merge point on master plan
Summary:
This should fix the issue of having a write followed by a read during
distributed execution. Any kind of merger of plans should behave like a
Cartesian with regards to planning the following ScanAll.

Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1624
2018-10-05 10:09:53 +02:00
Matej Ferencevic
75950664a7 Separate distributed from single node storage
Summary:
This diff splits single node and distributed storage from each other.
Currently all of the storage code is copied into two directories (one single
node, one distributed).  The logic used in the storage implementation isn't
touched, it will be refactored in following diffs.

To clean the working directory after this diff you should execute:
```
rm database/state_delta.capnp
rm database/state_delta.hpp
rm storage/concurrent_id_mapper_rpc_messages.capnp
rm storage/concurrent_id_mapper_rpc_messages.hpp
```

Reviewers: teon.banek, buda, msantl

Reviewed By: teon.banek, msantl

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1625
2018-10-05 09:19:33 +02:00
Teon Banek
96537eb181 Extract AST serialization to a new file
Reviewers: mtomic, mferencevic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1631
2018-10-04 14:33:45 +02:00
Marin Tomic
db52b35ab6 Convert AST to LCP
Reviewers: teon.banek, llugovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1628
2018-10-04 12:38:29 +02:00
Ivan Paljak
ed335e38ff Fix naming in how-to guides
Reviewers: buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1627
2018-10-02 16:49:22 +02:00
Ivan Paljak
43436c2534 Add security reference to user_technical
Reviewers: mferencevic, buda

Reviewed By: mferencevic

Differential Revision: https://phabricator.memgraph.io/D1621
2018-10-02 15:08:53 +02:00
Marko Culinovic
2bd0822155 Implement console client for memgraph
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek, mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1616
2018-10-02 13:18:54 +02:00
Lovro Lugovic
8fb7ccf6c9 RTrim the explain string before splitting on newlines
Reviewers: teon.banek, mtomic

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1620
2018-10-02 08:27:12 +02:00
Marin Tomic
6894e2aef8 Distributed BFS filter lambda
Summary:
TODO:

~~1. Figure out how to propagate exceptions during lambda evaluation to master.~~

~~2. Make some more complicated test cases to see if everything is~~
~~sent over the network properly (lambdas depending on frame, evaluation context).~~

~~3. Support only `GraphView::OLD`.~~

4. [MAYBE] Send only parts of the frame necessary for lambda evaluation.

~~5. Fix EdgeType handling~~

--------------------

Serialize frame and send it in PrepareForExpand RPC

Move Lambda out of ExpandVariable

Send symbol table and filter lambda in CreateBfsSubcursor RPC

Evaluate filter lambda during the expansion

Send evaluation context in CreateBfsSubcursor RPC

Reviewers: teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1600
2018-10-01 11:00:14 +02:00
Ivan Paljak
f53343a415 Fix naming conventions in user_technical
Summary:
* Prefix filenames with order in which they should appear in the sidebar.
* Fix links

Reviewers: buda

Reviewed By: buda

Differential Revision: https://phabricator.memgraph.io/D1618
2018-09-28 11:40:32 +02:00
Teon Banek
1fd9a72e10 Generate Load functions from LCP as top level
Summary: Depends on D1596

Reviewers: mtomic, msantl

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1601
2018-09-28 10:34:20 +02:00
Teon Banek
a5926b4e0f Generate Save functions from LCP as top level
Summary:
This should allow us to more easily decouple the code which should be
open sourced. Unfortunately, the downside of this approach is that we
cannot rely on virtual calls to dispatch the serialization to correct
type. Another downside is that members need to be publicly accessible
for serialization.

Reviewers: mtomic, msantl

Reviewed By: mtomic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1596
2018-09-28 10:26:56 +02:00
Teon Banek
d78022470c Explicitly list all test files in CMakeLists
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1619
2018-09-28 10:18:04 +02:00
Matej Ferencevic
53c405c699 Throw exceptions on RPC failure and Distributed error handling
Summary:
This diff changes the RPC layer to directly return `TResponse` to the user when
issuing a `Call<...>` RPC call. The call throws an exception on failure
(instead of the previous return `nullopt`).

All servers (network, RPC and distributed) are set to have explicit `Shutdown`
methods so that a controlled shutdown can always be performed. The object
destructors now have `CHECK`s to enforce that the `AwaitShutdown` methods were
called.

The distributed memgraph is changed that none of the binaries (master/workers)
crash when there is a communication failure. Instead, the whole cluster starts
a graceful shutdown when a persistent communication error is detected.
Transient errors are allowed during execution. The transaction that errored out
will be aborted on the whole cluster. The cluster state is managed using a new
Heartbeat RPC call.

Reviewers: buda, teon.banek, msantl

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1604
2018-09-27 16:27:40 +02:00
llugovic
13529411db Remove Boost from CMakeLists.txt
Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1615
2018-09-27 15:16:40 +02:00