Commit Graph

588 Commits

Author SHA1 Message Date
Mislav Bradac
cf7190ecc6 Refactor long running benchmark
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D801
2017-09-20 10:36:51 +02:00
florijan
d6a885cce6 Traversal API removed
Summary: Removed the traversal API, been waiting for named path to land because that data structure has replaced traversal's Path. I left the wiki page of the API, just put a warning that it's not used.

Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D809
2017-09-20 10:30:45 +02:00
florijan
9edc472eaf Resolving TODOs in the operator implementations
Summary:
Three TODOs resolved.

1. around line 897 - we currently don't support expansion into existing variable length edges (there is a TODO in symbol_generator.cpp:213), so this should not be done at the moment.

2. around line 1025 - This TODO was on review and nobody commented, so I'm removing it. Should have done that when the diff landed.

3. around line 1560 - This does not seem possible. Edge-uniqueness checks happen within a single `[OPTIONAL ] MATCH`. If it is OPTIONAL (the case interesting here), then the uniqueness check also gets planned under the optional branch. So, if an optional fails, the uniqueness check will get skipped, as opposed to getting executed over a Null. I added an edge-case test to verify this (and checked with the planner test).

Reviewers: buda, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D807
2017-09-19 09:48:28 +02:00
florijan
347611edfd Named path support
Reviewers: buda, teon.banek, mislav.bradac

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D794
2017-09-19 09:45:50 +02:00
Teon Banek
a2c56dd83e Build dataset and convert to neo4j and memgraph formats
Reviewers: mferencevic, buda

Reviewed By: mferencevic

Differential Revision: https://phabricator.memgraph.io/D808
2017-09-18 17:17:44 +02:00
Teon Banek
212f6cfa98 manual.query_planner: Print Produce symbols
Reviewers: florijan, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D792
2017-09-18 15:07:10 +02:00
Matej Ferencevic
2603209849 Replaced Python long running test with C++ version
Reviewers: buda, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D802
2017-09-18 15:06:22 +02:00
Matej Ferencevic
cd1892acc4 Rewritten long running test in C++.
Reviewers: florijan, mislav.bradac

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D784
2017-09-18 09:49:32 +02:00
florijan
48e446688f IndexInfo function added
Summary:
- Keys() functions in the indices can't be const because ConcurrentMap doesn't provide const accessors (and they are broken in skiplist) :D
- no cucumber tests because many tests create indices so it's hard to say what's inside and what not

Reviewers: buda, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D797
2017-09-16 14:09:35 +02:00
Matej Ferencevic
a14f8c6516 Removed buggy check from QA
Reviewers: buda, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D796
2017-09-15 16:24:51 +02:00
Teon Banek
b31478dfd8 Look into parameter value when estimating plan cost
Summary: Test cost estimator considers parameter values

Reviewers: florijan, mislav.bradac

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D798
2017-09-15 16:14:57 +02:00
Mislav Bradac
7e5dddecae Move harness stuff to parent directory
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D799
2017-09-15 14:05:26 +02:00
Mislav Bradac
d640ca3f1a Do preprocess in client
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D793
2017-09-15 13:13:39 +02:00
florijan
59b9b7af21 CounterSet function added
Reviewers: buda, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D795
2017-09-14 18:18:20 +02:00
Matej Ferencevic
84b0d03a5f Added max memory usage to harness
Reviewers: buda, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D790
2017-09-13 22:13:22 +02:00
Teon Banek
d116e0deb1 Generate ParameterLookup instead of PrimitiveLiteral
Summary:
This is done when the generated AST will be cached.
Remove LiteralsPlugger.

Reviewers: florijan, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D788
2017-09-13 17:49:34 +02:00
florijan
bfbec8d550 Counter function added
Reviewers: buda, mferencevic, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D787
2017-09-13 17:09:12 +02:00
Matej Ferencevic
9d63dfa9ca Added slave groups to Apollo generate.
Summary: Added large stress to daily release.

Reviewers: buda, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D786
2017-09-13 17:02:41 +02:00
Matej Ferencevic
90e95d344d Increased qa run wait time
Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D789
2017-09-13 16:53:44 +02:00
Teon Banek
a83bea0b74 Add ParameterLookup to AST
Reviewers: florijan, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D782
2017-09-13 14:28:48 +02:00
Mislav Bradac
3790fd252c Change MAX_RETRIES in query_client
Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D785
2017-09-13 14:19:03 +02:00
florijan
36fb7f9554 PropertyValueStore - remove Accept function (never used, never will be)
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D781
2017-09-13 10:11:55 +02:00
florijan
00723d34c3 Harness - match group setup fix
Summary: I was mistaken in my calculations before and gave it +-3sigma tolerance (0.0027 probability of failure). Now I changed it to +-5sigma, which is good enough for CERN, and should be for us too.

Reviewers: mislav.bradac, mferencevic

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D780
2017-09-13 10:11:30 +02:00
Mislav Bradac
7e99e93e47 Start work on parallel benchmark
Summary: First version of our benchmark

Reviewers: florijan, buda

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D740
2017-09-12 16:58:21 +02:00
Teon Banek
4601f6c368 Add filtering BFS by edge type
Summary:
Antlr grammar has been updated to support putting edge types after the BFS
symbol. Planner collects edge type filters for BFS and inlines them in the
operator by joining the filter with the user input BFS filter itself. This
requires no change from the standpoint of the operator. On the other hand, in
order to use the faster lookup by a single edge type, `ExpandBreadthFirst`
operator now accept an optional edge type. The edge type is passed from the
planner only if the user is filtering by a single type.

Unit tests as well as tck have been updated.

Reviewers: florijan, mislav.bradac

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D777
2017-09-12 11:29:38 +02:00
Teon Banek
35f726dfd2 Use EdgeType in Expand and ExpandVariable
Summary:
Add function First to utils.
Insert EdgeType into Expand during planning.

Reviewers: florijan, mislav.bradac

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D769
2017-09-11 11:16:04 +02:00
Mislav Bradac
74d0a426ab Use \u for utf16 and \U for utf32
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D773
2017-09-09 16:16:30 +02:00
Teon Banek
92b9bbd4bd Remove const requirement from DbAccessor in planning
Summary: The constness of the DbAccessor interferes with caching the results.

Reviewers: florijan, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D771
2017-09-09 14:06:32 +02:00
Teon Banek
0aee18544d Add caching VerticesCount during planning and estimation
Summary:
Benchmark planning and estimating indexed ScanAll. According to the benchmark,
caching speeds up the whole process of planning and estimation by a factor of
2. Most of the performance gain is in the `CostEstimator` itself, due to plenty
of calls to `VerticesCount` when estimating all of the generated plans.

Reviewers: mislav.bradac, florijan

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D765
2017-09-09 12:26:48 +02:00
Teon Banek
f0b1f24006 unit.recovery: Read vertex IDs into int64_t, not int
Summary:
This fixes the problem when the test would fail on some PCs...
Notably, mine...

Reviewers: florijan, mferencevic, mislav.bradac, buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D770
2017-09-09 09:24:42 +02:00
Matej Ferencevic
541a0dae24 Addded query error message to harness client.
Reviewers: florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D766
2017-09-08 15:30:58 +02:00
florijan
2c4966edfe Vertex edges data structure introduced
Summary:
- The `Edges` data structure now handles common ops, including providing an iterator over edges whose "other" vertex is know.
- This should improve performance on dense_expand tests in the harness without other side-effects.
- query::plan::Expand operator modified not to check for existing-node stuff since that now gets handled by the `Edges` data structure.
- `Edges::Iterator` implemented only for const iterators since that suffices for now. Can implement non-const if the need arrises.

Reviewers: buda, mislav.bradac, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D763
2017-09-08 13:21:21 +02:00
florijan
9f8361dd27 Edge storage benchmark added
Summary: For the time being we'll go with vector and a custom iterator for know-destination-vertex lookups, but this benchmark might be handy to keep for future work.

Reviewers: buda, mislav.bradac, teon.banek

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D762
2017-09-08 13:12:20 +02:00
florijan
6edf2cc5ab BFS feature test that uses edges added
Reviewers: buda, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D768
2017-09-08 13:03:57 +02:00
Matej Ferencevic
21782901c1 Added 3 iterations to macro benchmark.
Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D759
2017-09-08 11:23:25 +02:00
Teon Banek
c8d34c58ef Generate indexed ScanAll to make Expand existing
Summary: Test generating ScanAll before expand to use existing

Reviewers: florijan, mislav.bradac

Reviewed By: florijan

Subscribers: buda, lion, pullbot

Differential Revision: https://phabricator.memgraph.io/D746
2017-09-08 09:50:18 +02:00
florijan
63966c15a6 LDBC Query1 converted
Summary: Ugly formatting and bad variable naming due to 1024 limit on neo4j client.

Reviewers: buda, teon.banek

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D748
2017-09-08 09:30:22 +02:00
Mislav Bradac
38c83d80cb Rename Executioner to Executor
Reviewers: buda, florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D764
2017-09-07 15:13:17 +02:00
florijan
be16409da2 Harness MATCH tests refactored
Summary:
1. Test setup rewritten to take cca 8 seconds. Note that edges are created by using:
`MATCH (a) WITH a MATCH (b) WITH b WHERE rand() < X CREATE (a)-[:ET]->(b)`
Where `X` is a threshold calculated so the desired edge count is the expectation. This seems the only feasable way of generating a large number of edges since query execution does not depend on edge count, but on vertex count.

2. Using the new `assert` function to verify graph state. I recommend doing that in all the harness tests (I don't think we currently have something better).

3. All tests rewritten to take around 200ms per iteration.

4. Test are using SKIP to avoid sending data to the client, but ensure that appropriate operations get executed. This currently seems like the best way of removing unwanted side-effects.

Harness will cost us our sanity. And it doesn't even provide good quality regression testing we really need :(

Reviewers: buda, mislav.bradac, mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D752
2017-09-07 12:05:05 +02:00
Teon Banek
1693530d92 ldbc: Separate running short reads from updates
Reviewers: buda

Reviewed By: buda

Subscribers: mferencevic, pullbot

Differential Revision: https://phabricator.memgraph.io/D758
2017-09-07 10:45:09 +02:00
Matej Ferencevic
f848394e5d Fixed bolt chunked encoder buffer for big data.
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D756
2017-09-06 15:36:24 +02:00
Matej Ferencevic
642c2f07bb Added PostgreSQL support to harness client.
Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D753
2017-09-06 15:22:25 +02:00
florijan
cd0d8eb543 Harness - dense expand group added, more tests on bench
Summary: It turns out trivial if I use unwind for vertex creation, MATCH for edge creation and UNWIND for test duration. It took hours to converge to this :(

Reviewers: mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D747
2017-09-06 14:58:50 +02:00
Matej Ferencevic
8cf69738bf Fixed bolt decoder chunked buffer implementation.
Summary:
Bolt client now uses new chunked decoder.

Fixed bolt session to use new chunked decoder.

Fixed chunked decoder buffer test.

Fixed bolt session test.

Removed debug message from client.

Fixed bolt encoder to comply with specification.

Reviewers: buda, mislav.bradac

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D751
2017-09-06 13:19:42 +02:00
florijan
4975743f35 Assert function added
Summary: `assert` function added. Useful to us for asserting DB state in harness tests. Potentially useful to the client for breaking out of a query as soon as a predicate fails, as opposed to collecting result and checking them client-side.

Reviewers: buda, mislav.bradac, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D750
2017-09-05 13:57:56 +02:00
Matej Ferencevic
70d9f3f6f1 Refactored harness and added PostgreSQL support.
Summary:
Moved Neo4j config to config dir.

Neo4j and PostgreSQL are now downloaded to libs.

Renamed metadata flags in memgraph.

Changed apollo generate for new harness.

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D741
2017-09-05 09:45:41 +02:00
Marko Budiselic
b30e3252e2 A python script that visualizes latency results from LDBC.
Summary:
  * extension of run_benchmark script
  * tools/plot_latency.py

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: mislav.bradac, pullbot

Differential Revision: https://phabricator.memgraph.io/D743
2017-09-04 16:36:42 +02:00
Mislav Bradac
c56621682e Parse utf16 surrogate codepoints correctly
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D745
2017-09-04 16:03:45 +02:00
Teon Banek
d584a7b18b Test ExpandVariable reverses on Direction::BOTH
Summary: Add `is_flipped` to `Expansion`

Reviewers: florijan, mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D744
2017-09-04 15:35:17 +02:00
florijan
f68bac922f Map indexing added
Reviewers: mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D739
2017-09-04 09:12:46 +02:00