Summary:
- Keys() functions in the indices can't be const because ConcurrentMap doesn't provide const accessors (and they are broken in skiplist) :D
- no cucumber tests because many tests create indices so it's hard to say what's inside and what not
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D797
Summary:
This is done when the generated AST will be cached.
Remove LiteralsPlugger.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D788
Summary: I was mistaken in my calculations before and gave it +-3sigma tolerance (0.0027 probability of failure). Now I changed it to +-5sigma, which is good enough for CERN, and should be for us too.
Reviewers: mislav.bradac, mferencevic
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D780
Summary:
Antlr grammar has been updated to support putting edge types after the BFS
symbol. Planner collects edge type filters for BFS and inlines them in the
operator by joining the filter with the user input BFS filter itself. This
requires no change from the standpoint of the operator. On the other hand, in
order to use the faster lookup by a single edge type, `ExpandBreadthFirst`
operator now accept an optional edge type. The edge type is passed from the
planner only if the user is filtering by a single type.
Unit tests as well as tck have been updated.
Reviewers: florijan, mislav.bradac
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D777
Summary:
Add function First to utils.
Insert EdgeType into Expand during planning.
Reviewers: florijan, mislav.bradac
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D769
Summary: The constness of the DbAccessor interferes with caching the results.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D771
Summary:
Benchmark planning and estimating indexed ScanAll. According to the benchmark,
caching speeds up the whole process of planning and estimation by a factor of
2. Most of the performance gain is in the `CostEstimator` itself, due to plenty
of calls to `VerticesCount` when estimating all of the generated plans.
Reviewers: mislav.bradac, florijan
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D765
Summary:
This fixes the problem when the test would fail on some PCs...
Notably, mine...
Reviewers: florijan, mferencevic, mislav.bradac, buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D770
Summary:
- The `Edges` data structure now handles common ops, including providing an iterator over edges whose "other" vertex is know.
- This should improve performance on dense_expand tests in the harness without other side-effects.
- query::plan::Expand operator modified not to check for existing-node stuff since that now gets handled by the `Edges` data structure.
- `Edges::Iterator` implemented only for const iterators since that suffices for now. Can implement non-const if the need arrises.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D763
Summary: For the time being we'll go with vector and a custom iterator for know-destination-vertex lookups, but this benchmark might be handy to keep for future work.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D762
Summary: Ugly formatting and bad variable naming due to 1024 limit on neo4j client.
Reviewers: buda, teon.banek
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D748
Summary:
1. Test setup rewritten to take cca 8 seconds. Note that edges are created by using:
`MATCH (a) WITH a MATCH (b) WITH b WHERE rand() < X CREATE (a)-[:ET]->(b)`
Where `X` is a threshold calculated so the desired edge count is the expectation. This seems the only feasable way of generating a large number of edges since query execution does not depend on edge count, but on vertex count.
2. Using the new `assert` function to verify graph state. I recommend doing that in all the harness tests (I don't think we currently have something better).
3. All tests rewritten to take around 200ms per iteration.
4. Test are using SKIP to avoid sending data to the client, but ensure that appropriate operations get executed. This currently seems like the best way of removing unwanted side-effects.
Harness will cost us our sanity. And it doesn't even provide good quality regression testing we really need :(
Reviewers: buda, mislav.bradac, mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D752
Summary: It turns out trivial if I use unwind for vertex creation, MATCH for edge creation and UNWIND for test duration. It took hours to converge to this :(
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D747
Summary: `assert` function added. Useful to us for asserting DB state in harness tests. Potentially useful to the client for breaking out of a query as soon as a predicate fails, as opposed to collecting result and checking them client-side.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D750
Summary:
Moved Neo4j config to config dir.
Neo4j and PostgreSQL are now downloaded to libs.
Renamed metadata flags in memgraph.
Changed apollo generate for new harness.
Reviewers: mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D741
Summary:
Don't require setup_system to run as root, nor apt-get
Implement command_fail for ldbc/setup_dependencies
ldbc.setup_dataset: Find Java on ArchLinux
Reviewers: buda, mferencevic
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D729
Summary: Some TypedValue arithmetic ops threw exceptions for the wrong reasons, the op applicability type checks were wrong. Also the tests for that behavior were wrong. These are the fixes.
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D724
Summary:
Changed harness output to be compatible with script.
Changed cppcheck message formatting.
Changed qa message formatting.
Changed macro benchmark summary icons.
Fixed newlines in status generators.
Added group name to harness summary.
Macro benchmark summary now handles different files.
Added more output options to summary script.
Diff builds now build parent for performance comparison.
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D706
Summary:
This allows for inserting dummy DbAccessor in tests. Unfortunate side
effect of this change is that the whole implementation had to be moved
from cpp to hpp.
Also templatize remaining RuleBasedPlanner implementation
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D704
Summary:
Added:
- map support in PropertyValue
- conversion of map TypedValue to PropertyValue if appropriate flag is set (undocumented because it's private)
- ordering of map PropertyValue in LabelPropertyIndex
- issue raised regarding list and value property modifications in storage (currently unsupported)
Maybe I missed some feature or whatever?
Reviewers: mislav.bradac, buda, teon.banek
Reviewed By: mislav.bradac, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D692
Summary: Disabling it because it seems that (singleton + fixture + multiple_benchmarks) cause occasional crashes of the test. The more important test is left enabled.
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Differential Revision: https://phabricator.memgraph.io/D696
Summary:
It's purpose is to illustrate that currently expansion from a single node (with a fixed degree) does not execute in consistent time, but the execution time depends on the rest of the graph (that's disconnected). Current benchmark results:
```
florijan@florxps:~/Memgraph/memgraph/build$ ./tests/benchmark/expansion
Run on (8 X 3500 MHz CPU s)
2017-08-17 16:06:30
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
***WARNING*** Library was built as DEBUG. Timings may be affected.
Benchmark Time CPU Iterations
-------------------------------------------------------------------------
ExpansionBenchFixture/Match/1 0 ms 0 ms 4903
ExpansionBenchFixture/Match/1024 0 ms 0 ms 4785
ExpansionBenchFixture/Match/1024k 0 ms 0 ms 4623
ExpansionBenchFixture/Expand/1 9 ms 9 ms 83
ExpansionBenchFixture/Expand/1024 20 ms 20 ms 41
ExpansionBenchFixture/Expand/1024k 3430 ms 3430 ms 1
```
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D676
Summary:
This is needed in cases when the planner decides to start expanding from
the other end.
Reviewers: mislav.bradac, florijan
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D681
Summary: Because it was unnecessary and also implemented wrong (if someone tried using a Schduler with something other then std::mutex, it would not compile). We can trivially add this if it ever becomes necessary.
Reviewers: buda, mislav.bradac
Reviewed By: buda, mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D666
Summary:
A single line (graph_db.cpp:109 in the new code) was missing. This should have been done in D355 (made by DGleich, approved by Flor AND Buda AND Mislav :D).
Converted a lambda to a method for convenience.
Reviewers: buda, dgleich, mislav.bradac
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D665
Summary: Changed on-scope-exit-mechanism from macro (with two auto-generated variables and an all-capturing lambda) to an explicitly created variable that takes an std::function argument.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D659
Summary: Not strictly neccessary, but it's been itching me. It took an hour.
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D648
Summary:
- added only one function for getting the total (in + out) vertex degree, it's required for the Ravelin use-case
- specific `degree_in` and `degree_out` functions can be added as necessary
- also fixed random_graph_generator bug (needed it for testing)
Reviewers: buda, mislav.bradac
Reviewed By: buda, mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D652
Summary:
Now all ScanAll and Expand ops are covered by the cost estimator. For ScanAll with indices cost estimation is pretty good, for new Expand ops it is tragically bad (Expand to the power of expansion depth, plus arbitrary filtering). Static cost estimation is wrong wrong wrong.
Currently cost estimation of even trivial plans that use indices is wrong because the planner leaves filtering expressions that are implicitly handled by the index in the operator tree, IIRC. Tasking Teon to revise this, even though I'm not sure how bad an influence this has on cost estimation and it's use in plan choosing.
Reviewers: mislav.bradac, teon.banek, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D633
Summary:
This diff contains a bug fix for the expansion operators that are currently on dev.
More importantly, it proposes end-to-end testing for edge-cases for which it's a
pain to write single-phase tests. In my opinion this is OK, you're all reviewers so
you can comment.
The test relies on left-to-right query execution. We need this guarantee in tests
like this. I propose renaming "RuleBasedPlanner" to "LeftToRightPlanner" to make
this explicit. As Teon is not here at the moment, will make this a task/discussion.
Reviewers: buda, mislav.bradac, teon.banek, lion
Reviewed By: mislav.bradac
Subscribers: mferencevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D626
Summary:
Not complete (but review can start):
- implementation should be done
- still need to finish tests
- documentation missing
Reviewers: mislav.bradac, teon.banek, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D616
Summary: Fixed bug for SkipList::position_and_count for an item lesser then all skiplist elements.
Reviewers: mislav.bradac
Reviewed By: mislav.bradac
Differential Revision: https://phabricator.memgraph.io/D629
Summary: Index added to long running test to speed up edge creation and vertex deletion.
Reviewers: buda, mferencevic
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D621
Summary:
Implement trie and use it in stripper
Make it nicer
Reviewers: buda, florijan
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D614
Summary:
`UNWIND` can come before `MATCH`, so it needs to break query parts. If
it didn't, a query part would incorrectly grab all the matches and plan
them incorrectly. A test for such a case has been added.
Reviewers: florijan
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D598
Summary:
Add All expression to Ast
Evaluate All expression
Visit All and generate symbols
Handle All when collecting context during planning
Reviewers: florijan, mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D587
Summary:
Add EdgeList symbol type and check for redeclaration
The result of variable path is a list of edges, so the symbol type has
been added. In the future, we need to extend the type checker and the
type structure to have a generic list type.
We also currently do not support reusing an already bound symbol for a
variable path, so the SymbolGenerator will raise a redeclaration error.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot, lion
Differential Revision: https://phabricator.memgraph.io/D574
Summary:
The main init script now calls qa init.
QA CI script now runs the tests.
QA run script now returns a good exit code.
Explicitly bind to 127.0.0.1.
localhost on some machines resolves to an IPv6 address, and Memgraph
listens exclusively on IPv4.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D583
Summary: A helper function added for transferring graph elements into some GraphDbAccessor.
Reviewers: mislav.bradac, buda, teon.banek
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D581
Summary:
Variable expansion logical operator added. Some functionalities are missing:
- taking into account optional matching when expanding into existing symbol
- accepting Expression bounds (current implementation takes size_t)
Also, a TODO is added for handling optional matching in the uniqueness operator (with an Asana task)
All this will be done in the following diff, this is already substantial.
Also, please consider if we want to have all those `VLOG`s in the code. Not very pretty. And I think that `VLOG` is not compiled-away in release build, will put an asana task.
Reviewers: teon.banek, mislav.bradac, buda
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D569
Summary:
Allow expressions for variable length path bounds
Replace test which expected a syntax exception
Since we now allow variable length to have an arbitrary expression, the
test case is obsolete. It was replaced with something that excepts an
expression which wasn't allowed before.
Reviewers: florijan, mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: lion, pullbot
Differential Revision: https://phabricator.memgraph.io/D568
Summary:
- GraphDbAccessor - index range API added
- index api tests refactored
- skiplist minor cleanup.
Reviewers: teon.banek, buda, mislav.bradac
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D533
Summary:
Add optional bounds to PropertyFilter and collect them
Relation operators (e.g. `<`, `>` ...) should be used to produce
scanning the index by a range of values. For that reason, PropertyFilter
is extended to store either the equality expression or range bounds.
The `AnalyzeFilter` function is extended to look for those operators and
see if their top level expression contains a property lookup. If it
does, a filter with a bound is generated.
Test for property comparison preventing index use
Reviewers: florijan, mislav.bradac, buda
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D529
Summary: Update of tests/stress/create_match.py to use the right functions from tests/stress/common.py.
Reviewers: florijan
Reviewed By: florijan
Subscribers: pullbot, buda
Differential Revision: https://phabricator.memgraph.io/D547
Summary:
There was only two files in dbms directory so I moved them to database
directory.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D540
Summary:
FILE COPY copies files only in configuration stage, configure_file
copies files in make stage if file has changed.
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D531
Summary:
Add Filters class for storing additional info
Add FindOr to utils/algorithm.hpp
Use all collected labels when scanning by them
Collect label filters inside WHERE
Document the Filters class
Reviewers: florijan, mislav.bradac, buda
Reviewed By: florijan
Subscribers: pullbot, lion
Differential Revision: https://phabricator.memgraph.io/D515
Summary:
- added functionality to `GraphDbAccessor` for cardinality estimates
- changed all `GraphDbAccessor::Count...` functions to return `int64_t`
- added the need functionality into `LabelPropertyIndex`
- modified `SkipList::position_and_count` to accept a custom `equals` function. Equality could not be implemented using only the custom `less` because it compares a templated `TItem` with skiplist element type `T`, and is therefore not symetrical.
Reviewers: teon.banek, buda, mislav.bradac
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D521
Summary:
- refactored so `less` is used instead of `greater`
- added a fuzzy unit test
Reviewers: mislav.bradac, buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D519
Summary:
Add ScanAllByLabelPropertyRange operator
This operator uses the label + property indexing feature to iterate over
the vertices. The property value of each vertex is checked whether it is
inside the given range of values. The range is inclusive from both
sides. If the value isn't in range, the vertex is filtered out.
This manual filtering should be replaced by a database API when it
becomes available.
Add ScanAllByLabelPropertyValue operator
Reviewers: florijan, mislav.bradac, buda
Reviewed By: florijan, mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D503
Summary:
Changed run script to use built memgraph binary.
Added missing binary check to run script.
Reviewers: buda, teon.banek
Reviewed By: buda, teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D504
Summary: This is the first implementation that seems to work. I am not happy with it's complexity. Might attempt a simpler implementation, at the cost of some performance.
Reviewers: dgleich, buda
Reviewed By: dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D502
Summary:
PushQueue with concurrent lock-free pushing, and single-threaded deletion. Iteration without modification can also be concurrent. Deletion should NOT be concurrent with iteration and other deletions, but can be concurrent with pushing.
There is no const iteraton at the moment, we can add it when necessary. Also I've not handled std::iterator_traits, might be fun getting into that :D
Reviewers: buda, dgleich, mislav.bradac
Reviewed By: dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D420
Summary:
Remove utils/stacktrace/log.hpp
The single function defined in log.hpp is used only in
memgraph_bolt.cpp. Therefore, the function has been moved and the file
removed.
Move utils/stacktrace/stacktrace.hpp one level up
Move some logging from memgraph_bolt to Server
Reviewers: buda, dtomicevic
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D465
Summary:
This is an obvious bug, caused by an oversight. A test has been added
for this case.
Reviewers: florijan, mislav.bradac, buda
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D455
Summary: Bipartite stress test. Common methods useful in the process of testing with python Bolt driver.
Reviewers: mislav.bradac, florijan
Reviewed By: florijan
Subscribers: florijan, pullbot, buda
Differential Revision: https://phabricator.memgraph.io/D423
Summary:
- GC changed to evaluate old records w.r.t. the oldest transaction's id AND snapshot, as opposed to only id
- MVCC hints exp+aborted race condition prevented
- minor MVCC refactors and cleanups
- minor Transaction refactors and cleanups
Reviewers: buda, dgleich
Reviewed By: buda, dgleich
Subscribers: dtomicevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D434
Summary:
CMake is smart enough to transitively detect dependencies and link them
appropriately. Therefore, it is enough that we put all libraries that
memgraph uses to the dependency list of memgraph_lib and memgraph_pic
targets.
Patch the fmt library for C++14 and higher
fmt library would detect that C++11 is supported and then put the
compiler flag. This flag was set so it overrides parent project compiler
flags. This override from fmt would prevent us from using C++14
features. New version (3.1) of fmt resolves this issue, but it hasn't
been released yet. Therefore, this commit updates the script which
clones fmt to use the released 3.0.1 version and apply the fix on that.
Reviewers: dgleich, buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D441
Summary:
This fixes an issue when aggregations and/or group by expressions
weren't picked up from certain operators. In addition to that, we would
segfault in cases when the `has_aggregation_` is empty. For example,
function calls without arguments: `RETURN PI()`.
Test aggregations inside some operators
Reviewers: florijan, mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D425
Summary:
Permute query parts.
Permute matching only by selecting the starting node.
Flip the expansion when expanding from the other node.
Split planner into rule_based_planner and variable_start_planner
Use symbol hash when collecting expansion nodes
Multiple node atoms may point to the same symbol, and we could generate
multiple starting positions per atom which are the same. Using symbol
hash and equality prevents generating those redundant plans.
Correctly permute optional and merge matchings
Test VariableStartPlanner
Reviewers: florijan, mislav.bradac, buda, lion
Reviewed By: florijan, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D417
