Summary: Lets run daily tonight with malloc_trim and see what is memory usage
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D888
Summary:
This should be the basis for default configuration that is to be shipped
in release builds/packages.
Reviewers: mislav.bradac, florijan, buda, mferencevic
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D851
Summary:
This puts the whole installation and packaging under a single point of
entry. (Docker, DEB, RPM, etc.)
Rename alpha.dockerfile to beta.dockerfile
Use Debian Stretch for docker
Remove building old hardcoded compiler
Rename build_interpreter to build_memgraph
Remove unused config-file
Reviewers: mferencevic, buda
Reviewed By: mferencevic, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D857
Summary: This is not a very important functionality, but it turned out simple to do, so let's add it to have a consistent query support.
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D862
Summary:
- Removed BreadthFirstAtom, using EdgeAtom only with a Type enum.
- Both variable expansions (breadth and depth first) now have mandatory inner node and edge Identifiers.
- Both variable expansions use inline property filtering and support inline lambdas.
- BFS and variable expansion now have the same planning process.
- Planner modified in the following ways:
- Variable expansions support inline property filtering (two filters added to all_filters, one for inline, one for post-expand).
- Asserting against existing_edge since we don't support that anymore.
- Edge and node symbols bound after variable expansion to disallow post-expand filters to get inlined.
- Some things simplified due to different handling.
- BreadthFirstExpand logical operator merged into ExpandVariable. Two Cursor classes remain and are dynamically chosen from.
As part of planned planner refactor we should ensure that a filter is applied only once. The current implementation is very suboptimal for property filtering in variable expansions.
@buda: we will start refactoring this these days. This current planner logic is too dense and complex. It is becoming technical debt. Most of the time I spent working on this has been spent figuring the planning out, and I still needed Teon's help at times. Implementing the correct and optimal version of query execution (avoiding multiple potentially expensive filterings) was out of reach also due to tech debt.
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot, buda
Differential Revision: https://phabricator.memgraph.io/D852
Summary:
Log files aren't created by default anymore.
All logs are reported to stderr by default.
Normalized flag names.
Removed unnecessary flags from gflags.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D856
Summary: It's a sketch. I can tidy it up if you like this approach.
Reviewers: mislav.bradac, mferencevic
Reviewed By: mislav.bradac
Subscribers: buda, pullbot
Differential Revision: https://phabricator.memgraph.io/D841
Summary:
- The new BFS syntax implemented as proposed.
- AST BreadthFirstAtom now uses EdgeAtom members: has_range_{true}, upper_bound_, lower_bound_
- Edges data structure now handles all the edge filtering (single or multiple edges), to ease planning. Additional edge filtering (additional Filter op in the plan) is removed. AST EdgeTypeTest is no longer used and is removed.
Current state is stable but there are things left to do:
- BFS property filtering.
- BFS lower_bound_ support.
- Support for lambdas in variable length expansion. This includes obligatory (even if not user_defined) inner_node and inner_edge symbols for easier handling.
- Code-sharing between BFS and variable length expansions.
I'll add asana tasks (and probably start working on them immediately) when/if this lands.
Reviewers: buda, teon.banek, mislav.bradac
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D836
Summary:
Few days ago I put a task to reduce number of exceptions in memgraph codebase :)
I think we should use exceptions only in modules where we are using smart pointers.
Reviewers: florijan, buda
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D834
Summary:
AST caching should be well tested by now.
We should consider removing `Context.is_query_cached_` member as well as the
implementation and tests for `CypherMainVisitor`.
Reviewers: mislav.bradac, mferencevic, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D833
Summary: - modified all utils/algorithm functions to be inline and in the utils namespace
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D830
Summary:
I started with cleaning flags up (removing unused ones, documenting undocumented ones). There were some flags to remove in `QueryEngine`. Seeing how we never use hardcoded queries (AFAIK last Mislav's testing also indicated they aren't faster then interpretation), when removing those unused flags the `QueryEngine` becomes obsolete. That means that a bunch of other stuff becomes obsolete, along with the hardcoded queries. So I removed it all (this has been discussed and approved on the daily).
Some flags that were previously undocumented in `docs/user_technical/installation` are now documented. The following flags are NOT documented and in my opinion should not be displayed when starting `./memgraph --help` (@mferencevic):
```
query_vertex_count_to_expand_existsing (from rule_based_planner.cpp)
query_max_plans (rule_based_planner.cpp)
```
If you think that another organization is needed w.r.t. flag visibility, comment.
@teon.banek: I had to remove some stuff from CMakeLists to make it buildable. Please review what I removed and clean up if necessary if/when this lands. If the needed changes are minor, you can also comment.
Reviewers: buda, mislav.bradac, teon.banek, mferencevic
Reviewed By: buda, mislav.bradac
Subscribers: pullbot, mferencevic, teon.banek
Differential Revision: https://phabricator.memgraph.io/D825
Summary:
Added a warning to the log. Every 3 seconds (let's not make that configurable). Can be turned off.
I still don't like this. We are raising another thread and reading a file to do monitoring. We're developing a DB, not a sys monitor. Serious admins do that themselves. But, here it is.
UPDATE:
Cleaned up `utils/sysinfo/memory`. Removed all unused functions. Removed the faulty memory check in `tests/concurrent`.
Reviewers: buda, mferencevic, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D819
Summary:
Setup scaffolding for building Memgraph tools.
Change `utils::Split` without delimiter to split on whitespace.
This should make `Split` behave just like Python's `str.split`, which is
more practical for splitting on word boundaries.
Add `utils::StartsWith` function.
Rewrite csv_to_snapshot to C++.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D822
Summary:
Added path encoding and decoding in our Bolt layer. Returning paths to the Neo client seems to work fine.
Not sure how to test that the decoded works fine (our client?).
Tests are not written, that suite is complicated. Leaving that to mferencevic.
Also reduced DecodedValue a bit with macros. Less code, less error prone.
Reviewers: buda, florijan
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D810
Summary:
Test planner splits MATCH ... WHERE
Remove distinction between FilterAnd and AndOperator
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D814
Summary:
This fixes a bug, where streaming would try to get the name of the symbol from
an invalid token position. For example, `MATCH (n) WITH n RETURN *`
Reviewers: florijan, mislav.bradac
Reviewed By: florijan, mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D811
Summary: Removed the traversal API, been waiting for named path to land because that data structure has replaced traversal's Path. I left the wiki page of the API, just put a warning that it's not used.
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D809
Summary:
Three TODOs resolved.
1. around line 897 - we currently don't support expansion into existing variable length edges (there is a TODO in symbol_generator.cpp:213), so this should not be done at the moment.
2. around line 1025 - This TODO was on review and nobody commented, so I'm removing it. Should have done that when the diff landed.
3. around line 1560 - This does not seem possible. Edge-uniqueness checks happen within a single `[OPTIONAL ] MATCH`. If it is OPTIONAL (the case interesting here), then the uniqueness check also gets planned under the optional branch. So, if an optional fails, the uniqueness check will get skipped, as opposed to getting executed over a Null. I added an edge-case test to verify this (and checked with the planner test).
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D807
Summary:
- Keys() functions in the indices can't be const because ConcurrentMap doesn't provide const accessors (and they are broken in skiplist) :D
- no cucumber tests because many tests create indices so it's hard to say what's inside and what not
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D797
Summary:
This is done when the generated AST will be cached.
Remove LiteralsPlugger.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D788
Summary:
Replaced std::list with std::vector in all plan operators. Performance increase in harness tests is not visible. Defined a custom test:
```
unwind range(0, 1000000) as x
create ({a: tointeger(rand() * 100), b: tointeger(rand() * 100), c: tointeger(rand() * 100), d: tointeger(rand() * 10), e: tointeger(rand() * 10), f: tointeger(rand() * 10)});
match (n) return min(n.a), max(n.b), sum(n.c), n.d, n.e, n.f
match (n) with distinct n.a AS a, n.b AS b, n.c AS c, n.d AS d, n.e AS e, n.f AS f return count(*)
```
In that test performance gains are 9.8% on the aggregation query (mean 0.83s vs 092s) and 34% (mean 2.15s vs 3.25s) on the distinct query. Doubt we'll see much on any of the LDBC tests because they don't stress those operators nearly as much.
Reviewers: buda, teon.banek, mislav.bradac
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D778
Summary:
Antlr grammar has been updated to support putting edge types after the BFS
symbol. Planner collects edge type filters for BFS and inlines them in the
operator by joining the filter with the user input BFS filter itself. This
requires no change from the standpoint of the operator. On the other hand, in
order to use the faster lookup by a single edge type, `ExpandBreadthFirst`
operator now accept an optional edge type. The edge type is passed from the
planner only if the user is filtering by a single type.
Unit tests as well as tck have been updated.
Reviewers: florijan, mislav.bradac
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D777
Summary: Reduces latency on LDBC query 9 from 7.9sec to 6.8sec (14%). That query has 650k rows in ORDER BY, 3 ordering elements and 10ish values get returned (both of them are now accumulated into vectors).
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D775
Summary:
Add function First to utils.
Insert EdgeType into Expand during planning.
Reviewers: florijan, mislav.bradac
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D769
Summary: The constness of the DbAccessor interferes with caching the results.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D771
Summary:
Benchmark planning and estimating indexed ScanAll. According to the benchmark,
caching speeds up the whole process of planning and estimation by a factor of
2. Most of the performance gain is in the `CostEstimator` itself, due to plenty
of calls to `VerticesCount` when estimating all of the generated plans.
Reviewers: mislav.bradac, florijan
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D765
Summary:
- The `Edges` data structure now handles common ops, including providing an iterator over edges whose "other" vertex is know.
- This should improve performance on dense_expand tests in the harness without other side-effects.
- query::plan::Expand operator modified not to check for existing-node stuff since that now gets handled by the `Edges` data structure.
- `Edges::Iterator` implemented only for const iterators since that suffices for now. Can implement non-const if the need arrises.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D763
Summary:
Measured improvements in a test scenario with 4 labels, 1M vertices for each label:
Old code: ~2.5 seconds per index
New code: ~1.5 seconds per index
When building an index for a non-existing label the updated code is done immediately. The old code depends on the number of vertices in the database.
The new code *could* be slower when building an index for a label that has a lot of vertices, and the index is not garbage collected recently and contains a lot of junk. This can be avoided by a simple check in the `BuildIndex` function (if label_index cardinality > total cardinality), if you like.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D767
Summary: `assert` function added. Useful to us for asserting DB state in harness tests. Potentially useful to the client for breaking out of a query as soon as a predicate fails, as opposed to collecting result and checking them client-side.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D750
Summary:
Moved Neo4j config to config dir.
Neo4j and PostgreSQL are now downloaded to libs.
Renamed metadata flags in memgraph.
Changed apollo generate for new harness.
Reviewers: mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D741
Summary: Some TypedValue arithmetic ops threw exceptions for the wrong reasons, the op applicability type checks were wrong. Also the tests for that behavior were wrong. These are the fixes.
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D724
Summary:
This allows for inserting dummy DbAccessor in tests. Unfortunate side
effect of this change is that the whole implementation had to be moved
from cpp to hpp.
Also templatize remaining RuleBasedPlanner implementation
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D704
Summary:
Added:
- map support in PropertyValue
- conversion of map TypedValue to PropertyValue if appropriate flag is set (undocumented because it's private)
- ordering of map PropertyValue in LabelPropertyIndex
- issue raised regarding list and value property modifications in storage (currently unsupported)
Maybe I missed some feature or whatever?
Reviewers: mislav.bradac, buda, teon.banek
Reviewed By: mislav.bradac, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D692
Summary:
This change should allow for passing a different PlanningContext and/or
GraphDbAccessor. In turn, we can write tests which pass a dummy context
for decoupled testing of the planning process (from the rest of the
system).
Reviewers: mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D700
Summary:
It's purpose is to illustrate that currently expansion from a single node (with a fixed degree) does not execute in consistent time, but the execution time depends on the rest of the graph (that's disconnected). Current benchmark results:
```
florijan@florxps:~/Memgraph/memgraph/build$ ./tests/benchmark/expansion
Run on (8 X 3500 MHz CPU s)
2017-08-17 16:06:30
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
***WARNING*** Library was built as DEBUG. Timings may be affected.
Benchmark Time CPU Iterations
-------------------------------------------------------------------------
ExpansionBenchFixture/Match/1 0 ms 0 ms 4903
ExpansionBenchFixture/Match/1024 0 ms 0 ms 4785
ExpansionBenchFixture/Match/1024k 0 ms 0 ms 4623
ExpansionBenchFixture/Expand/1 9 ms 9 ms 83
ExpansionBenchFixture/Expand/1024 20 ms 20 ms 41
ExpansionBenchFixture/Expand/1024k 3430 ms 3430 ms 1
```
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D676
