Summary:
Fixing https://app.asana.com/0/170237629387822/481366792497820/f
Test plan:
Started two builds, one with the fix and the second one without the fix.
Connected to each of them using `neo4j` client.
Logs received from the build wihtout the fix:
```
neo4j> Create (n: BigInteger{id:12345678912345678912345}) return n;
<interactive>:0:0: error:
```
```
I1120 13:29:09.551208 30482 executing.hpp:69] [Run] 'Create (n: BigInteger{id:12345678912345678912345}) return n'
W1120 13:29:09.552387 30482 executing.hpp:145] Error message:
```
Logs received from the build with the fix:
```
neo4j> Create (n: BigInteger{id:12345678912345678912345}) return n;
<interactive>:0:0: error: Integer literal exceeds 64 bits
```
```
I1120 13:29:07.940943 30453 executing.hpp:69] [Run] 'Create (n: BigInteger{id:12345678912345678912345}) return n'
W1120 13:29:07.942919 30453 executing.hpp:146] Error message: Integer literal exceeds 64 bits
```
Reviewers: mislav.bradac, teon.banek
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D997
Summary: Vertex and Edge now use Address for storing connections to other Edges and Vertices, to support distributed storage.
Reviewers: mislav.bradac, dgleich, buda
Reviewed By: mislav.bradac, dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D977
Summary:
My dear fellow Memgraphians. It's friday afternoon, and I am as ready to pop as WAL is to get reviewed...
What's done:
- Vertices and Edges have global IDs, stored in `VersionList`. Main storage is now a concurrent map ID->vlist_ptr.
- WriteAheadLog class added. It's based around buffering WAL::Op objects (elementraly DB changes) and periodically serializing and flusing them to disk.
- Snapshot recovery refactored, WAL recovery added. Snapshot format changed again to include necessary info.
- Durability testing completely reworked.
What's not done (and should be when we decide how):
- Old WAL file purging.
- Config refactor (naming and organization). Will do when we discuss what we want.
- Changelog and new feature documentation (both depending on the point above).
- Better error handling and recovery feedback. Currently it's all returning bools, which is not fine-grained enough (neither for errors nor partial successes, also EOF is reported as a failure at the moment).
- Moving the implementation of WAL stuff to .cpp where possible.
- Not sure if there are transactions being created outside of `GraphDbAccessor` and it's `BuildIndex`. Need to look into.
- True write-ahead logic (flag controlled): not committing a DB transaction if the WAL has not flushed it's data. We can discuss the gain/effort ratio for this feature.
Reviewers: buda, mislav.bradac, teon.banek, dgleich
Reviewed By: dgleich
Subscribers: mtomic, pullbot
Differential Revision: https://phabricator.memgraph.io/D958
Summary:
We need to explicitly say that UNION clause isn't supported, otherwise
it gets silently ignored. So for example, `CREATE () UNION CREATE ()`
would create 2 nodes without a hitch. On the other hand,
`RETURN 1 UNION RETURN 2` would complain that there is more than 1
RETURN in the query, which was misleading.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D976
Summary:
Previously, named path symbols remained untracked as `new_symbols` during planning. This meant that
operator `Optional` would be left unaware of those symbols, and therefore not reset them to `Null`
if optional matching failed.
Test Optional operator will be aware of path symbols
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D974
Summary:
The warnings lifted to errors should help us track down cases which
haven't been covered in a switch. Obviously, using the default case will
not trigger these errors, so default should only be used when we
actually want to cover more cases.
Reviewers: florijan, dgleich, mferencevic
Reviewed By: dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D975
Summary:
Warnings I ignored:
Creates a new stacktrace object and then dumps it
102.570384 (102.495075) E[1]: [src/utils/exceptions.hpp:116]: (performance) Variable 'stacktrace_' is assigned in constructor body. Consider performing initialization in initialization list.
102.570390 (102.495081) E[1]: [src/utils/exceptions.hpp:127]: (performance) Variable 'stacktrace_' is assigned in constructor body. Consider performing initialization in initialization list.
Used all over the codebase without explicit cast
102.570412 (102.495103) E[1]: [src/utils/stacktrace.hpp:14]: (style) Class 'Line' has a constructor with 1 argument that is not explicit.
Not really used anywhere before initialized:
102.570526 (102.495217) E[1]: [src/data_structures/concurrent/skiplist.hpp:467]: (warning) Member variable 'Accessor::preds' is not initialized in the constructor.
102.570530 (102.495221) E[1]: [src/data_structures/concurrent/skiplist.hpp:467]: (warning) Member variable 'Accessor::succs' is not initialized in the constructor.
Implicit conversions between types are used all over the codebase:
102.570548 (102.495239) E[1]: [src/storage/property_value.hpp:41]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570552 (102.495243) E[1]: [src/storage/property_value.hpp:42]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570557 (102.495248) E[1]: [src/storage/property_value.hpp:43]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570561 (102.495252) E[1]: [src/storage/property_value.hpp:44]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570566 (102.495257) E[1]: [src/storage/property_value.hpp:47]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570570 (102.495261) E[1]: [src/storage/property_value.hpp:50]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570578 (102.495269) E[1]: [src/storage/property_value.hpp:53]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570582 (102.495273) E[1]: [src/storage/property_value.hpp:57]: (style) Class 'PropertyValue' has a constructor with 1 argument that is not explicit.
102.570591 (102.495282) E[1]: [src/query/typed_value.hpp:80]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570596 (102.495287) E[1]: [src/query/typed_value.hpp:81]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570601 (102.495292) E[1]: [src/query/typed_value.hpp:82]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570605 (102.495296) E[1]: [src/query/typed_value.hpp:83]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570609 (102.495300) E[1]: [src/query/typed_value.hpp:89]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570614 (102.495305) E[1]: [src/query/typed_value.hpp:92]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570618 (102.495309) E[1]: [src/query/typed_value.hpp:95]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570623 (102.495314) E[1]: [src/query/typed_value.hpp:98]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570627 (102.495318) E[1]: [src/query/typed_value.hpp:102]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570632 (102.495323) E[1]: [src/query/typed_value.hpp:105]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570636 (102.495327) E[1]: [src/query/typed_value.hpp:108]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570641 (102.495332) E[1]: [src/query/typed_value.hpp:109]: (style) Class 'TypedValue' has a constructor with 1 argument that is not explicit.
102.570645 (102.495336) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:88]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570650 (102.495341) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:89]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570654 (102.495345) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:90]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570659 (102.495350) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:91]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570663 (102.495354) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:94]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570668 (102.495359) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:97]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570672 (102.495363) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:100]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570677 (102.495368) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:104]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570681 (102.495372) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:107]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570690 (102.495381) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:110]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
102.570694 (102.495385) E[1]: [src/communication/bolt/v1/decoder/decoded_value.hpp:113]: (style) Class 'DecodedValue' has a constructor with 1 argument that is not explicit.
CypherParser:
102.570767 (102.495458) E[1]: [src/query/frontend/opencypher/generated/CypherParser.h:69]: (style) Class 'CypherParser' has a constructor with 1 argument that is not explicit.
102.570772 (102.495463) E[1]: [src/query/frontend/opencypher/generated/CypherLexer.h:40]: (style) Class 'CypherLexer' has a constructor with 1 argument that is not explicit.
102.570776 (102.495467) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:86]: (style) The scope of the variable '_la' can be reduced.
102.570781 (102.495472) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:311]: (style) The scope of the variable '_la' can be reduced.
102.570785 (102.495476) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:402]: (style) The scope of the variable '_la' can be reduced.
102.570789 (102.495480) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:497]: (style) The scope of the variable '_la' can be reduced.
102.570797 (102.495488) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:778]: (style) The scope of the variable '_la' can be reduced.
102.570802 (102.495493) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:895]: (style) The scope of the variable '_la' can be reduced.
102.570806 (102.495497) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:991]: (style) The scope of the variable '_la' can be reduced.
102.570811 (102.495502) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:1190]: (style) The scope of the variable '_la' can be reduced.
102.570815 (102.495506) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:1274]: (style) The scope of the variable '_la' can be reduced.
102.570820 (102.495511) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:1393]: (style) The scope of the variable '_la' can be reduced.
102.570824 (102.495515) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:1570]: (style) The scope of the variable '_la' can be reduced.
102.570829 (102.495520) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:1695]: (style) The scope of the variable '_la' can be reduced.
102.570834 (102.495525) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:1800]: (style) The scope of the variable '_la' can be reduced.
102.570839 (102.495530) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:1903]: (style) The scope of the variable '_la' can be reduced.
102.570843 (102.495534) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:2019]: (style) The scope of the variable '_la' can be reduced.
102.570848 (102.495539) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:2228]: (style) The scope of the variable '_la' can be reduced.
102.570852 (102.495543) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:2542]: (style) The scope of the variable '_la' can be reduced.
102.570857 (102.495548) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:2797]: (style) The scope of the variable '_la' can be reduced.
102.570861 (102.495552) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:2966]: (style) The scope of the variable '_la' can be reduced.
102.570866 (102.495557) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:3067]: (style) The scope of the variable '_la' can be reduced.
102.570870 (102.495561) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:3289]: (style) The scope of the variable '_la' can be reduced.
102.570875 (102.495566) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:3295]: (style) The scope of the variable 'alt' can be reduced.
102.570879 (102.495570) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:3419]: (style) The scope of the variable '_la' can be reduced.
102.570884 (102.495575) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:3596]: (style) The scope of the variable '_la' can be reduced.
102.570888 (102.495579) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:3688]: (style) The scope of the variable '_la' can be reduced.
102.570893 (102.495584) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:3963]: (style) The scope of the variable '_la' can be reduced.
102.570897 (102.495588) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:4452]: (style) The scope of the variable '_la' can be reduced.
102.570902 (102.495593) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:4586]: (style) The scope of the variable '_la' can be reduced.
102.570906 (102.495597) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:4813]: (style) The scope of the variable '_la' can be reduced.
102.570911 (102.495602) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:4943]: (style) The scope of the variable '_la' can be reduced.
102.570918 (102.495609) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:5026]: (style) The scope of the variable '_la' can be reduced.
102.570923 (102.495614) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:5569]: (style) The scope of the variable '_la' can be reduced.
102.570928 (102.495619) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:5664]: (style) The scope of the variable '_la' can be reduced.
102.570932 (102.495623) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:5755]: (style) The scope of the variable '_la' can be reduced.
102.570937 (102.495628) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:5888]: (style) The scope of the variable '_la' can be reduced.
102.570941 (102.495632) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:6045]: (style) The scope of the variable '_la' can be reduced.
102.570946 (102.495637) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:6142]: (style) The scope of the variable '_la' can be reduced.
102.570950 (102.495641) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:6347]: (style) The scope of the variable '_la' can be reduced.
102.570955 (102.495646) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:6523]: (style) The scope of the variable '_la' can be reduced.
102.570959 (102.495650) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:6614]: (style) The scope of the variable '_la' can be reduced.
102.570964 (102.495655) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:6899]: (style) The scope of the variable '_la' can be reduced.
102.570968 (102.495659) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:6992]: (style) The scope of the variable '_la' can be reduced.
102.570973 (102.495664) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:7147]: (style) The scope of the variable '_la' can be reduced.
102.570977 (102.495668) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:7680]: (style) The scope of the variable '_la' can be reduced.
102.570982 (102.495673) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:7759]: (style) The scope of the variable '_la' can be reduced.
102.570986 (102.495677) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:7938]: (style) The scope of the variable '_la' can be reduced.
102.570991 (102.495682) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:8126]: (style) The scope of the variable '_la' can be reduced.
102.570995 (102.495686) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:8220]: (style) The scope of the variable '_la' can be reduced.
102.571000 (102.495691) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:8313]: (style) The scope of the variable '_la' can be reduced.
102.571004 (102.495695) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:8491]: (style) The scope of the variable '_la' can be reduced.
102.571009 (102.495700) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:8703]: (style) The scope of the variable '_la' can be reduced.
102.571013 (102.495704) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:8783]: (style) The scope of the variable '_la' can be reduced.
102.571018 (102.495709) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:8914]: (style) The scope of the variable '_la' can be reduced.
102.571022 (102.495713) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:9119]: (style) The scope of the variable '_la' can be reduced.
102.571027 (102.495718) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:9220]: (style) The scope of the variable '_la' can be reduced.
102.571034 (102.495725) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:9414]: (style) The scope of the variable '_la' can be reduced.
102.571039 (102.495730) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:9660]: (style) The scope of the variable '_la' can be reduced.
102.571043 (102.495734) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10008]: (style) The scope of the variable '_la' can be reduced.
102.571048 (102.495739) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10158]: (style) The scope of the variable '_la' can be reduced.
102.571052 (102.495743) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10250]: (style) The scope of the variable '_la' can be reduced.
102.571057 (102.495748) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10370]: (style) The scope of the variable '_la' can be reduced.
102.571061 (102.495752) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10637]: (style) The scope of the variable '_la' can be reduced.
102.571065 (102.495756) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10749]: (style) The scope of the variable '_la' can be reduced.
102.571070 (102.495761) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10815]: (style) The scope of the variable '_la' can be reduced.
102.571075 (102.495766) E[1]: [src/query/frontend/opencypher/generated/CypherParser.cpp:10881]: (style) The scope of the variable '_la' can be reduced.
We know that we represented it correctly in memory:
102.571079 (102.495770) E[1]: [src/communication/bolt/v1/decoder/decoder.hpp:252]: (portability) Casting between integer* and double* which have an incompatible binary data representation.
Cont assigned but not used after:
102.571101 (102.495792) E[1]: [src/query/frontend/ast/ast.hpp:1008]: (style) Variable 'cont' is assigned a value that is never used.
Reviewers: teon.banek, buda, mferencevic
Reviewed By: teon.banek
Subscribers: mferencevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D967
Summary:
Tests have been updated to catch this error and other behaviour. Other
than this change, `AND` should behave as before.
Reviewers: florijan, mislav.bradac
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D970
Summary:
Use sigaction to register signal handlers.
This is preferred over `signal` function, according to `man 3p signal`.
Add global sig_atomic_t flag when shutting down.
Block other signal handlers when shutting down.
Reviewers: mislav.bradac, mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D943
Summary: Old code would start reactor thread first, and then try to insert it in unordered_map. If it failed to insert it, the thread handler would get destructed with thread still running, which raises an exception.
Reviewers: buda, mislav.bradac
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D956
Summary: TypedValue assignment operator=(TypedValue &other) is inefficient for types which already have a defined assignment operator, or can be assigned trivially, without using TypedValue to pass the value for assignment.
Reviewers: mislav.bradac, florijan, teon.banek
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D934
Summary:
Remove name from GraphDb.
Take GraphDb in query test macros instead of accessor.
Add is_accepting_transactions flag to GraphDb.
Reviewers: mislav.bradac, florijan, mferencevic
Reviewed By: mislav.bradac
Subscribers: mferencevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D940
Summary:
- Removed durability::Summary because it was wired into reader and stopped me from recovering WAL files.
- Refactored and renamed BufferedFile(Reader/Writer) to HashedFile(Reader/Writer).
- Vertex and edge counts in the snapshot are now hashed.
Breaking snapshot compatibility again (hashing), but since the previous version was not released, and we are not caching snapshots, the previous version does not need to be supported.
Reviewers: teon.banek, mislav.bradac, buda
Reviewed By: teon.banek, mislav.bradac
Subscribers: dgleich, pullbot
Differential Revision: https://phabricator.memgraph.io/D932
Summary:
Time csv_to_snapshot conversion and log it.
Check if writing csv_to_snapshot failed.
Extract LoadConfig from memgraph_bolt to config.hpp.
Read memgraph config in csv_to_snapshot for snapshot_directory.
Rename csv_to_snapshot to mg_import_csv.
Add tests for tools.
Run tools tests in apollo.
Reviewers: mislav.bradac, florijan, mferencevic, buda
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D931
Summary: This reduces the size of a single `mvcc::Record<T>` from 40B to 24B.
Reviewers: buda, mislav.bradac, dgleich
Reviewed By: mislav.bradac, dgleich
Subscribers: mferencevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D920
Summary:
Move QueryParts and Filters to a new file.
Reorganize FilterInfo struct.
Remove label filter if we do indexed scan by label.
Remove property filter used in indexed scan.
Reviewers: florijan
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D915
Summary:
There seems to be a gain all over memgraph.
I strongly suggest including this in the codebase.
Link to project:
https://github.com/r-lyeh/ltalloc
Reviewers: teon.banek, mferencevic, mislav.bradac, florijan
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D914
Summary:
This is an attempt at solving circular dependencies happening in WAL implementation. The cycle is:
BaseEncoder -> GraphDbAccessor -> GraphDb -> WAL -> BaseEncoder.
The cycle will be broken by this diff because the WAL only needs primitive encoding and will be able to use `PrimitiveEncoder` only.
This fix is not ideal, since the BaseEncoder -> GraphDbAccessor dependency is very unnatural. This could probably be fixes properly with a refactor of GraphDb/GraphDbAccessor that is in the post, but that takes time and this fix is not very dirty, more of an added complication.
Reviewers: buda, mferencevic
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D925
Summary: Locked version. There are some benchmarks, it seems the lock won't be the bottleneck in the WAL (DB ops causing WAL delta insertions into it will be slower, flushing the WAL be slower).
Reviewers: buda, mislav.bradac, dgleich
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D919
Summary:
New snapshot structure:
- magic number
- snapshot version (old-version recovery not yet implemented)
- transaction snapshot (will be used in the WAL)
- the rest is as before (indices, vertices, edges)
Not backward compatible with the old snapshotting.
Does not improve error handling (user feedback). A task for that has been added.
Reviewers: buda, mislav.bradac, mferencevic, teon.banek
Reviewed By: teon.banek
Subscribers: teon.banek, dgleich, pullbot
Differential Revision: https://phabricator.memgraph.io/D912
Summary: It wasn't used in MG, only in tests.
Reviewers: buda, dgleich, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D909
Summary: This change increases the planning time, but should reduce memory consumption.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D901
Summary: In the current state, it was not possible to iterate, or even access a const map, or const set structure because of an incorrect implementation of "ConstAccessors".
Reviewers: mislav.bradac, teon.banek, buda
Reviewed By: teon.banek, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D902
Summary: Lets run daily tonight with malloc_trim and see what is memory usage
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D888
Summary:
This should be the basis for default configuration that is to be shipped
in release builds/packages.
Reviewers: mislav.bradac, florijan, buda, mferencevic
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D851
Summary:
This puts the whole installation and packaging under a single point of
entry. (Docker, DEB, RPM, etc.)
Rename alpha.dockerfile to beta.dockerfile
Use Debian Stretch for docker
Remove building old hardcoded compiler
Rename build_interpreter to build_memgraph
Remove unused config-file
Reviewers: mferencevic, buda
Reviewed By: mferencevic, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D857
Summary: This is not a very important functionality, but it turned out simple to do, so let's add it to have a consistent query support.
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D862
Summary:
- Removed BreadthFirstAtom, using EdgeAtom only with a Type enum.
- Both variable expansions (breadth and depth first) now have mandatory inner node and edge Identifiers.
- Both variable expansions use inline property filtering and support inline lambdas.
- BFS and variable expansion now have the same planning process.
- Planner modified in the following ways:
- Variable expansions support inline property filtering (two filters added to all_filters, one for inline, one for post-expand).
- Asserting against existing_edge since we don't support that anymore.
- Edge and node symbols bound after variable expansion to disallow post-expand filters to get inlined.
- Some things simplified due to different handling.
- BreadthFirstExpand logical operator merged into ExpandVariable. Two Cursor classes remain and are dynamically chosen from.
As part of planned planner refactor we should ensure that a filter is applied only once. The current implementation is very suboptimal for property filtering in variable expansions.
@buda: we will start refactoring this these days. This current planner logic is too dense and complex. It is becoming technical debt. Most of the time I spent working on this has been spent figuring the planning out, and I still needed Teon's help at times. Implementing the correct and optimal version of query execution (avoiding multiple potentially expensive filterings) was out of reach also due to tech debt.
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot, buda
Differential Revision: https://phabricator.memgraph.io/D852
Summary:
Log files aren't created by default anymore.
All logs are reported to stderr by default.
Normalized flag names.
Removed unnecessary flags from gflags.
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D856
Summary: It's a sketch. I can tidy it up if you like this approach.
Reviewers: mislav.bradac, mferencevic
Reviewed By: mislav.bradac
Subscribers: buda, pullbot
Differential Revision: https://phabricator.memgraph.io/D841
Summary:
- The new BFS syntax implemented as proposed.
- AST BreadthFirstAtom now uses EdgeAtom members: has_range_{true}, upper_bound_, lower_bound_
- Edges data structure now handles all the edge filtering (single or multiple edges), to ease planning. Additional edge filtering (additional Filter op in the plan) is removed. AST EdgeTypeTest is no longer used and is removed.
Current state is stable but there are things left to do:
- BFS property filtering.
- BFS lower_bound_ support.
- Support for lambdas in variable length expansion. This includes obligatory (even if not user_defined) inner_node and inner_edge symbols for easier handling.
- Code-sharing between BFS and variable length expansions.
I'll add asana tasks (and probably start working on them immediately) when/if this lands.
Reviewers: buda, teon.banek, mislav.bradac
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D836
Summary:
Few days ago I put a task to reduce number of exceptions in memgraph codebase :)
I think we should use exceptions only in modules where we are using smart pointers.
Reviewers: florijan, buda
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D834
Summary:
AST caching should be well tested by now.
We should consider removing `Context.is_query_cached_` member as well as the
implementation and tests for `CypherMainVisitor`.
Reviewers: mislav.bradac, mferencevic, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D833
Summary: - modified all utils/algorithm functions to be inline and in the utils namespace
Reviewers: teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D830
Summary:
I started with cleaning flags up (removing unused ones, documenting undocumented ones). There were some flags to remove in `QueryEngine`. Seeing how we never use hardcoded queries (AFAIK last Mislav's testing also indicated they aren't faster then interpretation), when removing those unused flags the `QueryEngine` becomes obsolete. That means that a bunch of other stuff becomes obsolete, along with the hardcoded queries. So I removed it all (this has been discussed and approved on the daily).
Some flags that were previously undocumented in `docs/user_technical/installation` are now documented. The following flags are NOT documented and in my opinion should not be displayed when starting `./memgraph --help` (@mferencevic):
```
query_vertex_count_to_expand_existsing (from rule_based_planner.cpp)
query_max_plans (rule_based_planner.cpp)
```
If you think that another organization is needed w.r.t. flag visibility, comment.
@teon.banek: I had to remove some stuff from CMakeLists to make it buildable. Please review what I removed and clean up if necessary if/when this lands. If the needed changes are minor, you can also comment.
Reviewers: buda, mislav.bradac, teon.banek, mferencevic
Reviewed By: buda, mislav.bradac
Subscribers: pullbot, mferencevic, teon.banek
Differential Revision: https://phabricator.memgraph.io/D825
Summary:
Added a warning to the log. Every 3 seconds (let's not make that configurable). Can be turned off.
I still don't like this. We are raising another thread and reading a file to do monitoring. We're developing a DB, not a sys monitor. Serious admins do that themselves. But, here it is.
UPDATE:
Cleaned up `utils/sysinfo/memory`. Removed all unused functions. Removed the faulty memory check in `tests/concurrent`.
Reviewers: buda, mferencevic, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D819
Summary:
Setup scaffolding for building Memgraph tools.
Change `utils::Split` without delimiter to split on whitespace.
This should make `Split` behave just like Python's `str.split`, which is
more practical for splitting on word boundaries.
Add `utils::StartsWith` function.
Rewrite csv_to_snapshot to C++.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D822
Summary:
Added path encoding and decoding in our Bolt layer. Returning paths to the Neo client seems to work fine.
Not sure how to test that the decoded works fine (our client?).
Tests are not written, that suite is complicated. Leaving that to mferencevic.
Also reduced DecodedValue a bit with macros. Less code, less error prone.
Reviewers: buda, florijan
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D810
Summary:
Test planner splits MATCH ... WHERE
Remove distinction between FilterAnd and AndOperator
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D814
Summary:
This fixes a bug, where streaming would try to get the name of the symbol from
an invalid token position. For example, `MATCH (n) WITH n RETURN *`
Reviewers: florijan, mislav.bradac
Reviewed By: florijan, mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D811
Summary: Removed the traversal API, been waiting for named path to land because that data structure has replaced traversal's Path. I left the wiki page of the API, just put a warning that it's not used.
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D809
Summary:
Three TODOs resolved.
1. around line 897 - we currently don't support expansion into existing variable length edges (there is a TODO in symbol_generator.cpp:213), so this should not be done at the moment.
2. around line 1025 - This TODO was on review and nobody commented, so I'm removing it. Should have done that when the diff landed.
3. around line 1560 - This does not seem possible. Edge-uniqueness checks happen within a single `[OPTIONAL ] MATCH`. If it is OPTIONAL (the case interesting here), then the uniqueness check also gets planned under the optional branch. So, if an optional fails, the uniqueness check will get skipped, as opposed to getting executed over a Null. I added an edge-case test to verify this (and checked with the planner test).
Reviewers: buda, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D807
Summary:
- Keys() functions in the indices can't be const because ConcurrentMap doesn't provide const accessors (and they are broken in skiplist) :D
- no cucumber tests because many tests create indices so it's hard to say what's inside and what not
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D797
Summary:
This is done when the generated AST will be cached.
Remove LiteralsPlugger.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D788
Summary:
Replaced std::list with std::vector in all plan operators. Performance increase in harness tests is not visible. Defined a custom test:
```
unwind range(0, 1000000) as x
create ({a: tointeger(rand() * 100), b: tointeger(rand() * 100), c: tointeger(rand() * 100), d: tointeger(rand() * 10), e: tointeger(rand() * 10), f: tointeger(rand() * 10)});
match (n) return min(n.a), max(n.b), sum(n.c), n.d, n.e, n.f
match (n) with distinct n.a AS a, n.b AS b, n.c AS c, n.d AS d, n.e AS e, n.f AS f return count(*)
```
In that test performance gains are 9.8% on the aggregation query (mean 0.83s vs 092s) and 34% (mean 2.15s vs 3.25s) on the distinct query. Doubt we'll see much on any of the LDBC tests because they don't stress those operators nearly as much.
Reviewers: buda, teon.banek, mislav.bradac
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D778
Summary:
Antlr grammar has been updated to support putting edge types after the BFS
symbol. Planner collects edge type filters for BFS and inlines them in the
operator by joining the filter with the user input BFS filter itself. This
requires no change from the standpoint of the operator. On the other hand, in
order to use the faster lookup by a single edge type, `ExpandBreadthFirst`
operator now accept an optional edge type. The edge type is passed from the
planner only if the user is filtering by a single type.
Unit tests as well as tck have been updated.
Reviewers: florijan, mislav.bradac
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D777
Summary: Reduces latency on LDBC query 9 from 7.9sec to 6.8sec (14%). That query has 650k rows in ORDER BY, 3 ordering elements and 10ish values get returned (both of them are now accumulated into vectors).
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D775
Summary:
Add function First to utils.
Insert EdgeType into Expand during planning.
Reviewers: florijan, mislav.bradac
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D769
Summary: The constness of the DbAccessor interferes with caching the results.
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D771
Summary:
Benchmark planning and estimating indexed ScanAll. According to the benchmark,
caching speeds up the whole process of planning and estimation by a factor of
2. Most of the performance gain is in the `CostEstimator` itself, due to plenty
of calls to `VerticesCount` when estimating all of the generated plans.
Reviewers: mislav.bradac, florijan
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D765
Summary:
- The `Edges` data structure now handles common ops, including providing an iterator over edges whose "other" vertex is know.
- This should improve performance on dense_expand tests in the harness without other side-effects.
- query::plan::Expand operator modified not to check for existing-node stuff since that now gets handled by the `Edges` data structure.
- `Edges::Iterator` implemented only for const iterators since that suffices for now. Can implement non-const if the need arrises.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D763
Summary:
Measured improvements in a test scenario with 4 labels, 1M vertices for each label:
Old code: ~2.5 seconds per index
New code: ~1.5 seconds per index
When building an index for a non-existing label the updated code is done immediately. The old code depends on the number of vertices in the database.
The new code *could* be slower when building an index for a label that has a lot of vertices, and the index is not garbage collected recently and contains a lot of junk. This can be avoided by a simple check in the `BuildIndex` function (if label_index cardinality > total cardinality), if you like.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D767
Summary: `assert` function added. Useful to us for asserting DB state in harness tests. Potentially useful to the client for breaking out of a query as soon as a predicate fails, as opposed to collecting result and checking them client-side.
Reviewers: buda, mislav.bradac, teon.banek
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D750
Summary:
Moved Neo4j config to config dir.
Neo4j and PostgreSQL are now downloaded to libs.
Renamed metadata flags in memgraph.
Changed apollo generate for new harness.
Reviewers: mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D741
Summary: Some TypedValue arithmetic ops threw exceptions for the wrong reasons, the op applicability type checks were wrong. Also the tests for that behavior were wrong. These are the fixes.
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D724
Summary:
This allows for inserting dummy DbAccessor in tests. Unfortunate side
effect of this change is that the whole implementation had to be moved
from cpp to hpp.
Also templatize remaining RuleBasedPlanner implementation
Reviewers: florijan, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D704
Summary:
Added:
- map support in PropertyValue
- conversion of map TypedValue to PropertyValue if appropriate flag is set (undocumented because it's private)
- ordering of map PropertyValue in LabelPropertyIndex
- issue raised regarding list and value property modifications in storage (currently unsupported)
Maybe I missed some feature or whatever?
Reviewers: mislav.bradac, buda, teon.banek
Reviewed By: mislav.bradac, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D692
Summary:
This change should allow for passing a different PlanningContext and/or
GraphDbAccessor. In turn, we can write tests which pass a dummy context
for decoupled testing of the planning process (from the rest of the
system).
Reviewers: mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D700
Summary:
It's purpose is to illustrate that currently expansion from a single node (with a fixed degree) does not execute in consistent time, but the execution time depends on the rest of the graph (that's disconnected). Current benchmark results:
```
florijan@florxps:~/Memgraph/memgraph/build$ ./tests/benchmark/expansion
Run on (8 X 3500 MHz CPU s)
2017-08-17 16:06:30
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
***WARNING*** Library was built as DEBUG. Timings may be affected.
Benchmark Time CPU Iterations
-------------------------------------------------------------------------
ExpansionBenchFixture/Match/1 0 ms 0 ms 4903
ExpansionBenchFixture/Match/1024 0 ms 0 ms 4785
ExpansionBenchFixture/Match/1024k 0 ms 0 ms 4623
ExpansionBenchFixture/Expand/1 9 ms 9 ms 83
ExpansionBenchFixture/Expand/1024 20 ms 20 ms 41
ExpansionBenchFixture/Expand/1024k 3430 ms 3430 ms 1
```
Reviewers: mislav.bradac, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D676
Summary:
This is needed in cases when the planner decides to start expanding from
the other end.
Reviewers: mislav.bradac, florijan
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D681
