Commit Graph

3285 Commits

Author SHA1 Message Date
jeremy
4c5cd1f847 Add possibility to have MgBench working against local file 2022-10-31 16:10:04 +01:00
jeremy
e1f18f3733 Update location of Pokec datasets on aws 2022-10-31 15:19:34 +01:00
jeremy
c90b38faf0 Update aws address for datasets 2022-10-31 14:49:32 +01:00
jeremy
acbf3c764c Remove arg from __init__ 2022-10-31 13:35:41 +01:00
jeremy
37df41796f git puMerge branch 'T1079-MG-add-simple-query-to-benchmark_v2' of github.com:memgraph/memgraph into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 13:02:12 +01:00
jeremy
f04e1cda4b Add function 2022-10-31 13:02:05 +01:00
Jeremy B
f28ba89584
Update tests/mgbench/dataset_creator.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-31 13:01:42 +01:00
jeremy
60485311c8 Merge branch 'T1079-MG-add-simple-query-to-benchmark_v2' of github.com:memgraph/memgraph into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 12:57:46 +01:00
Jeremy B
36a1c43851
Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 12:57:41 +01:00
Jeremy B
bae8c084b1
Update tests/mgbench/datasets.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-31 12:56:02 +01:00
Jeremy B
2898120eeb
Update tests/mgbench/datasets.py
Co-authored-by: Jure Bajic <jure.bajic@memgraph.com>
2022-10-31 12:55:42 +01:00
jeremy
d62a45752a Remove unused variable 2022-10-31 12:55:38 +01:00
jeremy
c5ee6ffbc2 Update dataset_creator script 2022-10-31 12:41:28 +01:00
jeremy
5ef08f841a Update Dataset creation script 2022-10-31 11:56:20 +01:00
Kostas Kyrimis
59c7d81ae8
[🍍 < T1086-MG] Test distributed operators e2e (#607)
* Fix Explain queries
* Add Vertex/Edge accessor support for properties
* Fix projections
* Fix expansions to fetch destination vertex properties
* Fix improper use of ShardMap on bolt and replaced it with the ShardRequestManager 
* Add NameToId mappers on ShardRequestManager
* Add e2e tests for operators
* Fix OPTIONAL MATCH
2022-10-31 11:52:20 +02:00
jeremy
03c095e780 Update assert 2022-10-31 10:52:06 +01:00
Jeremy B
b542e49b3e
Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-31 09:44:34 +01:00
Jeremy B
ddb30f49ea
Update datasets.py 2022-10-31 09:43:22 +01:00
Tyler Neely
eafccaea84
Add a logarithmically bucketing histogram and record response latencies per message type (#616) 2022-10-26 15:57:11 +02:00
János Benjamin Antal
f6e78ce6da
Fix TODOs in storage engine (#614)
Fixed various TODOs that were easy to fix to improve the code quality of the
newly implemented storage.
2022-10-26 10:40:35 +02:00
János Benjamin Antal
3cf79f5bbf
Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-25 21:24:20 +02:00
gvolfing
ca2351124b
Make primary labels act as label indices (#605)
Because of the lexicographical sharding, the primary labels themselves are
acting as indexes. If a primary label is specified in a MATCH query we can
safely narrow the range of shards we have to scan through based on that label.
This PR introduces the necessary changes in order to achieve that.
2022-10-25 19:48:17 +02:00
János Benjamin Antal
332afadf21
Split file parsing (#600)
Add temporary support for split files. This is only temporary solution until we
get the shard splitting implemented.
2022-10-25 10:27:13 +02:00
Tyler Neely
acc655f4fd
Model-based testing of simulated full cluster (#584)
This PR adds support for generating randomized workloads that will be executed
against a simulated cluster, as well as against a correctness model. Initially
this just generates ScanAll and CreateVertex requests, and anything that it
creates, it also inserts into a `std::set`, and when we do a ScanAll, it asserts
that we get the same number of requests back. This will become much more
sophisticated over time, but it's already hitting pay-dirt.
2022-10-24 19:54:09 +02:00
Tyler Neely
12e7a261aa
operator<< implementations for a few foundational types (#604) 2022-10-21 15:25:40 +02:00
Marko Budiselić
1c3bb969e9
Decouple interactive planning manual test (#585) 2022-10-21 13:34:13 +02:00
Jure Bajic
e5437080c5
Handle OrderBy in ScanVertices request (#594)
- Refactor shard_rsm and move function into expr.hpp/expr.cpp and request_helper.hpp/request_helper.cpp
2022-10-21 12:33:40 +02:00
Tyler Neely
17090dd8ac
Load SPDLOG_LEVEL environment variable in unit tests binaries (#606)
Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>
2022-10-20 18:27:52 +02:00
gvolfing
281cebd386
Add filtering capabilities to ExpandOne request (#581) 2022-10-20 17:51:44 +02:00
János Benjamin Antal
f89a2bbf42
Make ExpandOne work in the query engine (#589) 2022-10-20 11:35:00 +02:00
gvolfing
85b8ce9101
Implement filtering capabilities for ScanAll (#578)
ScanVerticesrequest was not able to utilize filtering capabilities
before. With these modification it is now able to filter the scanned
vertices based on the filter_expressions field in the
ScanVerticesRequest message type.
2022-10-19 16:09:00 +02:00
Jure Bajic
6bb40a7f49
Create ExpandOne request (#580)
Refactor CreateEdge into CreateExpand
2022-10-19 13:55:46 +02:00
Jeremy B
1707ee648e
Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-19 08:47:13 +02:00
gvolfing
07f34838bd
Add e2e test to check ScanAll on new architecture (#603)
Add testing capabilities similar to what existed before on the older
architecture. With this test specifically, test if we can create
vertices and return them with ScanAll. There is a hard-coded 3s wait in
the test which is needed because the MachineManager need this time to
initialize itself.
2022-10-18 21:38:56 +02:00
Jeremy B
b6f8b5e130
Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-18 15:49:19 +02:00
jeremy
58243f4a26 Rename User->File
Use parser for argument i.o. simple variable in script
2022-10-18 15:47:13 +02:00
gvolfing
d06132cb33
Make ShardRequestManager work with futures (#588)
The communication between the ShardRequestManager and the RsmClient
used to be direct. In this PR this changes into a future-based
communication type. The RsmClient stores state about the currently
processed future (either read or write request) and exposes blocking
and non-blocking functionality to obtain the filled future. The
ShardRequestManager -for now- will send of the set of requests present
in the ExecutionState and block on each of them until the requests are
completed or the set of paginated responses(caused by, for example the
batch-limit in ScanAll) are ready for the next round.
2022-10-18 09:53:37 +02:00
Jeremy B
f063c1b1ad
Merge branch 'project-pineapples' into T1079-MG-add-simple-query-to-benchmark_v2 2022-10-17 11:41:55 +02:00
Marko Budiselić
5347c06d76
Add shutdown of LocalSystem during the shutdown process (#592) 2022-10-14 11:59:50 +02:00
Tyler Neely
4634ac484a
Optimize simulator RNG usage (#590)
This causes instructions for `basic_request.cpp` to drop from 281 million to 28 million
2022-10-12 19:19:06 +03:00
jeremy
a5dc818e19 Add new dataset for mgbench 2022-10-12 16:35:47 +02:00
Jeremy B
4cb3b064c4
Add filter to scan all (#575)
Add several versions of ScanAll with filters.
Add helper function to transform an expression into string that can be parsed again once on the storage.
2022-10-12 11:46:59 +02:00
Jure Bajic
23171e76b6
Integrate bolt server (#572)
* Use query-v2 in the main executable
* Set up machine manager in memgraph
* Add `ShardRequestManager` to `Interpreter`
* Make vertex creation work
* Make scan all work
* Add edge type map in shard request manager
* Send schema over request
* Empty out DbAccessor
* Store shard mapping at creation
* Remove failing CI steps

Cooltura is the best place in Zagreb!

Co-authored-by: János Benjamin Antal <benjamin.antal@memgraph.io>
2022-10-11 16:31:46 +02:00
Jeremy B
6fd64d31f2
Update usage of Shard (#574)
Updating tests to use new constructor of Shard
Commenting test shard_request_manager
2022-10-05 11:56:36 +02:00
gvolfing
87111b2f89
Implement missing message handlers (#563)
Implement the missing message handlers with basic functionality. The
implementation does not include any capabilities to filter based on
expressions.
2022-10-03 15:31:06 +02:00
Tyler Neely
b5c7078c7d
Stitch request manager with shard (#570)
Fix various bugs
2022-09-23 20:07:41 +02:00
Kostas Kyrimis
925835b080
Implement query engine client (#531)
- Add shard request manager
2022-09-22 15:05:43 +02:00
Tyler Neely
ce788f5f65
Machine manager and shard stitch (#569) 2022-09-22 13:55:16 +02:00
János Benjamin Antal
b8186bea2e
Lexicographically ordered storage (#454) 2022-09-21 18:58:31 +02:00
János Benjamin Antal
3c4856dcb7 Merge remote-tracking branch 'origin/project-pineapples' into E118-MG-lexicographically-ordered-storage 2022-09-21 18:50:00 +02:00