Summary: Run the basic test with two cluster sizes, 3 and 5.
Reviewers: ipaljak
Reviewed By: ipaljak
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1794
Summary:
Created a new integration test for Raft protocol.
The tests iterates through the Raft cluster and does the following:
* kill machine `X`
* execute a query
* bring `X` back to life
The first step is to insert a vertex in the cluster, and last step is to check
if the cluster has all the data.
I also edited some of the raft core files because this test surafaced some bugs.
The `tester` binary is a hacked version of the HA client and so are the parts in
the code that refuse to execute a query is the machine is not in `Leader` mode.o
Those parts will go away once we have a proper HA client.
I've run the `runner.py` for a while (215 times)
```
while ./runner.py &> log.txt; do echo -n "."; done
```
and it didn't break.
Reviewers: ipaljak, mferencevic
Reviewed By: ipaljak
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1788
Summary:
This diff splits single node and distributed storage from each other.
Currently all of the storage code is copied into two directories (one single
node, one distributed). The logic used in the storage implementation isn't
touched, it will be refactored in following diffs.
To clean the working directory after this diff you should execute:
```
rm database/state_delta.capnp
rm database/state_delta.hpp
rm storage/concurrent_id_mapper_rpc_messages.capnp
rm storage/concurrent_id_mapper_rpc_messages.hpp
```
Reviewers: teon.banek, buda, msantl
Reviewed By: teon.banek, msantl
Subscribers: teon.banek, pullbot
Differential Revision: https://phabricator.memgraph.io/D1625
Summary:
This diff changes the RPC layer to directly return `TResponse` to the user when
issuing a `Call<...>` RPC call. The call throws an exception on failure
(instead of the previous return `nullopt`).
All servers (network, RPC and distributed) are set to have explicit `Shutdown`
methods so that a controlled shutdown can always be performed. The object
destructors now have `CHECK`s to enforce that the `AwaitShutdown` methods were
called.
The distributed memgraph is changed that none of the binaries (master/workers)
crash when there is a communication failure. Instead, the whole cluster starts
a graceful shutdown when a persistent communication error is detected.
Transient errors are allowed during execution. The transaction that errored out
will be aborted on the whole cluster. The cluster state is managed using a new
Heartbeat RPC call.
Reviewers: buda, teon.banek, msantl
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1604
Summary:
This change improves detection of errorneous situations when starting a
distributed cluster on a single machine. It asserts that the user hasn't
started more memgraph nodes on the same machine with the same durability
directory. Also, this diff improves worker registration. Now workers don't have
to have explicitly set IP addresses. The master will deduce them from the
connecting IP when the worker registers.
Reviewers: teon.banek, buda, msantl
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1582
Summary: A quick clean-up of user visible error messages. Tried to make them gramatically correct by capitalizing the first word in the sentence and putting a dot at the end.
Reviewers: teon.banek, buda, ipaljak
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1571
Summary:
Changed GRANT ROLE to SET ROLE. Now it is `SET ROLE FOR user TO ROLE` instead of `GRANT ROLE role TO user`. It makes more sense because our users can only have 1 role.
Changed REVOKE ROLE to CLEAR ROLE. Now it is `CLEAR ROLE FOR user` instead of `REVOKE ROLE role FOR user`. REVOKE ROLE would throw exception if user was not a member of role. CLEAR ROLE clears the role whatever it is. I find that the latter makes more sense combined with SET ROLE.
Changed `SHOW ROLE FOR USER user` to `SHOW ROLE FOR user`.
Changed `SHOW USERS FOR ROLE role` to `SHOW USERS FOR role`.
Reviewers: mferencevic, teon.banek, buda
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1572
Summary:
First iteration in implementing kafka.
Currently, memgraph streams won't use the transform script provided in the
`CREATE STREAM` query.
There is a manual test that serves a POC purpose which we'll use to fully wire
kafka in memgraph.
Since streams need to download the script, I moved curl init from
telemetry.
Reviewers: teon.banek, mferencevic
Reviewed By: mferencevic
Subscribers: ipaljak, pullbot, buda
Differential Revision: https://phabricator.memgraph.io/D1491
Summary:
This diff implements OpenSSL support in the network stack.
Currently SSL support is only enabled for Bolt connections,
support for RPC connections will be added in another diff.
Reviewers: buda, teon.banek
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1328
Summary:
Add telemetry to main memgraph binary
Add resource usage collector
Add telemetry flag
Change telemetry collector logic
Fix utils compilation
Add timestamp
Add first version of interactive test
Started working on test runner
Implement all tests
Flake8 on runner.py
Integrate test with Apollo
Add TODO
Reviewers: buda, teon.banek
Reviewed By: buda, teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1419
Summary:
I started with cleaning flags up (removing unused ones, documenting undocumented ones). There were some flags to remove in `QueryEngine`. Seeing how we never use hardcoded queries (AFAIK last Mislav's testing also indicated they aren't faster then interpretation), when removing those unused flags the `QueryEngine` becomes obsolete. That means that a bunch of other stuff becomes obsolete, along with the hardcoded queries. So I removed it all (this has been discussed and approved on the daily).
Some flags that were previously undocumented in `docs/user_technical/installation` are now documented. The following flags are NOT documented and in my opinion should not be displayed when starting `./memgraph --help` (@mferencevic):
```
query_vertex_count_to_expand_existsing (from rule_based_planner.cpp)
query_max_plans (rule_based_planner.cpp)
```
If you think that another organization is needed w.r.t. flag visibility, comment.
@teon.banek: I had to remove some stuff from CMakeLists to make it buildable. Please review what I removed and clean up if necessary if/when this lands. If the needed changes are minor, you can also comment.
Reviewers: buda, mislav.bradac, teon.banek, mferencevic
Reviewed By: buda, mislav.bradac
Subscribers: pullbot, mferencevic, teon.banek
Differential Revision: https://phabricator.memgraph.io/D825
Summary: Not strictly neccessary, but it's been itching me. It took an hour.
Reviewers: buda, mislav.bradac
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D648
Summary:
There was only two files in dbms directory so I moved them to database
directory.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D540
Summary:
CMake is smart enough to transitively detect dependencies and link them
appropriately. Therefore, it is enough that we put all libraries that
memgraph uses to the dependency list of memgraph_lib and memgraph_pic
targets.
Patch the fmt library for C++14 and higher
fmt library would detect that C++11 is supported and then put the
compiler flag. This flag was set so it overrides parent project compiler
flags. This override from fmt would prevent us from using C++14
features. New version (3.1) of fmt resolves this issue, but it hasn't
been released yet. Therefore, this commit updates the script which
clones fmt to use the released 3.0.1 version and apply the fix on that.
Reviewers: dgleich, buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D441
Summary:
This should fix the integration query engine test. I'm not exactly sure what caused the SEGFAULT to occur, but from my debugging sessions; I would say, as an educated guess that it's closely related to logger. What I believe happens is that logger gets somehow initialized from the dynamic_library when it's opened. And when the library is closed the corresponding logger is freed, but, other parts of memgraph still see that instance of logger and trying to use to emit code makes the whole thing crash. I've removed one include of logger from something which gets included in the hardcoded queries, but I noticed there are some more.
I've changed the dlopen to now open with DEEPBIND, which should cause symbols to be resolved locally and not globally and stop the logger from the dynamic lib to interfer with the one used in the rest of memgraph.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D234
Summary: Bolt PullAll works. ChunkedBuffer is implemented by the specification (the tests can be found in the tests/unit/bolt_chunked_buffer.cpp). Bolt session states method are refactored (all run methods have only one argument - active Session). Modifications related to the style guide.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot, buda
Differential Revision: https://phabricator.memgraph.io/D180
Summary:
Removed old encoder.
Changed namespace from bolt to communication::bolt.
Removed old include from new encoder.
Added an empty message success to encoder.
Changed order in communication::Server.
Changed bolt session to use new encoder.
Merge remote-tracking branch 'origin/dev' into mg_hardcoded_queries
Fixed PrintRecordStream.
Reviewers: buda, dgleich
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D158
Summary: Db_accessor is tied to one transaction, and as such each query should work on seperate db_accessor.
Reviewers: florijan, buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D96
Summary:
#1 Hardcoded query integration test was failling because the .so files couldn't be linked, because of wrong compiled directory path (missing config flag).
#2 Hardcoded query not working in production because typed_value changed so some includes didn't work.
Reviewers: florijan, buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D90
Summary:
Moved server and worker from bolt to communication. Started
templatization.
Started removal of Bolt class.
Removed unnecessary files from network.
Converted states to template functions.
Bolt::Session is now a template.
Merge remote-tracking branch 'origin/dev' into mg_refactor_network
Merged bolt_serializer.cpp into hpp.
Removed obsolete include.
Initial version of bolt session unit test.
Uncommented leftover log commands.
Reimplemented io::Socket.
Added client-stress.sh script.
Reviewers: dgleich, buda
Reviewed By: dgleich, buda
Subscribers: pullbot, mferencevic, buda
Differential Revision: https://phabricator.memgraph.io/D64
Summary: database/GraphDbAccessor and storage/...Accessor now returns iterators over lazily created Accessors (filtered on visibility). Dependecies and forward declarations reorganized to support new code
Reviewers: buda, teon.banek
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D83
Summary:
Rename TypedValue to PropertyValue.
Move original TypedValue to query/backend/cpp.
Fix undefined behaviours and memory leaks in PropertyValue.
Add list type to PropertyValue.
Operators and appropriate tests will be revised in following commits.
Fix bugs in TypedValue
Fix bugs in typed value
Reviewers: buda, florijan
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D77
Summary: For each hardcoded query plan there is a build target -> that will improve Memgraph's testability + update of release script (release/alpha.sh)
Reviewers: florijan, mislav.bradac, mferencevic, dgleich
Reviewed By: dgleich
Subscribers: pullbot, buda
Differential Revision: https://phabricator.memgraph.io/D75
Summary:
Add vertex stream operator. Implement #0, #1 query.
Support first batch of queries for testDB. Improve record stream handler. Query engine now supports multiline queries patterns in file comments.
Second and third batch of tests complete. Add functionality to print_record_stream.
Include 4th batch.
Crete a shared function for properties output.
Reviewers: buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D60
Importes now use logger.
Refactored order of constrution of objects in Db.
Moved index creation/removing from Db to Indexes.
Completed Garbage class.
Cleaner now calls garbage.clean() for databases.
Renamed List to ConcurrentList which better names it.
