Summary:
This binary is installed and packaged for release. This is just a quick
solution for releasing the Community 0.10 version. We still need to
setup the installation and packaging for both the Enterprise and
Community versions. Additionally, the automated build system needs to
test both binaries for correct behaviour. Obviously, some tests can only
be run on one of the 2 versions.
Reviewers: mferencevic, buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1363
Summary:
Use /var/lib/memgraph as home dir for memgraph user.
Merge post and pre RPM scripts in RPM spec file.
Properly handle memgraph.service and directory permissions.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1350
Summary:
This is just the basic stuff needed to produce a RPM package of
Memgraph. Further work needs to be done in order for this to be up to
snuff.
Reviewers: mferencevic, buda
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1318
Summary:
This should allow us to distributed Memgraph executables on machines
which have different versions of libstdc++.
Reviewers: mferencevic, buda
Reviewed By: buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1307
Summary: It seems that boost serialize doesn't work well with ltalloc.
Reviewers: mferencevic, dgleich, teon.banek
Reviewed By: mferencevic, dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1233
Summary:
Added wrappers for some Unix system calls in utils/filesystem.hpp and implemented
a simple log storage interface for Raft. It is not very efficient, we will need
something more sophisticated later, but this is good enough for testing.
Reviewers: mferencevic, mislav.bradac, buda, mculinovic
Reviewed By: mferencevic
Subscribers: teon.banek, dgleich, pullbot
Differential Revision: https://phabricator.memgraph.io/D1091
Summary:
Although the first solution used cereal, the final implementation uses
boost. Since the cereal is still used in the codebase, compilation has
been modified to support multithreaded cereal.
In addition to serializing Ast classes, the following also needed to be
serialized:
* GraphDbTypes
* Symbol
* TypedValue
TypedValue is treated specially, by inlining the serialization code in
the Ast class, concretely PrimitiveLiteral.
Another special case was the Function Ast class, which now stores a
function name which is resolved to a concrete std::function on
construction.
Tests have been added for serialized Ast in
tests/unit/cypher_main_visitor
Reviewers: mferencevic, mislav.bradac, florijan
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1067
Summary:
Examples are added in release/examples directory. Each example
must have its own directory with populate.cyp file inside it.
This file contains graph creation queries written in OpenCypher.
When memgraph is built, database snapshots for each example are
created in release/examples/build directory. During memgraph
installation these snapshots are copied to share/memgraph/examples.
Reviewers: teon.banek, mferencevic
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1036
Summary:
Split main CMakeLists into src/CMakeLists
The main CMakeLists duty is to make all the required libraries and
variables visible to all of the other sub-CMakeLists. After doing that,
it should include those sub-CMakeLists according to configuration
options.
This should make global configurations easier to reuse without polluting
the global space with locally related configurations. It is a necessary
step for including other projects like 'tools' in the release
installation.
Building tools is automatically disabled, but can be enabled by setting
the TOOLS option to ON when running cmake. This should allow on demand
building as well as combined installation of Memgraph and its tools.
Reviewers: mferencevic, buda
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1018
Summary:
It occurred that part of the durability flakyness test might be that the
same durability directory is used always. If the test is run
simultaneously on a single system, there will be interference.
This might not actually fix all the flakyness :(
I also made the `utils::RandomString` function since that's now used in
multiple places, tested it etc.
Reviewers: buda, dgleich
Reviewed By: dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1020
Summary:
The current idea is that the same MG binary can be used for single-node,
distributed master and distributed worker. The transactional engine in
the single-node and distributed master is the same: it determines the
transactional time and exposes all the "global" functionalities. In the
distributed worker the "global" functions must contact the master.
Reviewers: dgleich, mislav.bradac, buda
Reviewed By: dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1013
Summary:
Looking for connected components in a random graph. This test performs the following:
- Generates a random graph that is NOT sequential in memory (otherwise itertion over edges is 2 or more times faster).
- Connectivity by iterating over all the edges.
- Ditto over vertices.
- Ditto over vertices in parallel.
Not done:
- Edge filtering based on XY. I could/should add that to see how it affects perf.
- Getting component info out from union-find.
Local results are encouraging. Iterating over the graph is the bottleneck. Still, I get connectivity of 10M vertices/edges in <7sec (parallel over vertices). Will test on 250M remote now.
Locally obtained results (20M/20M, 2 threads)
```
I1115 14:57:55.136875 357 otto_parallel.cpp:50] Generating 2000000 vertices...
I1115 14:58:19.057734 357 otto_parallel.cpp:74] Generated 2000000 vertices in 23.9208 seconds.
I1115 14:58:19.919221 357 otto_parallel.cpp:82] Generating 2000000 edges...
I1115 14:58:39.519951 357 otto_parallel.cpp:93] Generated 2000000 edges in 19.3398 seconds.
I1115 14:58:39.520349 357 otto_parallel.cpp:196] Running Edge iteration...
I1115 14:58:43.857264 357 otto_parallel.cpp:199] Done in 4.33691 seconds, result: 3999860270398
I1115 14:58:43.857316 357 otto_parallel.cpp:196] Running Vertex iteration...
I1115 14:58:49.498181 357 otto_parallel.cpp:199] Done in 5.64087 seconds, result: 4000090070787
I1115 14:58:49.498208 357 otto_parallel.cpp:196] Running Connected components - Edges...
I1115 14:58:54.232530 357 otto_parallel.cpp:199] Done in 4.73433 seconds, result: 323935
I1115 14:58:54.232570 357 otto_parallel.cpp:196] Running Connected components - Vertices...
I1115 14:59:00.412395 357 otto_parallel.cpp:199] Done in 6.17983 seconds, result: 323935
I1115 14:59:00.412422 357 otto_parallel.cpp:196] Running Parallel connected components - Vertices...
I1115 14:59:04.662087 357 otto_parallel.cpp:199] Done in 4.24967 seconds, result: 323935
I1115 14:59:04.662116 357 otto_parallel.cpp:196] Running Expansion...
I1115 14:59:13.913015 357 otto_parallel.cpp:199] Done in 9.25091 seconds, result: 323935
```
Reviewers: buda, mislav.bradac, dgleich, teon.banek
Reviewed By: buda, teon.banek
Subscribers: teon.banek, pullbot
Differential Revision: https://phabricator.memgraph.io/D982
Summary:
Add postinst script for DEB package
The script creates a 'memgraph' group and sets permission on installed
'/var/*/memgraph' directories. Only the group is created, while
'memgraph' user is not. It seems more sane only to require group
membership for using memgraph.
Add conffiles for DEB package
This allows for `dpkg` to detect changes in configuration files and
present them to the user. Therefore, we don't need to care whether the
configuration merges are handled correctly nor if we accidentally
overwrite them.
Add postrm script for DEB packaging
The script is only used so that `dpkg --purge` removes '/var/*/memgraph'
directories, even if they contain something.
Add email, longer description and license file to DEB packaging, as well
as a systemd service.
Provide a logrotate configuration and support it in memgraph.
Use DEB package for Docker installation
This way, the whole installation process and testing should go through
DEB.
Generate release archives in Apollo with standard names
Reviewers: buda, mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D989
Summary:
My dear fellow Memgraphians. It's friday afternoon, and I am as ready to pop as WAL is to get reviewed...
What's done:
- Vertices and Edges have global IDs, stored in `VersionList`. Main storage is now a concurrent map ID->vlist_ptr.
- WriteAheadLog class added. It's based around buffering WAL::Op objects (elementraly DB changes) and periodically serializing and flusing them to disk.
- Snapshot recovery refactored, WAL recovery added. Snapshot format changed again to include necessary info.
- Durability testing completely reworked.
What's not done (and should be when we decide how):
- Old WAL file purging.
- Config refactor (naming and organization). Will do when we discuss what we want.
- Changelog and new feature documentation (both depending on the point above).
- Better error handling and recovery feedback. Currently it's all returning bools, which is not fine-grained enough (neither for errors nor partial successes, also EOF is reported as a failure at the moment).
- Moving the implementation of WAL stuff to .cpp where possible.
- Not sure if there are transactions being created outside of `GraphDbAccessor` and it's `BuildIndex`. Need to look into.
- True write-ahead logic (flag controlled): not committing a DB transaction if the WAL has not flushed it's data. We can discuss the gain/effort ratio for this feature.
Reviewers: buda, mislav.bradac, teon.banek, dgleich
Reviewed By: dgleich
Subscribers: mtomic, pullbot
Differential Revision: https://phabricator.memgraph.io/D958
Summary:
The warnings lifted to errors should help us track down cases which
haven't been covered in a switch. Obviously, using the default case will
not trigger these errors, so default should only be used when we
actually want to cover more cases.
Reviewers: florijan, dgleich, mferencevic
Reviewed By: dgleich
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D975
Summary:
Remove name from GraphDb.
Take GraphDb in query test macros instead of accessor.
Add is_accepting_transactions flag to GraphDb.
Reviewers: mislav.bradac, florijan, mferencevic
Reviewed By: mislav.bradac
Subscribers: mferencevic, pullbot
Differential Revision: https://phabricator.memgraph.io/D940
Summary:
Move QueryParts and Filters to a new file.
Reorganize FilterInfo struct.
Remove label filter if we do indexed scan by label.
Remove property filter used in indexed scan.
Reviewers: florijan
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D915
Summary:
There seems to be a gain all over memgraph.
I strongly suggest including this in the codebase.
Link to project:
https://github.com/r-lyeh/ltalloc
Reviewers: teon.banek, mferencevic, mislav.bradac, florijan
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D914
Summary:
Benchmark conducted showed positive performance change, a gain of ~20%.
Setup:
ccache disabled (export CCACHE_DISABLE=1)
cd build
rm -rf *
cmake ..
time make -j8
without gold linker
real 3m25,384s
user 11m58,337s
sys 0m37,747s
real 3m48,087s
user 12m57,600s
sys 0m36,837s
real 3m20,116s
user 12m9,570s
sys 0m33,643s
with gold linker
real 2m48,073s
user 10m2,257s
sys 0m27,480s
real 2m55,673s
user 10m13,420s
sys 0m27,233s
real 2m47,866s
user 10m2,377s
sys 0m27,323s
Reviewers: teon.banek, mislav.bradac, florijan, mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D903
Summary:
This should be the basis for default configuration that is to be shipped
in release builds/packages.
Reviewers: mislav.bradac, florijan, buda, mferencevic
Reviewed By: florijan
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D851
Summary:
This puts the whole installation and packaging under a single point of
entry. (Docker, DEB, RPM, etc.)
Rename alpha.dockerfile to beta.dockerfile
Use Debian Stretch for docker
Remove building old hardcoded compiler
Rename build_interpreter to build_memgraph
Remove unused config-file
Reviewers: mferencevic, buda
Reviewed By: mferencevic, buda
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D857
Summary:
Add json and cppitertools to libs/CMakeLists.txt.
Import external projects as libraries.
This removes the need to use `add_dependencies` in order to link with
external project.
Extract common ExternalProject_Add function.
Add macro for easier addition of external libraries.
Reviewers: mislav.bradac, mferencevic
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D845
Summary:
There's no need to rebuild antlr every time we recompile memgraph. This
should speed up the `make clean all` cycle.
Reviewers: mislav.bradac, mferencevic, buda
Reviewed By: mislav.bradac
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D817
Summary:
I started with cleaning flags up (removing unused ones, documenting undocumented ones). There were some flags to remove in `QueryEngine`. Seeing how we never use hardcoded queries (AFAIK last Mislav's testing also indicated they aren't faster then interpretation), when removing those unused flags the `QueryEngine` becomes obsolete. That means that a bunch of other stuff becomes obsolete, along with the hardcoded queries. So I removed it all (this has been discussed and approved on the daily).
Some flags that were previously undocumented in `docs/user_technical/installation` are now documented. The following flags are NOT documented and in my opinion should not be displayed when starting `./memgraph --help` (@mferencevic):
```
query_vertex_count_to_expand_existsing (from rule_based_planner.cpp)
query_max_plans (rule_based_planner.cpp)
```
If you think that another organization is needed w.r.t. flag visibility, comment.
@teon.banek: I had to remove some stuff from CMakeLists to make it buildable. Please review what I removed and clean up if necessary if/when this lands. If the needed changes are minor, you can also comment.
Reviewers: buda, mislav.bradac, teon.banek, mferencevic
Reviewed By: buda, mislav.bradac
Subscribers: pullbot, mferencevic, teon.banek
Differential Revision: https://phabricator.memgraph.io/D825
Summary:
Setup scaffolding for building Memgraph tools.
Change `utils::Split` without delimiter to split on whitespace.
This should make `Split` behave just like Python's `str.split`, which is
more practical for splitting on word boundaries.
Add `utils::StartsWith` function.
Rewrite csv_to_snapshot to C++.
Reviewers: mferencevic
Reviewed By: mferencevic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D822
