Commit Graph

40 Commits

Author SHA1 Message Date
Teon Banek
eff843bbcc Tidy up the RPM packaging
Summary:
Use /var/lib/memgraph as home dir for memgraph user.
Merge post and pre RPM scripts in RPM spec file.
Properly handle memgraph.service and directory permissions.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1350
2018-04-12 10:31:38 +02:00
Matija Santl
7b88e514b8 Add ClusterDiscovery RPC for distributed BFS
Summary:
Implemented cluster discovery in distributed memgraph.
When a worker registers, it sends a RPC request to master.
The master assigns that worker an id and sends the information about other
workers (pairs of <worker_id, endpoint>) to the new worker.
Master also sends the information about the new worker to all existing workers
in the process of worker registration.

After the last worker registers, all memgraph instances in the clusters should
know about every other.

Reviewers: mtomic, buda, florijan

Reviewed By: mtomic

Subscribers: teon.banek, dgleich, pullbot

Differential Revision: https://phabricator.memgraph.io/D1339
2018-04-09 14:28:22 +02:00
Matej Ferencevic
71565287b8 Remove obsolete network buffers
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1336
2018-04-06 16:26:46 +02:00
Dominik Gleich
b20e31e800 Synchronize snapshooting
Summary: Make synchronized snapshot. This invokese the snapshooter on workers on the master snapshot scheduler interval.

Reviewers: msantl, mtomic

Reviewed By: msantl

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1334
2018-04-06 10:10:01 +02:00
Matej Ferencevic
485e1988c3 Split signals utils into cpp and hpp
Reviewers: buda

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1335
2018-04-03 11:17:11 +02:00
Teon Banek
b342b09b49 Remove hardcoded build directory in build_examples
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1323
2018-03-28 10:17:11 +02:00
Matej Ferencevic
f1a8d7cd3d Refactor network layer to use streams
Summary:
The network layer now has a `Session` that handles all things that should be
done before the `Execute` method is called on sessions. Also, all sessions
now communicate using streams instead of holding the input buffer and writing
to the `Socket`. This design will allow implementation of a SSL middleware.

Reviewers: buda, dgleich

Reviewed By: buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1314
2018-03-27 15:05:45 +02:00
florijan
ac8c96ccc2 Tidyup distributed stuff naming
Summary:
Remove "produce_" and "Produce" as prefix from all distributed stuff.
It's not removed in src/query/ stuff (operators).

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1315
2018-03-23 16:32:29 +01:00
florijan
eb0e2cb31b Extract cpp from hpp in distributed, fix includes
Summary:
Also removed some convenience code that became obsolete.
No logic changes.

Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1303
2018-03-21 09:38:44 +01:00
Marin Tomic
2263a423b8 Add deletion, more stats to card fraud and RWLock
Summary:
^^

this is a sample config to be used:
```
{
    "num_workers": 1,
    "cards_per_worker": 10001,
    "pos_per_worker": 10001,

    "fraud_probability": 0.01,
    "hop_probability": 0.05,

    "cleanup": {
        "check_interval_sec": 10,
        "tx_hi": 150000,
        "tx_lo": 100000
    },

    "analytic": {
        "query_interval_ms": 500,
        "pos_limit": 10
    }
}
```

I also added `RWLock` --- a wrapper around `pthread_rwlock`

Reviewers: mferencevic, mculinovic, florijan, teon.banek

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1280
2018-03-14 16:23:23 +01:00
florijan
848749cf25 Fix and rename REPL
Summary: Fix and rename REPL

Reviewers: teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1284
2018-03-08 10:58:43 +01:00
florijan
bb62f463f8 Refactor distributed transactional cache GC
Summary:
Release of per-transaction data in distributed Memgraph refactored. The
master node no longer releases each time a transaction is done, thus
offloading some work from the engine.

Reviewers: dgleich

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1235
2018-02-27 10:47:58 +01:00
Marin Tomic
5ff1ca4a5d Add client side RPC stats
Summary: ^^

Reviewers: mferencevic, buda

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1224
2018-02-22 12:34:48 +01:00
florijan
9721ccf61c Cleanup per-transaction caches in distributed
Summary:
On the master cleanups are hooked directly into the transaction engine.
This is beneficial because the master might have bigger caches and we
want to clear them as soon as possible.

On the workers there is a periodic RPC call to the master about living
transactions, which takes care of releasing local caches. This is
suboptimal because long transactions will prevent cache GC (like with
data GC). It is however fairly simple.

Note that all cleanup is not done automatically and `RemotePull` has
been reduced accordingly. @msantl, please verify correctness and
consider if the code can be additionally simplified.

Reviewers: teon.banek, msantl

Reviewed By: msantl

Subscribers: pullbot, msantl

Differential Revision: https://phabricator.memgraph.io/D1202
2018-02-16 15:30:05 +01:00
Matej Ferencevic
763bdaac0f Add support for large messages to RPC
Reviewers: buda, teon.banek, mculinovic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1196
2018-02-16 12:38:31 +01:00
Marin Tomic
d15464e181 Add some metric types and basic RPC server stats
Summary:
Get rid of client class

Fix cppcheck errors

Add documentation to metrics.hpp

Add documentation to stats.hpp

Remove stats from global namespace

Fix build failures

Refactor a bit

Refactor stopwatch into a function

Add rpc execution time stats

Fix segmentation fault

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1200
2018-02-16 08:33:15 +01:00
Marin Tomic
dca1e9eebc Add initial version of statsd
Reviewers: mferencevic, teon.banek, buda, mculinovic

Reviewed By: mferencevic

Subscribers: teon.banek, pullbot

Differential Revision: https://phabricator.memgraph.io/D1152
2018-02-02 17:02:37 +01:00
Teon Banek
dbc9c995c9 Plan distributed execution of basic read operators
Summary:
Remove AdvanceCommand operator declaration.
Add query/plan/distributed source files.
Add hacked cloning of LogicalOperator via serialization.
Add virtual Default methods to CompositeVisitor.

Use BOOST_CLASS_EXPORT_KEY in ast.hpp and operator.hpp.

This is needed in a single binary to correctly serialize polymorphic
classes. The previous implementation worked because the memgraph_lib was
linked with test binaries, but nobody was actually serializing things
inside the single memgraph binary itself.

Print PullRemote symbols in tests/manual/query_planner

Add names to implicitly created aggregation symbols

Reviewers: florijan, msantl

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1137
2018-01-25 14:20:51 +01:00
Matej Ferencevic
fc20ddcd25 RPC refactor
Summary:
Start removal of old logic
Remove more obsolete classes
Move Message class to RPC
Remove client logic from system
Remove messaging namespace
Move protocol from messaging to rpc
Move System from messaging to rpc
Remove unnecessary namespace
Remove System from RPC Client
Split Client and Server into separate files
Start implementing new client logic
First semi-working state
Changed network protocol layout
Rewrite client
Fix client receive bug
Cleanup code of debug lines
Migrate to accessors
Migrate back to binary boost archives
Remove debug logging from server
Disable timeout test
Reduce message_id from uint64_t to uint32_t
Add multiple workers to server
Fix compiler warnings
Apply clang-format

Reviewers: teon.banek, florijan, dgleich, buda, mtomic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1129
2018-01-24 15:27:40 +01:00
Matija Santl
3bee31d8f3 Add PlanDispatcher and PlanConsumer stubs
Reviewers: teon.banek, florijan

Reviewed By: florijan

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1126
2018-01-23 10:15:24 +01:00
Teon Banek
98cbd2b500 Cleanup utils::File API
Summary:
Close the file descriptor in File destructor. This will prevent
accidental crashes during unexpected destructor calls. For example, if
an exception is thrown before the file is closed. File now takes
ownership of the descriptor. These changes now honor RAII idiom, which
should handle most of the peculiarities of C++.

Use optional value for TryOpenFile function, instead of returning a File
without a descriptor. It makes the failure state more semantically clear
to the API user.

Merge utils/filesystem with utils/file

The files aren't that big, and the naming is a bit confusing because
functions aren't really grouped for file and filesystem distinction.

Reviewers: mferencevic, mtomic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1111
2018-01-16 15:38:12 +01:00
Marin Tomic
9e42ebbb67 Implement simple log file storage for raft
Summary:
Added wrappers for some Unix system calls in utils/filesystem.hpp and implemented
a simple log storage interface for Raft. It is not very efficient, we will need
something more sophisticated later, but this is good enough for testing.

Reviewers: mferencevic, mislav.bradac, buda, mculinovic

Reviewed By: mferencevic

Subscribers: teon.banek, dgleich, pullbot

Differential Revision: https://phabricator.memgraph.io/D1091
2018-01-15 18:07:45 +01:00
Dominik Gleich
5418dfb19e Rename NetworkEndpoint
Summary:
Rename redunant port str

Add endpoint << operator

Migrate everything to endpoint

Reviewers: mferencevic, florijan

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1100
2018-01-15 15:47:37 +01:00
Matej Ferencevic
41679b6ec5 Integrate code coverage with Apollo
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1050
2018-01-15 12:20:11 +01:00
florijan
6fc6a27288 Refactor GraphDb
Summary:
GraphDb is refactored to become an API exposing different parts
necessary for the database to function. These different parts can have
different implementations in SingleNode or distributed Master/Server
GraphDb implementations.

Interally GraphDb is implemented using two class heirarchies. One
contains all the members and correct wiring for each situation. The
other takes care of initialization and shutdown. This architecture is
practical because it can guarantee that the initialization of the
object structure is complete, before initializing state.

Reviewers: buda, mislav.bradac, dgleich, teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1093
2018-01-12 16:47:24 +01:00
florijan
c4327b26f4 Extract tx::SingleNodeEngine from tx::MasterEngine
Summary:
No logic changes, just split `tx::MasterEngine` into
`tx::SingleNodeEngine` and `tx::MasterEngine`. This gives better
responsibility separation and is more appropriate now there is no
Start/Shutdown.

Reviewers: dgleich, teon.banek, buda

Reviewed By: dgleich, teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1099
2018-01-11 15:44:42 +01:00
florijan
4a0345e1c5 Prepare counter for distributed
Reviewers: dgleich, teon.banek

Reviewed By: teon.banek

Differential Revision: https://phabricator.memgraph.io/D1090
2018-01-10 13:28:41 +01:00
Mislav Bradac
d3623585e7 Migrate cereal to boost_serialization
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1077
2017-12-27 13:44:46 +01:00
Teon Banek
9cacf970cc Install memgraph in subdirectory of lib
Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1076
2017-12-21 16:53:55 +01:00
Marin Tomic
e999207b2f Remove reactors
Summary: kill reactors

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Differential Revision: https://phabricator.memgraph.io/D1075
2017-12-20 17:56:57 +01:00
florijan
312016ca31 Support both locally and system installed libboost
Reviewers: teon.banek

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1071
2017-12-20 16:13:03 +01:00
Teon Banek
5f7837d613 Serialize Ast classes
Summary:
Although the first solution used cereal, the final implementation uses
boost. Since the cereal is still used in the codebase, compilation has
been modified to support multithreaded cereal.

In addition to serializing Ast classes, the following also needed to be
serialized:

  * GraphDbTypes
  * Symbol
  * TypedValue

TypedValue is treated specially, by inlining the serialization code in
the Ast class, concretely PrimitiveLiteral.

Another special case was the Function Ast class, which now stores a
function name which is resolved to a concrete std::function on
construction.

Tests have been added for serialized Ast in
tests/unit/cypher_main_visitor

Reviewers: mferencevic, mislav.bradac, florijan

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1067
2017-12-20 13:28:34 +01:00
florijan
3ae45e0d19 Add master/worker flags, main functions and coordination
Reviewers: dgleich, mislav.bradac, buda

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1060
2017-12-19 16:05:57 +01:00
florijan
37722e54b3 Add RPC to concurrent ID mapper
Summary:
The distributed ID mapper is not yet utilised in GraphDb as those
changes are in D1060. Depending on landing order it will be added.

Reviewers: dgleich, mislav.bradac

Reviewed By: dgleich

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1064
2017-12-19 09:11:16 +01:00
Marko Culinovic
52a9be3cc8 Add examples to installation
Summary:
Examples are added in release/examples directory. Each example
must have its own directory with populate.cyp file inside it.
This file contains graph creation queries written in OpenCypher.
When memgraph is built, database snapshots for each example are
created in release/examples/build directory. During memgraph
installation these snapshots are copied to share/memgraph/examples.

Reviewers: teon.banek, mferencevic

Reviewed By: teon.banek

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1036
2017-12-12 14:59:54 +01:00
Teon Banek
0d40f6a759 Merge memgraph_link_target into memgraph executable target
Summary:
Memgraph's executable target is unnecessarily complicated by having a
name which depends on the commit hash and build type. It is now renamed
to simply 'memgraph', while the executable output uses the encoding of
commit hash and build type as the name. This simplifies the dependency
handling in cmake, where now the full path to the executable can be
easily obtained by using the generator expression:
`$<TARGET_FILE:memgraph>`.

In addition to the above simplification, memgraph_link_target is now
replaced with a post build step on memgraph target itself. This
guarantees that the symlink always exists after building the executable,
as well as simplify the make invocation. The invocation now boils down
to `make memgraph` instead of `make memgraph_link_target`.

Reviewers: mferencevic

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1043
2017-12-11 16:11:19 +01:00
Marin Tomic
0bf692f8ec Refactor Raft leader election to use multiple threads
Summary:
Implement log replication

Rebase and fix src/CMakeLists.txt

Some style fixes

Changed shared_ptr to unique_ptr for RaftPeerState

Change Id and Leader to const

Move implementation to separate class

Fix raft_experiments.cpp

Reviewers: mislav.bradac

Reviewed By: mislav.bradac

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1033
2017-12-08 16:30:57 +01:00
Dominik Gleich
03db948d7e Refactor operations into StateDeltas
Summary: Operations are moved and renamed from WAL to a separate file in preparation for HA and distributed storage.

Reviewers: florijan, mtomic, mislav.bradac

Reviewed By: florijan

Subscribers: mislav.bradac, pullbot

Differential Revision: https://phabricator.memgraph.io/D1034
2017-12-07 14:23:41 +01:00
Mislav Bradac
60f4db2b9f Add first version of message passing and rpc
Reviewers: mtomic, buda

Reviewed By: mtomic, buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1002
2017-12-05 14:51:18 +01:00
Teon Banek
a17261038c Include tools CMakeLists in the top level CMakeLists
Summary:
Split main CMakeLists into src/CMakeLists

The main CMakeLists duty is to make all the required libraries and
variables visible to all of the other sub-CMakeLists. After doing that,
it should include those sub-CMakeLists according to configuration
options.

This should make global configurations easier to reuse without polluting
the global space with locally related configurations. It is a necessary
step for including other projects like 'tools' in the release
installation.

Building tools is automatically disabled, but can be enabled by setting
the TOOLS option to ON when running cmake. This should allow on demand
building as well as combined installation of Memgraph and its tools.

Reviewers: mferencevic, buda

Reviewed By: mferencevic

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D1018
2017-12-04 15:18:18 +01:00