2017-12-04 20:56:17 +08:00
|
|
|
# CMake configuration for the main memgraph library and executable
|
|
|
|
|
|
2018-06-19 20:37:02 +08:00
|
|
|
# add memgraph sub libraries, ordered by dependency
|
2018-05-29 17:13:13 +08:00
|
|
|
add_subdirectory(utils)
|
2018-07-06 15:28:05 +08:00
|
|
|
add_subdirectory(requests)
|
2018-06-19 20:37:02 +08:00
|
|
|
add_subdirectory(integrations)
|
2018-05-30 19:00:25 +08:00
|
|
|
add_subdirectory(io)
|
2018-06-20 19:46:54 +08:00
|
|
|
add_subdirectory(telemetry)
|
Extract communication to static library
Summary:
Session specifics have been move out of the Bolt `executing` state, and
are accessed via pure virtual Session type. Our server is templated on
the session and we are setting the concrete type, so there should be no
virtual call overhead. Abstract Session is used to indicate the
interface, this could have also been templated, but the explicit
interface definition makes it clearer.
Specific session implementation for running Memgraph is now implemented
in memgraph_bolt, which instantiates the concrete session type. This may
not be 100% appropriate place, but Memgraph specific session isn't
needed anywhere else.
Bolt/communication tests now use a dummy session and depend only on
communication, which significantly improves test run times.
All these changes make the communication a library which doesn't depend
on storage nor the database. Only shared connection points, which aren't
part of the base communication library are:
* glue/conversion -- which converts between storage and bolt types, and
* communication/result_stream_faker -- templated, but used in tests and query/repl
Depends on D1453
Reviewers: mferencevic, buda, mtomic, msantl
Reviewed By: mferencevic, mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1456
2018-07-10 22:18:19 +08:00
|
|
|
add_subdirectory(communication)
|
2018-08-22 21:26:51 +08:00
|
|
|
add_subdirectory(stats)
|
2018-07-27 16:54:20 +08:00
|
|
|
add_subdirectory(auth)
|
2018-05-29 17:13:13 +08:00
|
|
|
|
2017-12-04 20:56:17 +08:00
|
|
|
# all memgraph src files
|
|
|
|
|
set(memgraph_src_files
|
|
|
|
|
data_structures/concurrent/skiplist_gc.cpp
|
2018-01-12 22:17:04 +08:00
|
|
|
database/config.cpp
|
2018-08-31 16:32:53 +08:00
|
|
|
database/distributed_counters.cpp
|
Split GraphDb to distributed and single node files
Summary:
This change, hopefully, simplifies the implementation of different kinds
of GraphDb. The pimpl idiom is now simplified by removing all of the
crazy inheritance. Implementations classes are just plain data stores,
without any methods. The interface classes now have a more flat
hierarchy:
```
GraphDb (pure interface)
|
+----+---------- DistributedGraphDb (pure interface)
| |
Single Node +-----+------+
| |
Master Worker
```
DistributedGraphDb is used as an intermediate interface for all the
things that should work only in distributed. Therefore, virtual calls
for distributed stuff have been removed from GraphDb. Some are exposed
via DistributedGraphDb, other's are only in concrete Master and Worker
classes. The code which relied on those virtual calls has been
refactored to either use DistributedGraphDb, take a pointer to what is
actually needed or use dynamic_cast. Obviously, dynamic_cast is a
temporary solution and should be replaced with another mechanism (e.g.
virtual call, or some other function pointer style).
The cost of the above change is some code duplication in constructors
and destructors of classes. This duplication has a lot of little tweaks
that make it hard to generalize, not to mention that virtual calls do
not work in constructor and destructor. If we really care about
generalizing this, we should think about abandoning RAII in favor of
constructor + Init method.
The next steps for splitting the dependencies that seem logical are:
1) Split GraphDbAccessor implementation, either via inheritance or
passing in an implementation pointer. GraphDbAccessor should then
only be created by a virtual call on GraphDb.
2) Split Interpreter implementation. Besides allowing single node
interpreter to exist without depending on distributed, this will
enable the planner and operators to be correctly separated.
Reviewers: msantl, mferencevic, ipaljak
Reviewed By: msantl
Subscribers: dgleich, pullbot
Differential Revision: https://phabricator.memgraph.io/D1493
2018-07-19 23:00:50 +08:00
|
|
|
database/distributed_graph_db.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
database/graph_db.cpp
|
|
|
|
|
database/graph_db_accessor.cpp
|
2017-12-07 20:09:34 +08:00
|
|
|
database/state_delta.cpp
|
2018-05-15 23:38:47 +08:00
|
|
|
distributed/bfs_rpc_clients.cpp
|
|
|
|
|
distributed/bfs_subcursor.cpp
|
2018-04-03 22:19:17 +08:00
|
|
|
distributed/cluster_discovery_master.cpp
|
|
|
|
|
distributed/cluster_discovery_worker.cpp
|
|
|
|
|
distributed/coordination.cpp
|
2017-12-19 19:40:30 +08:00
|
|
|
distributed/coordination_master.cpp
|
|
|
|
|
distributed/coordination_worker.cpp
|
2018-07-02 21:34:33 +08:00
|
|
|
distributed/data_manager.cpp
|
|
|
|
|
distributed/data_rpc_clients.cpp
|
|
|
|
|
distributed/data_rpc_server.cpp
|
2018-09-08 01:45:09 +08:00
|
|
|
distributed/dgp/partitioner.cpp
|
|
|
|
|
distributed/dgp/vertex_migrator.cpp
|
2018-07-17 17:03:03 +08:00
|
|
|
distributed/durability_rpc_master.cpp
|
|
|
|
|
distributed/durability_rpc_worker.cpp
|
2018-03-19 21:42:32 +08:00
|
|
|
distributed/index_rpc_server.cpp
|
2018-01-22 22:24:04 +08:00
|
|
|
distributed/plan_consumer.cpp
|
|
|
|
|
distributed/plan_dispatcher.cpp
|
2018-03-23 22:21:46 +08:00
|
|
|
distributed/produce_rpc_server.cpp
|
|
|
|
|
distributed/pull_rpc_clients.cpp
|
|
|
|
|
distributed/updates_rpc_clients.cpp
|
|
|
|
|
distributed/updates_rpc_server.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
durability/paths.cpp
|
|
|
|
|
durability/recovery.cpp
|
|
|
|
|
durability/snapshooter.cpp
|
|
|
|
|
durability/wal.cpp
|
2018-08-22 16:59:46 +08:00
|
|
|
glue/auth.cpp
|
|
|
|
|
glue/communication.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
query/common.cpp
|
2018-08-24 16:12:04 +08:00
|
|
|
query/distributed_interpreter.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
query/frontend/ast/ast.cpp
|
|
|
|
|
query/frontend/ast/cypher_main_visitor.cpp
|
2018-08-16 16:13:04 +08:00
|
|
|
query/frontend/semantic/required_privileges.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
query/frontend/semantic/symbol_generator.cpp
|
|
|
|
|
query/frontend/stripped.cpp
|
|
|
|
|
query/interpret/awesome_memgraph_functions.cpp
|
|
|
|
|
query/interpreter.cpp
|
2018-01-25 21:09:06 +08:00
|
|
|
query/plan/distributed.cpp
|
2018-08-27 22:43:34 +08:00
|
|
|
query/plan/distributed_ops.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
query/plan/operator.cpp
|
|
|
|
|
query/plan/preprocess.cpp
|
2018-08-23 19:15:15 +08:00
|
|
|
query/plan/pretty_print.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
query/plan/rule_based_planner.cpp
|
|
|
|
|
query/plan/variable_start_planner.cpp
|
2018-07-02 21:34:33 +08:00
|
|
|
query/repl.cpp
|
Clean-up TypedValue misuse
Summary:
In a bunch of places `TypedValue` was used where `PropertyValue` should be. A lot of times it was only because `TypedValue` serialization code could be reused for `PropertyValue`, only without providing callbacks for `VERTEX`, `EDGE` and `PATH`. So first I wrote separate serialization code for `PropertyValue` and put it into storage folder. Then I fixed all the places where `TypedValue` was incorrectly used instead of `PropertyValue`. I also disabled implicit `TypedValue` to `PropertyValue` conversion in hopes of preventing misuse in the future.
After that, I wrote code for `VertexAccessor` and `EdgeAccessor` serialization and put it into `storage` folder because it was almost duplicated in distributed BFS and pull produce RPC messages. On the sender side, some subset of records (old or new or both) is serialized, and on the reciever side, records are deserialized and immediately put into transaction cache.
Then I rewrote the `TypedValue` serialization functions (`SaveCapnpTypedValue` and `LoadCapnpTypedValue`) to not take callbacks for `VERTEX`, `EDGE` and `PATH`, but use accessor serialization functions instead. That means that any code that wants to use `TypedValue` serialization must hold a reference to `GraphDbAccessor` and `DataManager`, so that should make clients reconsider if they really want to use `TypedValue` instead of `PropertyValue`.
Reviewers: teon.banek, msantl
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1598
2018-09-13 18:12:07 +08:00
|
|
|
query/serialization.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
query/typed_value.cpp
|
2017-12-15 17:48:21 +08:00
|
|
|
storage/concurrent_id_mapper_master.cpp
|
|
|
|
|
storage/concurrent_id_mapper_worker.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
storage/edge_accessor.cpp
|
|
|
|
|
storage/locking/record_lock.cpp
|
|
|
|
|
storage/property_value.cpp
|
2018-05-10 22:33:41 +08:00
|
|
|
storage/property_value_store.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
storage/record_accessor.cpp
|
Clean-up TypedValue misuse
Summary:
In a bunch of places `TypedValue` was used where `PropertyValue` should be. A lot of times it was only because `TypedValue` serialization code could be reused for `PropertyValue`, only without providing callbacks for `VERTEX`, `EDGE` and `PATH`. So first I wrote separate serialization code for `PropertyValue` and put it into storage folder. Then I fixed all the places where `TypedValue` was incorrectly used instead of `PropertyValue`. I also disabled implicit `TypedValue` to `PropertyValue` conversion in hopes of preventing misuse in the future.
After that, I wrote code for `VertexAccessor` and `EdgeAccessor` serialization and put it into `storage` folder because it was almost duplicated in distributed BFS and pull produce RPC messages. On the sender side, some subset of records (old or new or both) is serialized, and on the reciever side, records are deserialized and immediately put into transaction cache.
Then I rewrote the `TypedValue` serialization functions (`SaveCapnpTypedValue` and `LoadCapnpTypedValue`) to not take callbacks for `VERTEX`, `EDGE` and `PATH`, but use accessor serialization functions instead. That means that any code that wants to use `TypedValue` serialization must hold a reference to `GraphDbAccessor` and `DataManager`, so that should make clients reconsider if they really want to use `TypedValue` instead of `PropertyValue`.
Reviewers: teon.banek, msantl
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1598
2018-09-13 18:12:07 +08:00
|
|
|
storage/serialization.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
storage/vertex_accessor.cpp
|
2018-09-05 02:30:58 +08:00
|
|
|
transactions/distributed/engine_master.cpp
|
|
|
|
|
transactions/distributed/engine_worker.cpp
|
|
|
|
|
transactions/single_node/engine_single_node.cpp
|
2018-08-22 21:00:16 +08:00
|
|
|
memgraph_init.cpp
|
2017-12-04 20:56:17 +08:00
|
|
|
)
|
|
|
|
|
# -----------------------------------------------------------------------------
|
|
|
|
|
|
2018-08-30 16:40:59 +08:00
|
|
|
define_add_capnp(memgraph_src_files generated_capnp_files)
|
2018-05-16 19:48:56 +08:00
|
|
|
|
2018-08-30 16:40:59 +08:00
|
|
|
define_add_lcp(memgraph_src_files generated_lcp_files)
|
Add Lisp C++ Preprocessing (LCP)
Summary:
In order to enhance C++ metaprogramming capabilities, a custom
preprocessing step is added before compilation. C++ code may be mixed
with Lisp code in order to generate a complete C++ source code. The
mechanism is hooked into cmake. To notify cmake of .lcp files, `add_lcp`
function in src/CMakeLists.txt needs to be invoked.
The main executable entry point is in tools/lcp, while the source code
is in src/lisp/lcp.lisp
The main goal of LCP is to auto generate class serialization code and
member variable getter functions. This should now be significantly less
error prone, since you cannot forget to serialize a member variable
through this mechanism. Future uses should be generating other repeating
code, such as `Clone` methods or perhaps some debug information.
.lcp files may contain mixed C++ code (enclosed in #>cpp ... cpp<#
blocks) with Common Lisp code.
NOTE: With great power comes great responsibility. Lisp metaprogramming
capabilities are incredibly powerful. To keep the sanity of the team
intact, use Lisp preprocessing only when *really* necessary.
Reviewers: buda, mferencevic, msantl, dgleich, ipaljak, mculinovic, mtomic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1361
2018-04-27 21:48:30 +08:00
|
|
|
|
Replace boost with capnp in RPC
Summary:
Converts the RPC stack to use Cap'n Proto for serialization instead of
boost. There are still some traces of boost in other places in the code,
but most of it is removed. A future diff should cleanup boost for good.
The RPC API is now changed to be more flexible with regards to how
serialize data. This makes the simplest cases a bit more verbose, but
allows complex serialization code to be correctly written instead of
relying on hacks. (For reference, look for the old serialization of
`PullRpc` which had a nasty pointer hacks to inject accessors in
`TypedValue`.)
Since RPC messages were uselessly modeled via inheritance of Message
base class, that class is now removed. Furthermore, that approach
doesn't really work with Cap'n Proto. Instead, each message type is
required to have some type information. This can be automated, so
`define-rpc` has been added to LCP, which hopefully simplifies defining
new RPC request and response messages.
Specify Cap'n Proto schema ID in cmake
This preserves Cap'n Proto generated typeIds across multiple generations
of capnp schemas through LCP. It is imperative that typeId stays the
same to ensure that different compilations of Memgraph may communicate
via RPC in a distributed cluster.
Use CLOS for meta information on C++ types in LCP
Since some structure slots and functions have started to repeat
themselves, it makes sense to model C++ meta information via Common Lisp
Object System.
Depends on D1391
Reviewers: buda, dgleich, mferencevic, mtomic, mculinovic, msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1407
2018-06-04 15:48:48 +08:00
|
|
|
add_lcp(database/counters_rpc_messages.lcp CAPNP_SCHEMA @0x95a2c3ea3871e945)
|
|
|
|
|
add_capnp(database/counters_rpc_messages.capnp)
|
|
|
|
|
add_lcp(database/state_delta.lcp CAPNP_SCHEMA @0xdea01657b3563887)
|
|
|
|
|
add_capnp(database/state_delta.capnp)
|
|
|
|
|
add_lcp(distributed/bfs_rpc_messages.lcp CAPNP_SCHEMA @0x8e508640b09b6d2a)
|
|
|
|
|
add_capnp(distributed/bfs_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/coordination_rpc_messages.lcp CAPNP_SCHEMA @0x93df0c4703cf98fb)
|
|
|
|
|
add_capnp(distributed/coordination_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/data_rpc_messages.lcp CAPNP_SCHEMA @0xc1c8a341ba37aaf5)
|
|
|
|
|
add_capnp(distributed/data_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/durability_rpc_messages.lcp CAPNP_SCHEMA @0xf5e53bc271e2163d)
|
|
|
|
|
add_capnp(distributed/durability_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/index_rpc_messages.lcp CAPNP_SCHEMA @0xa8aab46862945bd6)
|
|
|
|
|
add_capnp(distributed/index_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/plan_rpc_messages.lcp CAPNP_SCHEMA @0xfcbc48dc9f106d28)
|
|
|
|
|
add_capnp(distributed/plan_rpc_messages.capnp)
|
2018-09-05 02:30:58 +08:00
|
|
|
add_lcp(distributed/pull_produce_rpc_messages.lcp CAPNP_SCHEMA @0xa78a9254a73685bd
|
|
|
|
|
DEPENDS transactions/distributed/serialization.lcp)
|
Replace boost with capnp in RPC
Summary:
Converts the RPC stack to use Cap'n Proto for serialization instead of
boost. There are still some traces of boost in other places in the code,
but most of it is removed. A future diff should cleanup boost for good.
The RPC API is now changed to be more flexible with regards to how
serialize data. This makes the simplest cases a bit more verbose, but
allows complex serialization code to be correctly written instead of
relying on hacks. (For reference, look for the old serialization of
`PullRpc` which had a nasty pointer hacks to inject accessors in
`TypedValue`.)
Since RPC messages were uselessly modeled via inheritance of Message
base class, that class is now removed. Furthermore, that approach
doesn't really work with Cap'n Proto. Instead, each message type is
required to have some type information. This can be automated, so
`define-rpc` has been added to LCP, which hopefully simplifies defining
new RPC request and response messages.
Specify Cap'n Proto schema ID in cmake
This preserves Cap'n Proto generated typeIds across multiple generations
of capnp schemas through LCP. It is imperative that typeId stays the
same to ensure that different compilations of Memgraph may communicate
via RPC in a distributed cluster.
Use CLOS for meta information on C++ types in LCP
Since some structure slots and functions have started to repeat
themselves, it makes sense to model C++ meta information via Common Lisp
Object System.
Depends on D1391
Reviewers: buda, dgleich, mferencevic, mtomic, mculinovic, msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1407
2018-06-04 15:48:48 +08:00
|
|
|
add_capnp(distributed/pull_produce_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/storage_gc_rpc_messages.lcp CAPNP_SCHEMA @0xd705663dfe36cf81)
|
|
|
|
|
add_capnp(distributed/storage_gc_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/token_sharing_rpc_messages.lcp CAPNP_SCHEMA @0x8f295db54ec4caec)
|
|
|
|
|
add_capnp(distributed/token_sharing_rpc_messages.capnp)
|
|
|
|
|
add_lcp(distributed/updates_rpc_messages.lcp CAPNP_SCHEMA @0x82d5f38d73c7b53a)
|
|
|
|
|
add_capnp(distributed/updates_rpc_messages.capnp)
|
2018-08-27 22:43:34 +08:00
|
|
|
|
|
|
|
|
# distributed_ops.lcp is leading the capnp code generation, so we only want
|
|
|
|
|
# function declarations in generated operator.hpp
|
|
|
|
|
add_lcp(query/plan/operator.lcp CAPNP_DECLARATION)
|
|
|
|
|
add_lcp(query/plan/distributed_ops.lcp CAPNP_SCHEMA @0xe5cae8d045d30c42
|
|
|
|
|
DEPENDS query/plan/operator.lcp)
|
|
|
|
|
add_capnp(query/plan/distributed_ops.capnp)
|
|
|
|
|
|
Replace boost with capnp in RPC
Summary:
Converts the RPC stack to use Cap'n Proto for serialization instead of
boost. There are still some traces of boost in other places in the code,
but most of it is removed. A future diff should cleanup boost for good.
The RPC API is now changed to be more flexible with regards to how
serialize data. This makes the simplest cases a bit more verbose, but
allows complex serialization code to be correctly written instead of
relying on hacks. (For reference, look for the old serialization of
`PullRpc` which had a nasty pointer hacks to inject accessors in
`TypedValue`.)
Since RPC messages were uselessly modeled via inheritance of Message
base class, that class is now removed. Furthermore, that approach
doesn't really work with Cap'n Proto. Instead, each message type is
required to have some type information. This can be automated, so
`define-rpc` has been added to LCP, which hopefully simplifies defining
new RPC request and response messages.
Specify Cap'n Proto schema ID in cmake
This preserves Cap'n Proto generated typeIds across multiple generations
of capnp schemas through LCP. It is imperative that typeId stays the
same to ensure that different compilations of Memgraph may communicate
via RPC in a distributed cluster.
Use CLOS for meta information on C++ types in LCP
Since some structure slots and functions have started to repeat
themselves, it makes sense to model C++ meta information via Common Lisp
Object System.
Depends on D1391
Reviewers: buda, dgleich, mferencevic, mtomic, mculinovic, msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1407
2018-06-04 15:48:48 +08:00
|
|
|
add_lcp(storage/concurrent_id_mapper_rpc_messages.lcp CAPNP_SCHEMA @0xa6068dae93d225dd)
|
|
|
|
|
add_capnp(storage/concurrent_id_mapper_rpc_messages.capnp)
|
2018-09-05 02:30:58 +08:00
|
|
|
add_lcp(transactions/distributed/engine_rpc_messages.lcp CAPNP_SCHEMA @0xde02b7c49180cad5
|
|
|
|
|
DEPENDS transactions/distributed/serialization.lcp)
|
|
|
|
|
add_capnp(transactions/distributed/engine_rpc_messages.capnp)
|
2018-04-23 22:23:42 +08:00
|
|
|
|
Add Lisp C++ Preprocessing (LCP)
Summary:
In order to enhance C++ metaprogramming capabilities, a custom
preprocessing step is added before compilation. C++ code may be mixed
with Lisp code in order to generate a complete C++ source code. The
mechanism is hooked into cmake. To notify cmake of .lcp files, `add_lcp`
function in src/CMakeLists.txt needs to be invoked.
The main executable entry point is in tools/lcp, while the source code
is in src/lisp/lcp.lisp
The main goal of LCP is to auto generate class serialization code and
member variable getter functions. This should now be significantly less
error prone, since you cannot forget to serialize a member variable
through this mechanism. Future uses should be generating other repeating
code, such as `Clone` methods or perhaps some debug information.
.lcp files may contain mixed C++ code (enclosed in #>cpp ... cpp<#
blocks) with Common Lisp code.
NOTE: With great power comes great responsibility. Lisp metaprogramming
capabilities are incredibly powerful. To keep the sanity of the team
intact, use Lisp preprocessing only when *really* necessary.
Reviewers: buda, mferencevic, msantl, dgleich, ipaljak, mculinovic, mtomic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1361
2018-04-27 21:48:30 +08:00
|
|
|
add_custom_target(generate_lcp DEPENDS ${generated_lcp_files})
|
2018-05-02 15:54:28 +08:00
|
|
|
|
2018-05-16 19:48:56 +08:00
|
|
|
# Registering capnp must come after registering lcp files.
|
2018-05-02 15:54:28 +08:00
|
|
|
|
Replace boost with capnp in RPC
Summary:
Converts the RPC stack to use Cap'n Proto for serialization instead of
boost. There are still some traces of boost in other places in the code,
but most of it is removed. A future diff should cleanup boost for good.
The RPC API is now changed to be more flexible with regards to how
serialize data. This makes the simplest cases a bit more verbose, but
allows complex serialization code to be correctly written instead of
relying on hacks. (For reference, look for the old serialization of
`PullRpc` which had a nasty pointer hacks to inject accessors in
`TypedValue`.)
Since RPC messages were uselessly modeled via inheritance of Message
base class, that class is now removed. Furthermore, that approach
doesn't really work with Cap'n Proto. Instead, each message type is
required to have some type information. This can be automated, so
`define-rpc` has been added to LCP, which hopefully simplifies defining
new RPC request and response messages.
Specify Cap'n Proto schema ID in cmake
This preserves Cap'n Proto generated typeIds across multiple generations
of capnp schemas through LCP. It is imperative that typeId stays the
same to ensure that different compilations of Memgraph may communicate
via RPC in a distributed cluster.
Use CLOS for meta information on C++ types in LCP
Since some structure slots and functions have started to repeat
themselves, it makes sense to model C++ meta information via Common Lisp
Object System.
Depends on D1391
Reviewers: buda, dgleich, mferencevic, mtomic, mculinovic, msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1407
2018-06-04 15:48:48 +08:00
|
|
|
add_capnp(communication/rpc/messages.capnp)
|
|
|
|
|
add_capnp(durability/recovery.capnp)
|
|
|
|
|
add_capnp(query/common.capnp)
|
2018-04-03 16:27:50 +08:00
|
|
|
add_capnp(query/frontend/ast/ast.capnp)
|
Replace boost with capnp in RPC
Summary:
Converts the RPC stack to use Cap'n Proto for serialization instead of
boost. There are still some traces of boost in other places in the code,
but most of it is removed. A future diff should cleanup boost for good.
The RPC API is now changed to be more flexible with regards to how
serialize data. This makes the simplest cases a bit more verbose, but
allows complex serialization code to be correctly written instead of
relying on hacks. (For reference, look for the old serialization of
`PullRpc` which had a nasty pointer hacks to inject accessors in
`TypedValue`.)
Since RPC messages were uselessly modeled via inheritance of Message
base class, that class is now removed. Furthermore, that approach
doesn't really work with Cap'n Proto. Instead, each message type is
required to have some type information. This can be automated, so
`define-rpc` has been added to LCP, which hopefully simplifies defining
new RPC request and response messages.
Specify Cap'n Proto schema ID in cmake
This preserves Cap'n Proto generated typeIds across multiple generations
of capnp schemas through LCP. It is imperative that typeId stays the
same to ensure that different compilations of Memgraph may communicate
via RPC in a distributed cluster.
Use CLOS for meta information on C++ types in LCP
Since some structure slots and functions have started to repeat
themselves, it makes sense to model C++ meta information via Common Lisp
Object System.
Depends on D1391
Reviewers: buda, dgleich, mferencevic, mtomic, mculinovic, msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1407
2018-06-04 15:48:48 +08:00
|
|
|
add_capnp(query/frontend/semantic/symbol.capnp)
|
Clean-up TypedValue misuse
Summary:
In a bunch of places `TypedValue` was used where `PropertyValue` should be. A lot of times it was only because `TypedValue` serialization code could be reused for `PropertyValue`, only without providing callbacks for `VERTEX`, `EDGE` and `PATH`. So first I wrote separate serialization code for `PropertyValue` and put it into storage folder. Then I fixed all the places where `TypedValue` was incorrectly used instead of `PropertyValue`. I also disabled implicit `TypedValue` to `PropertyValue` conversion in hopes of preventing misuse in the future.
After that, I wrote code for `VertexAccessor` and `EdgeAccessor` serialization and put it into `storage` folder because it was almost duplicated in distributed BFS and pull produce RPC messages. On the sender side, some subset of records (old or new or both) is serialized, and on the reciever side, records are deserialized and immediately put into transaction cache.
Then I rewrote the `TypedValue` serialization functions (`SaveCapnpTypedValue` and `LoadCapnpTypedValue`) to not take callbacks for `VERTEX`, `EDGE` and `PATH`, but use accessor serialization functions instead. That means that any code that wants to use `TypedValue` serialization must hold a reference to `GraphDbAccessor` and `DataManager`, so that should make clients reconsider if they really want to use `TypedValue` instead of `PropertyValue`.
Reviewers: teon.banek, msantl
Reviewed By: teon.banek
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1598
2018-09-13 18:12:07 +08:00
|
|
|
add_capnp(query/serialization.capnp)
|
Replace boost with capnp in RPC
Summary:
Converts the RPC stack to use Cap'n Proto for serialization instead of
boost. There are still some traces of boost in other places in the code,
but most of it is removed. A future diff should cleanup boost for good.
The RPC API is now changed to be more flexible with regards to how
serialize data. This makes the simplest cases a bit more verbose, but
allows complex serialization code to be correctly written instead of
relying on hacks. (For reference, look for the old serialization of
`PullRpc` which had a nasty pointer hacks to inject accessors in
`TypedValue`.)
Since RPC messages were uselessly modeled via inheritance of Message
base class, that class is now removed. Furthermore, that approach
doesn't really work with Cap'n Proto. Instead, each message type is
required to have some type information. This can be automated, so
`define-rpc` has been added to LCP, which hopefully simplifies defining
new RPC request and response messages.
Specify Cap'n Proto schema ID in cmake
This preserves Cap'n Proto generated typeIds across multiple generations
of capnp schemas through LCP. It is imperative that typeId stays the
same to ensure that different compilations of Memgraph may communicate
via RPC in a distributed cluster.
Use CLOS for meta information on C++ types in LCP
Since some structure slots and functions have started to repeat
themselves, it makes sense to model C++ meta information via Common Lisp
Object System.
Depends on D1391
Reviewers: buda, dgleich, mferencevic, mtomic, mculinovic, msantl
Reviewed By: msantl
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1407
2018-06-04 15:48:48 +08:00
|
|
|
add_capnp(storage/serialization.capnp)
|
2018-05-16 19:48:56 +08:00
|
|
|
|
|
|
|
|
add_custom_target(generate_capnp DEPENDS generate_lcp ${generated_capnp_files})
|
2018-05-02 15:54:28 +08:00
|
|
|
|
Add Lisp C++ Preprocessing (LCP)
Summary:
In order to enhance C++ metaprogramming capabilities, a custom
preprocessing step is added before compilation. C++ code may be mixed
with Lisp code in order to generate a complete C++ source code. The
mechanism is hooked into cmake. To notify cmake of .lcp files, `add_lcp`
function in src/CMakeLists.txt needs to be invoked.
The main executable entry point is in tools/lcp, while the source code
is in src/lisp/lcp.lisp
The main goal of LCP is to auto generate class serialization code and
member variable getter functions. This should now be significantly less
error prone, since you cannot forget to serialize a member variable
through this mechanism. Future uses should be generating other repeating
code, such as `Clone` methods or perhaps some debug information.
.lcp files may contain mixed C++ code (enclosed in #>cpp ... cpp<#
blocks) with Common Lisp code.
NOTE: With great power comes great responsibility. Lisp metaprogramming
capabilities are incredibly powerful. To keep the sanity of the team
intact, use Lisp preprocessing only when *really* necessary.
Reviewers: buda, mferencevic, msantl, dgleich, ipaljak, mculinovic, mtomic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1361
2018-04-27 21:48:30 +08:00
|
|
|
# -----------------------------------------------------------------------------
|
|
|
|
|
|
2018-01-15 19:19:55 +08:00
|
|
|
string(TOLOWER ${CMAKE_BUILD_TYPE} lower_build_type)
|
|
|
|
|
|
2017-12-04 20:56:17 +08:00
|
|
|
# memgraph_lib depend on these libraries
|
2017-12-22 21:36:25 +08:00
|
|
|
set(MEMGRAPH_ALL_LIBS stdc++fs Threads::Threads fmt cppitertools
|
2018-05-02 15:54:28 +08:00
|
|
|
antlr_opencypher_parser_lib dl glog gflags capnp kj
|
2018-09-02 03:09:31 +08:00
|
|
|
mg-utils mg-io mg-integrations-kafka mg-requests mg-communication mg-auth mg-stats)
|
2017-12-04 20:56:17 +08:00
|
|
|
|
|
|
|
|
if (USE_LTALLOC)
|
|
|
|
|
list(APPEND MEMGRAPH_ALL_LIBS ltalloc)
|
|
|
|
|
# TODO(mferencevic): Enable this when clang is updated on apollo.
|
|
|
|
|
# set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -flto")
|
|
|
|
|
endif()
|
|
|
|
|
|
|
|
|
|
if (READLINE_FOUND)
|
|
|
|
|
list(APPEND MEMGRAPH_ALL_LIBS readline)
|
|
|
|
|
endif()
|
|
|
|
|
|
|
|
|
|
# STATIC library used by memgraph executables
|
|
|
|
|
add_library(memgraph_lib STATIC ${memgraph_src_files})
|
2018-05-30 19:00:25 +08:00
|
|
|
target_link_libraries(memgraph_lib ${MEMGRAPH_ALL_LIBS})
|
2017-12-04 20:56:17 +08:00
|
|
|
add_dependencies(memgraph_lib generate_opencypher_parser)
|
Add Lisp C++ Preprocessing (LCP)
Summary:
In order to enhance C++ metaprogramming capabilities, a custom
preprocessing step is added before compilation. C++ code may be mixed
with Lisp code in order to generate a complete C++ source code. The
mechanism is hooked into cmake. To notify cmake of .lcp files, `add_lcp`
function in src/CMakeLists.txt needs to be invoked.
The main executable entry point is in tools/lcp, while the source code
is in src/lisp/lcp.lisp
The main goal of LCP is to auto generate class serialization code and
member variable getter functions. This should now be significantly less
error prone, since you cannot forget to serialize a member variable
through this mechanism. Future uses should be generating other repeating
code, such as `Clone` methods or perhaps some debug information.
.lcp files may contain mixed C++ code (enclosed in #>cpp ... cpp<#
blocks) with Common Lisp code.
NOTE: With great power comes great responsibility. Lisp metaprogramming
capabilities are incredibly powerful. To keep the sanity of the team
intact, use Lisp preprocessing only when *really* necessary.
Reviewers: buda, mferencevic, msantl, dgleich, ipaljak, mculinovic, mtomic
Reviewed By: mtomic
Subscribers: pullbot
Differential Revision: https://phabricator.memgraph.io/D1361
2018-04-27 21:48:30 +08:00
|
|
|
add_dependencies(memgraph_lib generate_lcp)
|
2018-05-02 15:54:28 +08:00
|
|
|
add_dependencies(memgraph_lib generate_capnp)
|
2017-12-04 20:56:17 +08:00
|
|
|
|
2018-04-27 17:23:40 +08:00
|
|
|
# STATIC library used to store key-value pairs
|
2018-05-29 17:13:13 +08:00
|
|
|
add_library(kvstore_lib STATIC storage/kvstore.cpp)
|
2018-06-19 20:37:02 +08:00
|
|
|
target_link_libraries(kvstore_lib stdc++fs mg-utils rocksdb bzip2 zlib glog gflags)
|
2018-04-27 17:23:40 +08:00
|
|
|
|
2018-06-12 17:29:22 +08:00
|
|
|
# STATIC library for dummy key-value storage
|
|
|
|
|
add_library(kvstore_dummy_lib STATIC storage/kvstore_dummy.cpp)
|
|
|
|
|
target_link_libraries(kvstore_dummy_lib mg-utils)
|
|
|
|
|
|
2017-12-04 20:56:17 +08:00
|
|
|
# Generate a version.hpp file
|
|
|
|
|
set(VERSION_STRING ${memgraph_VERSION})
|
|
|
|
|
configure_file(version.hpp.in version.hpp @ONLY)
|
|
|
|
|
include_directories(${CMAKE_CURRENT_BINARY_DIR})
|
|
|
|
|
|
|
|
|
|
# memgraph main executable
|
2017-12-11 22:51:53 +08:00
|
|
|
add_executable(memgraph memgraph_bolt.cpp)
|
2018-06-20 19:46:54 +08:00
|
|
|
target_link_libraries(memgraph memgraph_lib kvstore_lib telemetry_lib)
|
2017-12-11 22:51:53 +08:00
|
|
|
set_target_properties(memgraph PROPERTIES
|
|
|
|
|
# Set the executable output name to include version information.
|
|
|
|
|
OUTPUT_NAME "memgraph-${memgraph_VERSION}-${COMMIT_HASH}_${CMAKE_BUILD_TYPE}"
|
|
|
|
|
# Output the executable in main binary dir.
|
2017-12-04 20:56:17 +08:00
|
|
|
RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
|
2017-12-11 22:51:53 +08:00
|
|
|
# Create symlink to the built executable.
|
|
|
|
|
add_custom_command(TARGET memgraph POST_BUILD
|
|
|
|
|
COMMAND ${CMAKE_COMMAND} -E create_symlink $<TARGET_FILE:memgraph> ${CMAKE_BINARY_DIR}/memgraph
|
|
|
|
|
BYPRODUCTS ${CMAKE_BINARY_DIR}/memgraph
|
|
|
|
|
COMMENT Creating symlink to memgraph executable)
|
|
|
|
|
|
2017-12-04 20:56:17 +08:00
|
|
|
# Strip the executable in release build.
|
|
|
|
|
if (lower_build_type STREQUAL "release")
|
2017-12-11 22:51:53 +08:00
|
|
|
add_custom_command(TARGET memgraph POST_BUILD
|
|
|
|
|
COMMAND strip -s $<TARGET_FILE:memgraph>
|
|
|
|
|
COMMENT Stripping symbols and sections from memgraph)
|
2017-12-04 20:56:17 +08:00
|
|
|
endif()
|
|
|
|
|
|
|
|
|
|
# Everything here is under "memgraph" install component.
|
|
|
|
|
set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME "memgraph")
|
|
|
|
|
|
2018-04-10 16:40:03 +08:00
|
|
|
# TODO: Default directory permissions to 755
|
|
|
|
|
# NOTE: This is added in CMake 3.11, so enable it then
|
|
|
|
|
#set(CMAKE_INSTALL_DEFAULT_DIRECTORY_PERMISSIONS
|
|
|
|
|
# OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ WORLD_READ)
|
|
|
|
|
|
2017-12-04 20:56:17 +08:00
|
|
|
# Install and rename executable to just 'memgraph' Since we have to rename,
|
|
|
|
|
# we cannot use the recommended `install(TARGETS ...)`.
|
2017-12-11 22:51:53 +08:00
|
|
|
install(PROGRAMS $<TARGET_FILE:memgraph>
|
2017-12-21 23:03:54 +08:00
|
|
|
DESTINATION lib/memgraph RENAME memgraph)
|
2017-12-04 20:56:17 +08:00
|
|
|
# Install the config file (must use absolute path).
|
|
|
|
|
install(FILES ${CMAKE_SOURCE_DIR}/config/community.conf
|
|
|
|
|
DESTINATION /etc/memgraph RENAME memgraph.conf)
|
|
|
|
|
# Install logrotate configuration (must use absolute path).
|
|
|
|
|
install(FILES ${CMAKE_SOURCE_DIR}/release/logrotate.conf
|
|
|
|
|
DESTINATION /etc/logrotate.d RENAME memgraph)
|
|
|
|
|
# Create empty directories for default location of lib and log.
|
|
|
|
|
install(CODE "file(MAKE_DIRECTORY \$ENV{DESTDIR}/var/log/memgraph
|
|
|
|
|
\$ENV{DESTDIR}/var/lib/memgraph)")
|
|
|
|
|
# Install the license file.
|
|
|
|
|
install(FILES ${CMAKE_SOURCE_DIR}/release/LICENSE.md
|
|
|
|
|
DESTINATION share/doc/memgraph RENAME copyright)
|
|
|
|
|
# Install systemd service (must use absolute path).
|
|
|
|
|
install(FILES ${CMAKE_SOURCE_DIR}/release/memgraph.service
|
|
|
|
|
DESTINATION /lib/systemd/system)
|
|
|
|
|
|
2017-12-12 20:34:27 +08:00
|
|
|
# Install examples
|
|
|
|
|
set(examples ${CMAKE_SOURCE_DIR}/release/examples)
|
2018-04-20 20:58:49 +08:00
|
|
|
install(
|
|
|
|
|
CODE
|
|
|
|
|
"execute_process(COMMAND ${examples}/build_examples
|
2018-04-24 22:06:45 +08:00
|
|
|
${CMAKE_BINARY_DIR}/memgraph
|
2018-04-20 20:58:49 +08:00
|
|
|
${CMAKE_BINARY_DIR}/tests/manual/bolt_client
|
|
|
|
|
WORKING_DIRECTORY ${examples})")
|
2017-12-12 20:34:27 +08:00
|
|
|
install(DIRECTORY ${examples}/build/ DESTINATION share/memgraph/examples)
|
