tursom/memgraph
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Remove customers/experimental/poc

Browse Source 
Reviewers: teon.banek, buda

Reviewed By: teon.banek, buda

Subscribers: pullbot

Differential Revision: https://phabricator.memgraph.io/D2648
This commit is contained in:
Matej Ferencevic 2020-01-28 13:42:50 +01:00
parent 84a6ab75cb
commit d156d5b41c
94 changed files with 0 additions and 6972 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
CMakeLists.txt
View File

@ -193,9 +193,6 @@ target_link_libraries(antlr_opencypher_parser_lib antlr4)
# -----------------------------------------------------------------------------
# Optional subproject configuration -------------------------------------------
option(POC "Build proof of concept binaries" OFF)
option(EXPERIMENTAL "Build experimental binaries" OFF)
option(CUSTOMERS "Build customer binaries" OFF)
option(TEST_COVERAGE "Generate coverage reports from running memgraph" OFF)
option(TOOLS "Build tools binaries" ON)
option(QUERY_MODULES "Build query modules containing custom procedures" ON)
@ -278,18 +275,6 @@ endif()
include_directories(src)
add_subdirectory(src)
if(POC)
add_subdirectory(poc)
endif()
if(EXPERIMENTAL)
add_subdirectory(experimental)
endif()
if(CUSTOMERS)
add_subdirectory(customers)
endif()
enable_testing()
add_subdirectory(tests)

1
Doxyfile
View File

@ -839,7 +839,6 @@ EXCLUDE_PATTERNS += */Testing/*
EXCLUDE_PATTERNS += */tests/*
EXCLUDE_PATTERNS += */dist/*
EXCLUDE_PATTERNS += */tools/*
EXCLUDE_PATTERNS += */customers/*
# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names
# (namespaces, classes, functions, etc.) that should be excluded from the

2
customers/CMakeLists.txt
View File

@ -1,2 +0,0 @@
project(mg_customers)
add_subdirectory(otto)

70
customers/elliott/README_internal.md
View File

@ -1,70 +0,0 @@
DISCLAIMER: this is just an initial test, graph might not resemble
the graph in the use case at all and the data might be completely
irrelevant.
We tried generating a few sample graphs from the vague description
given in the use case doc. Then we tried writing queries that would
solve the problem of updating nodes when a leaf value changes,
assuming all the internal nodes compute only the sum function.
We start by creating an index on `id` property to improve initial lookup
performance:
CREATE INDEX ON :Leaf(id)
Set values of all leafs to 1:
MATCH (u:Leaf) SET u.value = 1
Now we initialize the values of all other nodes in the graph:
MATCH (u) WHERE NOT u:Leaf SET u.value = 0
MATCH (u) WITH u
ORDER BY u.topological_index DESC
MATCH (u)-->(v) SET u.value = u.value + v.value
Change the value of a leaf:
MATCH (u:Leaf {id: "18"}) SET u.value = 10
We have to reset all the updated nodes to a neutral element:
MATCH (u:Leaf {id: "18"})<-[* bfs]-(v)
WHERE NOT v:Leaf SET v.value = 0
Finally, we recalculate their values in topological order:
MATCH (u:Leaf {id: "18"})<-[* bfs]-(v)
WITH v ORDER BY v.topological_index DESC
MATCH (v)-->(w) SET v.value = v.value + w.value
There are a few assumptions made worth pointing out.
* We are able to efficiently maintain topological order
of vertices in the graph.
* It is possible to accumulate the value of the function. Formally:
$$f(x_1, x_2, ..., x_n) = g(...(g(g(x_1, x_2), x_3), ...), x_n)$$
* There is a neutral element for the operation. However, this
assumption can be dropped by introducing an artificial neutral element.
Number of operations required is proportional to the sum of degrees of affected
nodes.
We generated graph with $10^5$ nodes ($20\ 000$ nodes in each layer), varied the
degree distribution in node layers and measured time for the query to execute:
| # | Root-Category-Group degree | Group-CustomGroup-Leaf degree | Time |
|:-:|:---------------------------:|:-----------------------------:|:---------:|
| 1 | [1, 10] | [20, 40] | ~1.1s |
| 2 | [1, 10] | [50, 100] | ~2.5s |
| 3 | [10, 50] | [50, 100] | ~3.3s |
Due to the structure of the graph, update of a leaf required update of almost
all the nodes in the graph so we don't show times required for initial graph
update and update after leaf change separately.
However, there is not enough info on the use case to make the test more
sophisticated.

71
customers/elliott/README_public.md
View File

@ -1,71 +0,0 @@
---
title: "Elliott Management"
subtitle: "Proof of Concept Report"
header-title: "Elliott Management POC"
date: 2017-10-28
copyright: "©2017 Memgraph Ltd. All rights reserved."
titlepage: true
titlepage-color: FFFFFF
titlepage-text-color: 101010
titlepage-rule-color: 101010
titlepage-rule-height: 1
...
# Introduction
We tried generating a few sample graphs from the description given at
the in-person meetings. Then, we tried writing queries that would solve
the problem of updating nodes when a leaf value changes, assuming all the
internal nodes compute only the sum function.
# Technical details
We started by creating an index on `id` property to improve initial lookup
performance:
CREATE INDEX ON :Leaf(id)
Afther that, we set values of all leafs to 1:
MATCH (u:Leaf) SET u.value = 1
We then initialized the values of all other nodes in the graph:
MATCH (u) WHERE NOT u:Leaf SET u.value = 0
MATCH (u) WITH u
ORDER BY u.topological_index DESC
MATCH (u)-->(v) SET u.value = u.value + v.value
Leaf value change and update of affected values in the graph can
be done using three queries. To change the value of a leaf:
MATCH (u:Leaf {id: "18"}) SET u.value = 10
Then we had to reset all the affected nodes to the neutral element:
MATCH (u:Leaf {id: "18"})<-[* bfs]-(v)
WHERE NOT v:Leaf SET v.value = 0
Finally, we recalculated their values in topological order:
MATCH (u:Leaf {id: "18"})<-[* bfs]-(v)
WITH v ORDER BY v.topological_index DESC
MATCH (v)-->(w) SET v.value = v.value + w.value
There are a few assumptions necessary for the approach above to work.
* We are able to maintain topological order of vertices during graph
structure changes.
* It is possible to accumulate the value of the function. Formally:
$$f(x_1, x_2, ..., x_n) = g(...(g(g(x_1, x_2), x_3), ...), x_n)$$
* There is a neutral element for the operation. However, this
assumption can be dropped by introducing an artificial neutral element.
Above assumptions could be changed, relaxed or dropped, depending on the
specifics of the use case.
Number of operations required is proportional to the sum of degrees of affected
nodes.

12
customers/elliott/commands.txt
View File

@ -1,12 +0,0 @@
CREATE INDEX ON :Leaf(id);
MATCH (u:Leaf) SET u.value = 1;
MATCH (u) WHERE NOT u:Leaf SET u.value = 0;
MATCH (u) WITH u
ORDER BY u.topological_index DESC
MATCH (u)-->(v) SET u.value = u.value + v.value;
MATCH (u:Leaf {id: "85000"}) SET u.value = 10;
MATCH (u:Leaf {id: "85000"})<-[* bfs]-(v)
WHERE NOT v:Leaf SET v.value = 0;
MATCH (u:Leaf {id: "85000"})<-[* bfs]-(v)
WITH v ORDER BY v.topological_index DESC
MATCH (v)-->(w) SET v.value = v.value + w.value;

125
customers/elliott/generate_dag
View File

@ -1,125 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Generates a DAG from JSON spec in [config] and outputs nodes to
[filename]_nodes, and edges to [filename]_edges in format convertible
to Memgraph snapshot.
Here's an example JSON spec:
{
"layers": [
{
"name": "A",
"sublayers": 1,
"degree_lo": 1,
"degree_hi": 3,
"nodes": 4
},
{
"name": "B",
"sublayers": 3,
"degree_lo": 2,
"degree_hi": 3,
"nodes": 10
},
{
"name": "C",
"sublayers": 1,
"degree_lo": 1,
"degree_hi": 1,
"nodes": 5
}
]
}
Nodes from each layer will be randomly divided into sublayers. A node can
only have edges pointing to nodes in lower sublayers of the same layer, or
to nodes from the layer directly below it. Out-degree is chosen uniformly
random from [degree_lo, degree_hi] interval."""
import argparse
from itertools import accumulate
import json
import random
def _split_into_sum(n, k):
assert 1 <= n, "n should be at least 1"
assert k <= n, "k shouldn't be greater than n"
xs = [0] + sorted(random.sample(range(1, n), k-1)) + [n]
return [b - a for a, b in zip(xs, xs[1:])]
def generate_dag(graph_config, seed=None):
random.seed(seed)
nodes = []
edges = []
layer_lo = 1
for layer in graph_config:
sublayers = _split_into_sum(layer['nodes'], layer['sublayers'])
sub_range = accumulate([layer_lo] + sublayers)
layer['sublayer_range'] = list(sub_range)
nodes.extend([
(u, layer['name'])
for u in range(layer_lo, layer_lo + layer['nodes'])
])
layer_lo += layer['nodes']
edges = []
for layer, next_layer in zip(graph_config, graph_config[1:]):
degree_lo = layer['degree_lo']
degree_hi = layer['degree_hi']
sub_range = layer['sublayer_range']
sub_range_next = next_layer['sublayer_range']
layer_lo = sub_range[0]
next_layer_hi = sub_range_next[-1]
for sub_lo, sub_hi in zip(sub_range, sub_range[1:]):
for u in range(sub_lo, sub_hi):
num_edges = random.randint(degree_lo, degree_hi)
for _ in range(num_edges):
v = random.randint(sub_hi, next_layer_hi - 1)
edges.append((u, v))
for sub_lo, sub_hi in zip(sub_range_next, sub_range_next[1:]):
for u in range(sub_lo, sub_hi):
v = random.randint(layer_lo, sub_lo - 1)
edges.append((v, u))
return nodes, edges
if __name__ == '__main__':
parser = argparse.ArgumentParser(
formatter_class=argparse.RawDescriptionHelpFormatter,
description=__doc__)
parser.add_argument('config', type=str, help='graph config JSON file')
parser.add_argument('filename', type=str,
help='nodes will be stored to filename_nodes, '
'edges to filename_edges')
parser.add_argument('--seed', type=int,
help='seed for the random generator (default = '
'current system time)')
args = parser.parse_args()
with open(args.config, 'r') as f:
graph_config = json.loads(f.read())['layers']
nodes, edges = generate_dag(graph_config, seed=args.seed)
# print nodes into CSV file
with open('{}_nodes'.format(args.filename), 'w') as out:
out.write('nodeId:ID(Node),name,topological_index:Int,:LABEL\n')
for node_id, layer in nodes:
out.write('{0},{1}{0},{0},{1}\n'.format(node_id, layer))
# print edges into CSV file
with open('{}_edges'.format(args.filename), 'w') as out:
out.write(':START_ID(Node),:END_ID(Node),:TYPE\n')
for u, v in edges:
out.write('{},{},child\n'.format(u, v))

39
customers/elliott/graph1.json
View File

@ -1,39 +0,0 @@
{
"layers": [
{
"name": "Root",
"sublayers": 1,
"degree_lo": 1,
"degree_hi": 10,
"nodes": 20000
},
{
"name": "Category",
"sublayers": 5,
"degree_lo": 1,
"degree_hi": 10,
"nodes": 20000
},
{
"name": "Group",
"sublayers": 1,
"degree_lo": 20,
"degree_hi": 40,
"nodes": 20000
},
{
"name": "CustomGroup",
"sublayers": 15,
"degree_lo": 20,
"degree_hi": 40,
"nodes": 20000
},
{
"name": "Leaf",
"sublayers": 1,
"degree_lo": 1,
"degree_hi": 1,
"nodes": 20000
}
]
}

39
customers/elliott/graph2.json
View File

@ -1,39 +0,0 @@
{
"layers": [
{
"name": "Root",
"sublayers": 1,
"degree_lo": 1,
"degree_hi": 10,
"nodes": 20000
},
{
"name": "Category",
"sublayers": 5,
"degree_lo": 1,
"degree_hi": 10,
"nodes": 20000
},
{
"name": "Group",
"sublayers": 1,
"degree_lo": 50,
"degree_hi": 100,
"nodes": 20000
},
{
"name": "CustomGroup",
"sublayers": 15,
"degree_lo": 50,
"degree_hi": 100,
"nodes": 20000
},
{
"name": "Leaf",
"sublayers": 1,
"degree_lo": 50,
"degree_hi": 100,
"nodes": 20000
}
]
}

39
customers/elliott/graph3.json
View File

@ -1,39 +0,0 @@
{
"layers": [
{
"name": "Root",
"sublayers": 1,
"degree_lo": 10,
"degree_hi": 50,
"nodes": 20000
},
{
"name": "Category",
"sublayers": 5,
"degree_lo": 10,
"degree_hi": 50,
"nodes": 20000
},
{
"name": "Group",
"sublayers": 1,
"degree_lo": 50,
"degree_hi": 100,
"nodes": 20000
},
{
"name": "CustomGroup",
"sublayers": 15,
"degree_lo": 50,
"degree_hi": 100,
"nodes": 20000
},
{
"name": "Leaf",
"sublayers": 1,
"degree_lo": 50,
"degree_hi": 100,
"nodes": 20000
}
]
}

3
customers/otto/CMakeLists.txt
View File

@ -1,3 +0,0 @@
set(exec_name customers_otto_parallel_connected_components)
add_executable(${exec_name} parallel_connected_components.cpp)
target_link_libraries(${exec_name} memgraph_lib)

118
customers/otto/connected_components_test.py
View File

@ -1,118 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
This script attempts to evaluate the feasibility of using Memgraph for
Otto group's usecase. The usecase is finding connected componentes in
a large, very sparse graph (cca 220M nodes, 250M edges), based on a dynamic
inclusion / exclusion of edges (w.r.t variable parameters and the source node
type).
This implementation defines a random graph with the given number of nodes
and edges and looks for connected components using breadth-first expansion.
Edges are included / excluded based on a simple expression, only demonstrating
possible usage.
"""
from argparse import ArgumentParser
import logging
from time import time
from collections import defaultdict
from math import log2
from random import randint
from neo4j.v1 import GraphDatabase
log = logging.getLogger(__name__)
def generate_graph(sess, node_count, edge_count):
# An index that will speed-up edge creation.
sess.run("CREATE INDEX ON :Node(id)").consume()
# Create the given number of nodes with a randomly selected type from:
# [0.5, 1.5, 2.5].
sess.run(("UNWIND range(0, {} - 1) AS id CREATE "
"(:Node {{id: id, type: 0.5 + tointeger(rand() * 3)}})").format(
node_count)).consume()
# Create the given number of edges, each with a 'value' property of
# a random [0, 3.0) float. Each edge connects two random nodes, so the
# expected node degree is (edge_count * 2 / node_count). Generate edges
# so the connectivity is non-uniform (to produce connected components of
# various sizes).
sess.run(("UNWIND range(0, {0} - 1) AS id WITH id "
"MATCH (from:Node {{id: tointeger(rand() * {1})}}), "
"(to:Node {{id: tointeger(rand() * {1} * id / {0})}}) "
"CREATE (from)-[:Edge {{value: 3 * rand()}}]->(to)").format(
edge_count, node_count)).consume()
def get_connected_ids(sess, node_id):
# Matches a node with the given ID and returns the IDs of all the nodes
# it is connected to. Note that within the BFS lambda expression there
# is an expression used to filter out edges expanded over.
return sess.run((
"MATCH (from:Node {{id: {}}})-"
"[*bfs (e, n | abs(from.type - e.value) < 0.80)]-(d) "
"RETURN count(*) AS c").format(node_id)).data()[0]['c']
def parse_args():
parser = ArgumentParser(description=__doc__)
parser.add_argument('--endpoint', type=str, default='localhost:7687',
help='Memgraph instance endpoint. ')
parser.add_argument('--node-count', type=int, default=1000,
help='The number of nodes in the graph')
parser.add_argument('--edge-count', type=int, default=1000,
help='The number of edges in the graph')
parser.add_argument('--sample-count', type=int, default=None,
help='The number of samples to take')
return parser.parse_args()
def main():
args = parse_args()
logging.basicConfig(level=logging.INFO)
log.info("Memgraph - Otto test database generator")
logging.getLogger("neo4j").setLevel(logging.WARNING)
driver = GraphDatabase.driver(
'bolt://' + args.endpoint,
auth=("ignored", "ignored"),
encrypted=False)
sess = driver.session()
sess.run("MATCH (n) DETACH DELETE n").consume()
log.info("Generating graph with %s nodes and %s edges...",
args.node_count, args.edge_count)
generate_graph(sess, args.node_count, args.edge_count)
# Track which vertices have been found as part of a component.
start_time = time()
max_query_time = 0
log.info("Looking for connected components...")
# Histogram of log2 sizes of connected components found.
histogram = defaultdict(int)
sample_count = args.sample_count if args.sample_count else args.node_count
for i in range(sample_count):
node_id = randint(0, args.node_count - 1)
query_start_time = time()
log2_size = int(log2(1 + get_connected_ids(sess, node_id)))
max_query_time = max(max_query_time, time() - query_start_time)
histogram[log2_size] += 1
elapsed = time() - start_time
log.info("Connected components found in %.2f sec (avg %.2fms, max %.2fms)",
elapsed, elapsed / sample_count * 1000, max_query_time * 1000)
log.info("Component size histogram (count | range)")
for log2_size, count in sorted(histogram.items()):
log.info("\t%5d | %d - %d", count, 2 ** log2_size,
2 ** (log2_size + 1) - 1)
sess.close()
driver.close()
if __name__ == '__main__':
main()

207
customers/otto/parallel_connected_components.cpp
View File

@ -1,207 +0,0 @@
#include <algorithm>
#include <limits>
#include <mutex>
#include <random>
#include <set>
#include <stack>
#include <thread>
#include "gflags/gflags.h"
#include "glog/logging.h"
#include "data_structures/union_find.hpp"
#include "database/graph_db.hpp"
#include "database/graph_db_accessor.hpp"
#include "storage/property_value.hpp"
#include "threading/sync/spinlock.hpp"
#include "utils/bound.hpp"
#include "utils/timer.hpp"
DEFINE_int32(thread_count, 1, "Number of threads");
DEFINE_int32(vertex_count, 1000, "Number of vertices");
DEFINE_int32(edge_count, 1000, "Number of edges");
DECLARE_int32(gc_cycle_sec);
static const std::string kLabel{"kLabel"};
static const std::string kProperty{"kProperty"};
void GenerateGraph(database::GraphDb &db) {
{
database::GraphDbAccessor dba{db};
dba.BuildIndex(dba.Label(kLabel), dba.Property(kProperty));
dba.Commit();
}
// Randomize the sequence of IDs of created vertices and edges to simulate
// real-world lack of locality.
auto make_id_vector = [](size_t size) {
gid::Generator generator{0};
std::vector<gid::Gid> ids(size);
for (size_t i = 0; i < size; ++i)
ids[i] = generator.Next(std::experimental::nullopt);
std::random_shuffle(ids.begin(), ids.end());
return ids;
};
std::vector<VertexAccessor> vertices;
vertices.reserve(FLAGS_vertex_count);
{
CHECK(FLAGS_vertex_count % FLAGS_thread_count == 0)
<< "Thread count must be a factor of vertex count";
LOG(INFO) << "Generating " << FLAGS_vertex_count << " vertices...";
utils::Timer timer;
auto vertex_ids = make_id_vector(FLAGS_vertex_count);
std::vector<std::thread> threads;
SpinLock vertices_lock;
for (int i = 0; i < FLAGS_thread_count; ++i) {
threads.emplace_back([&db, &vertex_ids, &vertices, &vertices_lock, i]() {
database::GraphDbAccessor dba{db};
auto label = dba.Label(kLabel);
auto property = dba.Property(kProperty);
auto batch_size = FLAGS_vertex_count / FLAGS_thread_count;
for (int j = i * batch_size; j < (i + 1) * batch_size; ++j) {
auto vertex = dba.InsertVertex(vertex_ids[j]);
vertex.add_label(label);
vertex.PropsSet(property, static_cast<int64_t>(vertex_ids[j]));
vertices_lock.lock();
vertices.emplace_back(vertex);
vertices_lock.unlock();
}
dba.Commit();
});
}
for (auto &t : threads) t.join();
LOG(INFO) << "Generated " << FLAGS_vertex_count << " vertices in "
<< timer.Elapsed().count() << " seconds.";
}
{
database::GraphDbAccessor dba{db};
for (int i = 0; i < FLAGS_vertex_count; ++i)
vertices[i] = *dba.Transfer(vertices[i]);
LOG(INFO) << "Generating " << FLAGS_edge_count << " edges...";
auto edge_ids = make_id_vector(FLAGS_edge_count);
std::mt19937 pseudo_rand_gen{std::random_device{}()};
std::uniform_int_distribution<> rand_dist{0, FLAGS_vertex_count - 1};
auto edge_type = dba.EdgeType("edge");
utils::Timer timer;
for (int i = 0; i < FLAGS_edge_count; ++i)
dba.InsertEdge(vertices[rand_dist(pseudo_rand_gen)],
vertices[rand_dist(pseudo_rand_gen)], edge_type,
edge_ids[i]);
dba.Commit();
LOG(INFO) << "Generated " << FLAGS_edge_count << " edges in "
<< timer.Elapsed().count() << " seconds.";
}
}
auto EdgeIteration(database::GraphDb &db) {
database::GraphDbAccessor dba{db};
int64_t sum{0};
for (auto edge : dba.Edges(false)) sum += edge.from().gid() + edge.to().gid();
return sum;
}
auto VertexIteration(database::GraphDb &db) {
database::GraphDbAccessor dba{db};
int64_t sum{0};
for (auto v : dba.Vertices(false))
for (auto e : v.out()) sum += e.gid() + e.to().gid();
return sum;
}
auto ConnectedComponentsEdges(database::GraphDb &db) {
UnionFind<int64_t> connectivity{FLAGS_vertex_count};
database::GraphDbAccessor dba{db};
for (auto edge : dba.Edges(false))
connectivity.Connect(edge.from().gid(), edge.to().gid());
return connectivity.Size();
}
auto ConnectedComponentsVertices(database::GraphDb &db) {
UnionFind<int64_t> connectivity{FLAGS_vertex_count};
database::GraphDbAccessor dba{db};
for (auto from : dba.Vertices(false)) {
for (auto out_edge : from.out())
connectivity.Connect(from.gid(), out_edge.to().gid());
}
return connectivity.Size();
}
auto ConnectedComponentsVerticesParallel(database::GraphDb &db) {
UnionFind<int64_t> connectivity{FLAGS_vertex_count};
SpinLock connectivity_lock;
// Define bounds of vertex IDs for each thread to use.
std::vector<PropertyValue> bounds;
for (int64_t i = 0; i < FLAGS_thread_count; ++i)
bounds.emplace_back(i * FLAGS_vertex_count / FLAGS_thread_count);
bounds.emplace_back(std::numeric_limits<int64_t>::max());
std::vector<std::thread> threads;
for (int i = 0; i < FLAGS_thread_count; ++i) {
threads.emplace_back(
[&connectivity, &connectivity_lock, &bounds, &db, i]() {
database::GraphDbAccessor dba{db};
for (auto from :
dba.Vertices(dba.Label(kLabel), dba.Property(kProperty),
utils::MakeBoundInclusive(bounds[i]),
utils::MakeBoundExclusive(bounds[i + 1]), false)) {
for (auto out_edge : from.out()) {
std::lock_guard<SpinLock> lock{connectivity_lock};
connectivity.Connect(from.gid(), out_edge.to().gid());
}
}
});
}
for (auto &t : threads) t.join();
return connectivity.Size();
}
auto Expansion(database::GraphDb &db) {
std::vector<int> component_ids(FLAGS_vertex_count, -1);
int next_component_id{0};
std::stack<VertexAccessor> expansion_stack;
database::GraphDbAccessor dba{db};
for (auto v : dba.Vertices(false)) {
if (component_ids[v.gid()] != -1) continue;
auto component_id = next_component_id++;
expansion_stack.push(v);
while (!expansion_stack.empty()) {
auto next_v = expansion_stack.top();
expansion_stack.pop();
if (component_ids[next_v.gid()] != -1) continue;
component_ids[next_v.gid()] = component_id;
for (auto e : next_v.out()) expansion_stack.push(e.to());
for (auto e : next_v.in()) expansion_stack.push(e.from());
}
}
return next_component_id;
}
int main(int argc, char **argv) {
gflags::ParseCommandLineFlags(&argc, &argv, true);
google::InitGoogleLogging(argv[0]);
FLAGS_gc_cycle_sec = -1;
database::SingleNode db;
GenerateGraph(db);
auto timed_call = [&db](auto callable, const std::string &descr) {
LOG(INFO) << "Running " << descr << "...";
utils::Timer timer;
auto result = callable(db);
LOG(INFO) << "\tDone in " << timer.Elapsed().count()
<< " seconds, result: " << result;
};
timed_call(EdgeIteration, "Edge iteration");
timed_call(VertexIteration, "Vertex iteration");
timed_call(ConnectedComponentsEdges, "Connected components - Edges");
timed_call(ConnectedComponentsVertices, "Connected components - Vertices");
timed_call(ConnectedComponentsVerticesParallel,
"Parallel connected components - Vertices");
timed_call(Expansion, "Expansion");
return 0;
}

5
customers/ravelin/fraud_detection.oc
View File

@ -1,5 +0,0 @@
WITH tointeger(rand() * 40000000) AS from_id
MATCH (from:Node {id : from_id}) WITH from
MATCH path = (from)-[*bfs..50 (e, n | degree(n) < 50)]->(to) WITH path LIMIT 10000 WHERE to.fraudulent
RETURN path, size(path)

31
customers/ravelin/snapshot_gen_config.json
View File

@ -1,31 +0,0 @@
{
"indexes":[
"Node.id"
],
"nodes":[
{
"count":40000000,
"labels":[
"Node"
],
"properties":{
"id":{
"type":"counter",
"param":"Node.id"
},
"fraudulent":{
"type":"bernoulli",
"param":0.0005
}
}
}
],
"edges":[
{
"count":80000000,
"from":"Node",
"to":"Node",
"type":"Edge"
}
]
}

20
customers/strata2018/card_fraud_config.json
View File

@ -1,20 +0,0 @@
{
"indexes" : ["Card.id", "Pos.id", "Transaction.fraud_reported"],
"nodes" : [
{
"count_per_worker" : 1250000,
"label" : "Card"
},
{
"count_per_worker" : 1250000,
"label" : "Pos"
},
{
"count_per_worker" : 2500000,
"label" : "Transaction"
}
],
"compromised_pos_probability" : 0.2,
"fraud_reported_probability" : 0.1,
"hop_probability" : 0.1
}

2
experimental/CMakeLists.txt
View File

@ -1,2 +0,0 @@
# distributed
add_subdirectory(distributed)

8
experimental/distributed/.clang-format
View File

@ -1,8 +0,0 @@
---
Language: Cpp
BasedOnStyle: Google
Standard: "C++11"
UseTab: Never
DerivePointerAlignment: false
PointerAlignment: Right
...

7
experimental/distributed/.gitignore vendored
View File

@ -1,7 +0,0 @@
*.out
*.pyc
main
libs/
*.cereal
*.backup
*.out

42
experimental/distributed/CMakeLists.txt
View File

@ -1,42 +0,0 @@
project(distributed)
# set directory variables
set(src_dir ${PROJECT_SOURCE_DIR}/src)
set(libs_dir ${PROJECT_SOURCE_DIR}/libs)
# includes
include_directories(SYSTEM ${libs_dir}/cereal/include)
include_directories(${src_dir})
# library from distributed sources
file(GLOB_RECURSE src_files ${src_dir}/*.cpp)
add_library(distributed_lib STATIC ${src_files})
## distributed Memgraph executable
set(executable_name main)
add_executable(${executable_name} ${PROJECT_SOURCE_DIR}/main.cpp)
target_link_libraries(${executable_name} distributed_lib memgraph_lib)
## dummy distributed Memgraph client
set(executable_name main-client)
add_executable(${executable_name} ${PROJECT_SOURCE_DIR}/main-client.cpp)
target_link_libraries(${executable_name} distributed_lib memgraph_lib)
# tests
enable_testing()
add_subdirectory(tests)
# copy test scripts into the build/ directory (for distributed tests)
configure_file(${PROJECT_SOURCE_DIR}/tests/start_distributed
${PROJECT_BINARY_DIR}/tests/start_distributed COPYONLY)
configure_file(${PROJECT_SOURCE_DIR}/tests/config
${PROJECT_BINARY_DIR}/tests/config COPYONLY)
# copy main scripts into build/ directory (for distributed Memgraph)
configure_file(${PROJECT_SOURCE_DIR}/start_main.py
${PROJECT_BINARY_DIR}/start_main.py COPYONLY)
configure_file(${PROJECT_SOURCE_DIR}/config
${PROJECT_BINARY_DIR}/config COPYONLY)

35
experimental/distributed/README.md
View File

@ -1,35 +0,0 @@
# distributed memgraph
This subdirectory structure implements distributed infrastructure of Memgraph.
## Terminology
* Memgraph Node Id (mnid): a machine (processs) that runs a (distributed) Memgraph program.
* Node: a computer that performs (distributed) work.
* Vertex: an abstract graph concept.
* Reactor: a unit of concurrent execution, lives on its own thread.
* Channel: a (one-way) communication abstraction between Reactors. The reactors can be on the same machine or on different processes.
* Message: gets sent through channels. Must be serializable if sent via network layer (library: cereal).
* Event: arrival of a (subclass of) Message. You can register callbacks. Register exact callbacks (not for derivated/subclasses).
* EventStream: read-end of a channel, is owned by exactly one Reactor/thread.
* ChannelWriter: write-end of a channel, can be owned (wrote into) by multiple threads.
## Ownership:
* System, Distributed are singletons. They should be always alive.
* ChannelWriter (write-end) should be lightweight and can be copied arbitrarily.
* EventStream (read-end) should never be written by anyone except the owner (the reactor that created it).
* In general: always think about who owns an object. Preferably write it in its comment block.
## Code Conventions
* Locked: A method having a "Locked..." prefix indicates that you
have to lock the appropriate mutex before calling this function.
* ALWAYS close channels. You will memory leak if you don't.
Reactor::CloseChannel or Subscription::Close will do the trick.
## Dependencies
* cereal
* <other memgraph dependencies>

3
experimental/distributed/config
View File

@ -1,3 +0,0 @@
0 127.0.0.1 10010
1 127.0.0.1 10011
2 127.0.0.1 10012

13
experimental/distributed/init
View File

@ -1,13 +0,0 @@
#!/usr/bin/env bash
working_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
libs_dir=$working_dir/libs
if [ ! -d $libs_dir ]; then
mkdir $libs_dir
fi
cd $libs_dir
git clone https://github.com/USCiLab/cereal.git
cd $libs_dir/cereal
git checkout v1.2.2

77
experimental/distributed/main-client.cpp
View File

@ -1,77 +0,0 @@
#include <fstream>
#include <iostream>
#include <memory>
#include "memgraph_config.hpp"
#include "memgraph_distributed.hpp"
#include "memgraph_transactions.hpp"
#include "reactors_distributed.hpp"
/**
* List of queries that should be executed.
*/
std::vector<std::string> queries = {
{"create vertex", "create vertex", "create vertex", "create vertex",
"create vertex", "create vertex", "create vertex", "create vertex",
"create vertex", "create vertex", "vertex count", "create vertex",
"create vertex", "vertex count"}};
/**
* This is the client that issues some hard-coded queries.
*/
class Client : public Reactor {
public:
Client(std::string name) : Reactor(name) {}
void IssueQueries(std::shared_ptr<ChannelWriter> channel_to_leader) {
// (concurrently) create a couple of vertices
for (int query_idx = 0; query_idx < static_cast<int64_t>(queries.size());
++query_idx) {
// register callback
std::string channel_name = "query-" + std::to_string(query_idx);
auto stream = Open(channel_name).first;
stream->OnEventOnce().ChainOnce<ResultMsg>(
[this, query_idx](const ResultMsg &msg, const Subscription &sub) {
std::cout << "Result of query " << query_idx << " ("
<< queries[query_idx] << "):" << std::endl
<< " " << msg.result() << std::endl;
sub.CloseChannel();
});
// then issue the query (to avoid race conditions)
std::cout << "Issuing command " << query_idx << " (" << queries[query_idx]
<< ")" << std::endl;
channel_to_leader->Send<QueryMsg>(channel_name, queries[query_idx]);
}
}
virtual void Run() {
MemgraphDistributed &memgraph = MemgraphDistributed::GetInstance();
auto mnid = memgraph.LeaderMnid();
memgraph.FindChannel(mnid, "master", "client-queries")
->OnEventOnce()
.ChainOnce<ChannelResolvedMessage>(
[this](const ChannelResolvedMessage &msg, const Subscription &sub) {
sub.CloseChannel();
IssueQueries(msg.channelWriter());
});
}
};
int main(int argc, char *argv[]) {
gflags::ParseCommandLineFlags(&argc, &argv, true);
google::InitGoogleLogging(argv[0]);
System &system = System::GetInstance();
Distributed &distributed = Distributed::GetInstance();
MemgraphDistributed &memgraph = MemgraphDistributed::GetInstance();
memgraph.RegisterConfig(ParseConfig());
distributed.StartServices();
system.Spawn<Client>("client");
system.AwaitShutdown();
distributed.StopServices();
return 0;
}

215
experimental/distributed/main.cpp
View File

@ -1,215 +0,0 @@
#include <fstream>
#include <iostream>
#include <random>
#include <gflags/gflags.h>
#include <glog/logging.h>
#include "memgraph_config.hpp"
#include "memgraph_distributed.hpp"
#include "memgraph_transactions.hpp"
#include "reactors_distributed.hpp"
#include "storage.hpp"
DEFINE_uint64(my_mnid, -1, "Memgraph node id"); // TODO(zuza): this should be assigned by the leader once in the future
class Master : public Reactor {
public:
Master(std::string name, MnidT mnid) : Reactor(name), mnid_(mnid) {
MemgraphDistributed& memgraph = MemgraphDistributed::GetInstance();
worker_mnids_ = memgraph.GetAllMnids();
// remove the leader (itself), because it is not a worker
auto leader_it = std::find(worker_mnids_.begin(), worker_mnids_.end(), memgraph.LeaderMnid());
worker_mnids_.erase(leader_it);
}
virtual void Run() {
Distributed &distributed = Distributed::GetInstance();
std::cout << "Master (" << mnid_ << ") @ " << distributed.network().Address()
<< ":" << distributed.network().Port() << std::endl;
// TODO(zuza): check if all workers are up
// start listening on queries arriving from the client
auto stream = Open("client-queries").first;
stream->OnEvent<QueryMsg>([this](const QueryMsg &msg, const Subscription &){
// process query message
if (msg.query() == "create vertex") {
InstallMakeVertex(msg.GetReturnChannelWriter());
} else if (msg.query() == "vertex count") {
InstallVertexCount(msg.GetReturnChannelWriter());
} else {
std::cerr << "unknown query" << std::endl;
}
});
}
private:
/**
* Organizes communication with all workers and performs VertexCount.
*/
void InstallVertexCount(std::shared_ptr<ChannelWriter> return_channel) {
// open channel through which answers will arrive
auto channel_name = "response" + std::to_string(xid++);
auto result = Open(channel_name).first;
// create struct to keep track of responses
struct VertexCountResponse {
VertexCountResponse(int64_t count, int64_t remaining)
: count_(count), remaining_(remaining) {}
int64_t count_;
int64_t remaining_;
};
// allocate it dynamically so it lives outside the scope of this function
// it will be deallocated once all responses arrive and channel is closed
auto response = std::make_shared<VertexCountResponse>(0, worker_mnids_.size());
// register callbacks
result->OnEvent<ResultQueryVertexCount>(
[this, response, return_channel](const ResultQueryVertexCount &msg,
const Subscription &sub){
response->count_ += msg.count();
--response->remaining_;
if (response->remaining_ == 0) {
sub.CloseChannel();
return_channel->Send<ResultMsg>(std::to_string(response->count_));
}
});
// instruct workers to count vertices
for (auto wmnid : worker_mnids_)
VertexCount(wmnid, channel_name);
}
/**
* Asynchronously counts vertices on the given node.
*
* @param mnid Id of the node whose vertices should be counted.
* @param channel_name Name of the channel on which response will arrive.
*/
void VertexCount(MnidT mnid, std::string channel_name) {
MemgraphDistributed::GetInstance().FindChannel(mnid, "worker", "main")
->OnEventOnceThenClose<ChannelResolvedMessage>(
[this, channel_name](const ChannelResolvedMessage &msg){
msg.channelWriter()->Send<QueryVertexCount>(channel_name);
});
}
/**
* Organizes communication with a random worker and performs MakeVertex.
*/
void InstallMakeVertex(std::shared_ptr<ChannelWriter> return_channel) {
// choose worker on random and instruct it to make vertex
auto wmnid = worker_mnids_[rand() % worker_mnids_.size()];
// open channel through which answer will arrive
auto channel_name = "response" + std::to_string(xid++);
auto result = Open(channel_name).first;
// register callbacks for the answer
// TODO(zuza): this is actually pretty bad because if SuccessQueryCreateVertex arrives, then
// FailureQueryCreateVertex never gets unsubscribed. This could cause memory leaks
// in the future (not currently since all callbacks get destroyed when channel is closed).
// The best thing to do is to implement a ThenOnce and Either. Perhaps even a ThenClose.
// An Either in conjunction with a failure detector event stream should eventually fail
// the transaction and close the channel.
result->OnEventOnceThenClose<SuccessQueryCreateVertex>(
[this, return_channel](const SuccessQueryCreateVertex &) {
return_channel->Send<ResultMsg>("success");
});
result->OnEventOnceThenClose<FailureQueryCreateVertex>(
[this, return_channel](const FailureQueryCreateVertex &) {
return_channel->Send<ResultMsg>("failure");
});
// instruct worker to make vertex
MakeVertex(wmnid, channel_name);
}
/**
* Asynchronously creates vertex on the give node.
*
* @param mnid Id of the node on which vertex should be created.
* @param channel_name Name of the channel on which response will arrive.
*/
void MakeVertex(MnidT mnid, std::string channel_name) {
MemgraphDistributed::GetInstance().FindChannel(mnid, "worker", "main")
->OnEventOnceThenClose<ChannelResolvedMessage>(
[this, channel_name](const ChannelResolvedMessage &msg){
msg.channelWriter()->Send<QueryCreateVertex>(channel_name);
});
}
protected:
// node id
const MnidT mnid_;
// transaction id
int64_t xid{0};
// list of ids of nodes that act as worker
std::vector<MnidT> worker_mnids_;
};
class Worker : public Reactor {
public:
Worker(std::string name, MnidT mnid)
: Reactor(name), mnid_(mnid), storage_(mnid) {}
virtual void Run() {
Distributed &distributed = Distributed::GetInstance();
std::cout << "Worker (" << mnid_ << ") @ " << distributed.network().Address()
<< ":" << distributed.network().Port() << std::endl;
main_.first->OnEvent<QueryCreateVertex>([this](const QueryCreateVertex& msg,
const Subscription &) {
std::random_device rd; // slow random number generator
// succeed and fail with 50-50 (just for testing)
// TODO: remove random failure
if (rd() % 2 == 0) {
storage_.MakeVertex();
std::cout << "Vertex created" << std::endl;
msg.GetReturnChannelWriter()->Send<SuccessQueryCreateVertex>();
} else {
msg.GetReturnChannelWriter()->Send<FailureQueryCreateVertex>();
}
});
main_.first->OnEvent<QueryVertexCount>([this](const QueryVertexCount &msg,
const Subscription &){
auto count = storage_.VertexCount();
msg.GetReturnChannelWriter()->Send<ResultQueryVertexCount>(count);
});
}
protected:
const MnidT mnid_;
ShardedStorage storage_;
};
int main(int argc, char *argv[]) {
gflags::ParseCommandLineFlags(&argc, &argv, /* remove flags from command line */ true);
std::string logging_name = std::string(argv[0]) + "-mnid-" + std::to_string(FLAGS_my_mnid);
google::InitGoogleLogging(logging_name.c_str());
System &system = System::GetInstance();
Distributed& distributed = Distributed::GetInstance();
MemgraphDistributed& memgraph = MemgraphDistributed::GetInstance();
memgraph.RegisterConfig(ParseConfig());
distributed.StartServices();
if (FLAGS_my_mnid == memgraph.LeaderMnid()) {
system.Spawn<Master>("master", FLAGS_my_mnid);
} else {
system.Spawn<Worker>("worker", FLAGS_my_mnid);
}
system.AwaitShutdown();
distributed.StopServices();
return 0;
}

164
experimental/distributed/src/graph.hpp
View File

@ -1,164 +0,0 @@
#pragma once
#include <cassert>
#include <unordered_map>
#include <vector>
#include "uid.hpp"
enum class EdgeType { OUTGOING, INCOMING };
/** A vertex in the graph. Has incoming and outgoing edges which
* are defined as global addresses of other vertices */
class Vertex {
public:
Vertex(const UniqueVid &id) : id_(id) {}
const auto &id() const { return id_; };
const auto &edges_out() const { return edges_out_; }
const auto &edges_in() const { return edges_in_; }
void AddConnection(EdgeType edge_type, const GlobalVertAddress &gad) {
(edge_type == EdgeType::INCOMING ? edges_in_ : edges_out_)
.emplace_back(gad);
}
/** Changes all old_address edges to have the new Memgraph node id */
void RedirectEdges(const GlobalVertAddress& old_address, int64_t new_mnid) {
for (auto &address : edges_in_)
if (address == old_address) address.cur_mnid_ = new_mnid;
for (auto &address : edges_out_)
if (address == old_address) address.cur_mnid_ = new_mnid;
}
private:
UniqueVid id_;
// global addresses of vertices this vertex is connected to
std::vector<GlobalVertAddress> edges_out_;
std::vector<GlobalVertAddress> edges_in_;
};
/**
* A storage that doesn't assume everything is in-memory.
*/
class ShardedStorage {
public:
// Unique Memgraph node ID. Uniqueness is ensured by the (distributed) system.
const int64_t mnid_;
ShardedStorage(int64_t mnid) : mnid_(mnid) {}
int64_t VertexCount() const { return vertices_.size(); }
/** Gets a vertex. */
Vertex &GetVertex(const UniqueVid &gid) {
auto found = vertices_.find(gid);
assert(found != vertices_.end());
return found->second;
}
/**
* Returns the number of edges that cross from this
* node into another one
*/
int64_t BoundaryEdgeCount() const {
int64_t count = 0;
auto count_f = [this, &count](const auto &edges) {
for (const GlobalVertAddress &address : edges)
if (address.cur_mnid_ != mnid_) count++;
};
for (const auto &vertex : vertices_) {
count_f(vertex.second.edges_out());
count_f(vertex.second.edges_in());
}
return count;
}
/** Creates a new vertex on this node. Returns its global id */
const UniqueVid &MakeVertex() {
UniqueVid new_id(mnid_, next_vertex_sequence_++);
auto new_vertex = vertices_.emplace(std::make_pair(new_id, Vertex(new_id)));
return new_vertex.first->first;
};
/** Places the existing vertex on this node */
void PlaceVertex(const UniqueVid &gid, const Vertex &vertex) {
vertices_.emplace(gid, vertex);
}
/** Removes the vertex with the given ID from this node */
void RemoveVertex(const UniqueVid &gid) { vertices_.erase(gid); }
auto begin() const { return vertices_.begin(); }
auto end() const { return vertices_.end(); }
private:
// counter of sequences numbers of vertices created on this node
int64_t next_vertex_sequence_{0};
// vertex storage of this node
std::unordered_map<UniqueVid, Vertex> vertices_;
};
/**
* A distributed system consisting of mulitple nodes.
* For the time being it's not modelling a distributed
* system correctly in terms of message passing (as opposed
* to operating on nodes and their data directly).
*/
class Distributed {
public:
/** Creates a distributed with the given number of nodes */
Distributed(int initial_mnode_count = 0) {
for (int mnode_id = 0; mnode_id < initial_mnode_count; mnode_id++)
AddMnode();
}
int64_t AddMnode() {
int64_t new_mnode_id = mnodes_.size();
mnodes_.emplace_back(new_mnode_id);
return new_mnode_id;
}
int MnodeCount() const { return mnodes_.size(); }
auto &GetMnode(int64_t mnode_id) { return mnodes_[mnode_id]; }
GlobalVertAddress MakeVertex(int64_t mnid) {
return {mnid, mnodes_[mnid].MakeVertex()};
}
Vertex &GetVertex(const GlobalVertAddress &address) {
return mnodes_[address.cur_mnid_].GetVertex(address.uvid_);
}
/** Moves a vertex with the given global id to the given mnode */
void MoveVertex(const GlobalVertAddress &gad, int64_t destination) {
const Vertex &vertex = GetVertex(gad);
// make sure that all edges to and from the vertex are updated
for (auto &edge : vertex.edges_in())
GetVertex(edge).RedirectEdges(gad, destination);
for (auto &edge : vertex.edges_out())
GetVertex(edge).RedirectEdges(gad, destination);
// change vertex destination
mnodes_[destination].PlaceVertex(gad.uvid_, vertex);
mnodes_[gad.cur_mnid_].RemoveVertex(gad.uvid_);
}
void MakeEdge(const GlobalVertAddress &from, const GlobalVertAddress &to) {
GetVertex(from).AddConnection(EdgeType::OUTGOING, to);
GetVertex(to).AddConnection(EdgeType::INCOMING, from);
}
auto begin() const { return mnodes_.begin(); }
auto end() const { return mnodes_.end(); }
private:
std::vector<ShardedStorage> mnodes_;
};

25
experimental/distributed/src/memgraph_config.cpp
View File

@ -1,25 +0,0 @@
#include "memgraph_config.hpp"
DEFINE_string(config_filename, "", "File containing list of all processes");
Config ParseConfig(const std::string &filename) {
std::ifstream file(filename, std::ifstream::in);
assert(file.good());
Config config;
while (file.good()) {
MnidT mnid;
std::string address;
uint16_t port;
file >> mnid >> address >> port;
if (file.eof())
break;
config.nodes.push_back(Config::NodeConfig{mnid, address, port});
}
file.close();
return config;
}

39
experimental/distributed/src/memgraph_config.hpp
View File

@ -1,39 +0,0 @@
#pragma once
#include <cstdint>
#include <cassert>
#include <fstream>
#include <utility>
#include <vector>
#include <string>
#include <gflags/gflags.h>
/**
* About config file
*
* Each line contains three strings:
* memgraph node id, ip address of the worker, and port of the worker
* Data on the first line is used to start master.
* Data on the remaining lines is used to start workers.
*/
DECLARE_string(config_filename);
using MnidT = uint64_t;
struct Config {
struct NodeConfig {
MnidT mnid;
std::string address;
uint16_t port;
};
std::vector<NodeConfig> nodes;
};
/**
* Parse config file.
*
* @return config object.
*/
Config ParseConfig(const std::string &filename = FLAGS_config_filename);

74
experimental/distributed/src/memgraph_distributed.hpp
View File

@ -1,74 +0,0 @@
#pragma once
#include "memgraph_config.hpp"
#include "reactors_distributed.hpp"
#include <mutex>
#include <string>
#include <unordered_map>
#include <utility>
#include <vector>
class MemgraphDistributed {
private:
using Location = std::pair<std::string, uint16_t>;
public:
/**
* Get the (singleton) instance of MemgraphDistributed.
*/
static MemgraphDistributed &GetInstance() {
static MemgraphDistributed memgraph;
return memgraph;
}
EventStream *FindChannel(MnidT mnid, const std::string &reactor,
const std::string &channel) {
std::unique_lock<std::mutex> lock(mutex_);
const auto &location = mnodes_.at(mnid);
return Distributed::GetInstance().FindChannel(
location.first, location.second, reactor, channel);
}
void RegisterConfig(const Config &config) {
config_ = config;
for (auto &node : config_.nodes) {
RegisterMemgraphNode(node.mnid, node.address, node.port);
}
}
std::vector<MnidT> GetAllMnids() {
std::vector<MnidT> mnids;
for (auto &node : config_.nodes) {
mnids.push_back(node.mnid);
}
return mnids;
}
/**
* The leader is currently the first node in the config.
*/
MnidT LeaderMnid() const { return config_.nodes.front().mnid; }
protected:
MemgraphDistributed() {}
/** Register memgraph node id to the given location. */
void RegisterMemgraphNode(MnidT mnid, const std::string &address,
uint16_t port) {
std::unique_lock<std::mutex> lock(mutex_);
mnodes_[mnid] = Location(address, port);
}
private:
Config config_;
std::mutex mutex_;
std::unordered_map<MnidT, Location> mnodes_;
MemgraphDistributed(const MemgraphDistributed &) = delete;
MemgraphDistributed(MemgraphDistributed &&) = delete;
MemgraphDistributed &operator=(const MemgraphDistributed &) = delete;
MemgraphDistributed &operator=(MemgraphDistributed &&) = delete;
};

135
experimental/distributed/src/memgraph_transactions.hpp
View File

@ -1,135 +0,0 @@
#pragma once
#include <string>
#include "reactors_local.hpp"
#include "reactors_distributed.hpp"
/**
* Message which encapsulates query.
* It is create on Client and sent to Master which will process it.
*/
class QueryMsg : public ReturnAddressMsg {
public:
QueryMsg(std::string return_channel, std::string query)
: ReturnAddressMsg(return_channel), query_(query) {}
const std::string &query() const { return query_; }
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<ReturnAddressMsg>(this), query_);
}
protected:
// Cereal needs access to default constructor.
friend class cereal::access;
QueryMsg() = default;
std::string query_;
};
CEREAL_REGISTER_TYPE(QueryMsg);
/**
* Message which encapuslates result of a query.
* Currently, result is string.
*/
class ResultMsg : public Message {
public:
ResultMsg(std::string result) : result_(result) {}
const std::string &result() const { return result_; }
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<Message>(this), result_);
}
protected:
friend class cereal::access;
ResultMsg() = default;
std::string result_;
};
CEREAL_REGISTER_TYPE(ResultMsg);
/**
* Below are message that are exchanged between Master and Workers.
*/
class QueryCreateVertex : public ReturnAddressMsg {
public:
QueryCreateVertex(std::string return_channel)
: ReturnAddressMsg(return_channel) {}
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<ReturnAddressMsg>(this));
}
protected:
// Cereal needs access to default constructor.
friend class cereal::access;
QueryCreateVertex() {}
};
CEREAL_REGISTER_TYPE(QueryCreateVertex);
class SuccessQueryCreateVertex : public Message {
public:
SuccessQueryCreateVertex() {}
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<Message>(this));
}
};
CEREAL_REGISTER_TYPE(SuccessQueryCreateVertex);
class FailureQueryCreateVertex : public Message {
public:
FailureQueryCreateVertex() {}
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<Message>(this));
}
};
CEREAL_REGISTER_TYPE(FailureQueryCreateVertex);
class QueryVertexCount : public ReturnAddressMsg {
public:
QueryVertexCount(std::string return_channel)
: ReturnAddressMsg(return_channel) {}
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<ReturnAddressMsg>(this));
}
protected:
// Cereal needs access to default constructor.
friend class cereal::access;
QueryVertexCount() {}
};
CEREAL_REGISTER_TYPE(QueryVertexCount);
class ResultQueryVertexCount : public Message {
public:
ResultQueryVertexCount(int64_t count) : count_(count) {}
int64_t count() const { return count_; }
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<Message>(this), count_);
}
protected:
// Cereal needs access to default constructor.
friend class cereal::access;
ResultQueryVertexCount() {}
int64_t count_;
};
CEREAL_REGISTER_TYPE(ResultQueryVertexCount);

147
experimental/distributed/src/spinner.hpp
View File

@ -1,147 +0,0 @@
#include <algorithm>
#include <cassert>
#include <cstddef>
#include <experimental/tuple>
#include <iostream>
#include <numeric>
#include <random>
#include <tuple>
#include <vector>
#include "graph.hpp"
namespace spinner {
// const for balancing penalty
double c = 2.0;
/**
* Returns the index of the maximum score in the given vector.
* If there are multiple minimums, one is chosen at random.
*/
auto MaxRandom(const std::vector<double> &scores) {
std::vector<size_t> best_indices;
double current_max = std::numeric_limits<double>::lowest();
for (size_t ind = 0; ind < scores.size(); ind++) {
if (scores[ind] > current_max) {
current_max = scores[ind];
best_indices.clear();
}
if (scores[ind] == current_max) {
best_indices.emplace_back(ind);
}
}
return best_indices[rand() % best_indices.size()];
}
/**
* Returns the index of the best (highest scored) mnode
* for the given vertex. If there are multiple mnodes with
* the best score, vertex prefers to remain on the same mnode
* (if among the best), or one is chosen at random.
*
* @param distributed - the distributed system.
* @param vertex - the vertex which is being evaluated.
* @param penalties - a vector of penalties (per mnode).
* @param current_mnode - the mnode on which the given
* vertex is currently residing.
* @return - std::pair<int, std::vector<double>> which is a
* pair of (best mnode, score_per_mnode).
*/
auto BestMnode(const Distributed &distributed, const Vertex &vertex,
const std::vector<double> &penalties, int current_mnode) {
// scores per mnode
std::vector<double> scores(distributed.MnodeCount(), 0.0);
for (auto &edge : vertex.edges_in()) scores[edge.cur_mnid_] += 1.0;
for (auto &edge : vertex.edges_out()) scores[edge.cur_mnid_] += 1.0;
for (int mnode = 0; mnode < distributed.MnodeCount(); ++mnode) {
// normalize contribution of mnode over neighbourhood size
scores[mnode] /= vertex.edges_out().size() + vertex.edges_in().size();
// add balancing penalty
scores[mnode] -= penalties[mnode];
}
// pick the best destination, but prefer to stay if you can
size_t destination = MaxRandom(scores);
if (scores[current_mnode] == scores[destination])
destination = current_mnode;
return std::make_pair(destination, scores);
}
/** Indication if Spinner mnode penality is calculated based on
* vertex or edge mnode cardinalities */
enum class PenaltyType { Vertex, Edge };
/** Calcualtes Spinner penalties for mnodes in the given
* distributed system. */
auto Penalties(const Distributed &distributed,
PenaltyType penalty_type = PenaltyType::Edge) {
std::vector<double> penalties;
int64_t total_count{0};
for (const auto &mnode : distributed) {
int64_t mnode_count{0};
switch (penalty_type) {
case PenaltyType::Vertex:
mnode_count += mnode.VertexCount();
break;
case PenaltyType::Edge:
for (const auto &vertex_kv : mnode) {
// Spinner counts the edges on a mnode as the sum
// of degrees of vertices on that mnode. In that sense
// both incoming and outgoing edges are individually
// added...
mnode_count += vertex_kv.second.edges_out().size();
mnode_count += vertex_kv.second.edges_in().size();
}
break;
}
total_count += mnode_count;
penalties.emplace_back(mnode_count);
}
for (auto &penalty : penalties)
penalty /= c * total_count / distributed.MnodeCount();
return penalties;
}
/** Do one spinner step (modifying the given distributed) */
void PerformSpinnerStep(Distributed &distributed) {
auto penalties = Penalties(distributed);
// here a strategy can be injected for limiting
// the number of movements performed in one step.
// limiting could be based on (for example):
// - limiting the number of movements per mnode
// - limiting only to movements that are above
// a treshold (score improvement or something)
// - not executing on all the mnodes (also prevents
// oscilations)
//
// in the first implementation just accumulate all
// the movements and execute together.
// relocation info: contains the address of the Vertex
// that needs to relocate and it's destination mnode
std::vector<std::pair<GlobalVertAddress, int>> movements;
for (const ShardedStorage &mnode : distributed)
for (const auto &gid_vertex_pair : mnode) {
// (best destination, scores) pair for vertex
std::pair<int, std::vector<double>> destination_scores =
BestMnode(distributed, gid_vertex_pair.second, penalties, mnode.mnid_);
if (destination_scores.first != mnode.mnid_)
movements.emplace_back(GlobalVertAddress(mnode.mnid_, gid_vertex_pair.first),
destination_scores.first);
}
// execute movements. it is likely that in the real system
// this will need to happen as a single db transaction
for (const auto &m : movements) distributed.MoveVertex(m.first, m.second);
}
} // namespace spinner

45
experimental/distributed/src/storage.hpp
View File

@ -1,45 +0,0 @@
#pragma once
#include <string>
#include <unordered_map>
#include "uid.hpp"
/**
* Mock-up storage to test basic hardcoded queries.
*
* Current code is taken from graph.hpp. It will grow as needed to support
* new queries, but not more. Once all functionality from graph.hpp is
* transfered here, this file will be included in graph.hpp.
*/
class Vertex {};
/**
* Storage that is split over multiple nodes.
*/
class ShardedStorage {
public:
ShardedStorage(int64_t mnid) : mnid_(mnid) {}
/** Returns number of vertices on this node. */
int64_t VertexCount() const { return vertices_.size(); }
/** Creates a new vertex on this node. Returns its global id */
const UniqueVid &MakeVertex() {
UniqueVid new_id(mnid_, next_vertex_sequence_++);
auto new_vertex = vertices_.emplace(std::make_pair(new_id, Vertex()));
return new_vertex.first->first;
};
private:
// unique Memgraph node ID
// uniqueness is ensured by the (distributed) system
const int64_t mnid_;
// counter of vertices created on this node.
int64_t next_vertex_sequence_{0};
// vertex storage of this node
std::unordered_map<UniqueVid, Vertex> vertices_;
};

66
experimental/distributed/src/uid.hpp
View File

@ -1,66 +0,0 @@
#pragma once
#include "utils/hashing/fnv.hpp"
#include <cstdint>
#include <vector>
/**
* Globally unique id (in the entire distributed system) of a vertex.
*
* It is identified by a pair of a (original memgraph node, local vertex id)
*/
class UniqueVid {
public:
UniqueVid(int64_t orig_mnid, int64_t vid)
: orig_mnid_(orig_mnid), vid_(vid) {}
/** Original Memgraph node the vertex was created **/
int64_t orig_mnid_;
/** Original vertex id it was assigned on creation. **/
int64_t vid_;
bool operator==(const UniqueVid &other) const {
return orig_mnid_ == other.orig_mnid_ &&
vid_ == other.vid_;
}
bool operator!=(const UniqueVid &other) const { return !(*this == other); }
};
/**
* Specifies where a vertex is in the distributed system.
*/
class GlobalVertAddress {
public:
GlobalVertAddress(int64_t cur_mnid, const UniqueVid &uvid)
: cur_mnid_(cur_mnid), uvid_(uvid) {}
/** The current Memgraph node where the vertex is **/
int64_t cur_mnid_;
UniqueVid uvid_;
bool operator==(const GlobalVertAddress &other) const {
return cur_mnid_ == other.cur_mnid_ && uvid_ == other.uvid_;
}
bool operator!=(const GlobalVertAddress &other) const {
return !(*this == other);
}
};
namespace std {
template <>
struct hash<UniqueVid> {
size_t operator()(const UniqueVid &uid) const {
return HashCombine<decltype(uid.orig_mnid_), decltype(uid.vid_)>()(uid.orig_mnid_, uid.vid_);
}
};
template <>
struct hash<GlobalVertAddress> {
size_t operator()(const GlobalVertAddress &ga) const {
return HashCombine<decltype(ga.cur_mnid_), decltype(ga.uvid_)>()(ga.cur_mnid_, ga.uvid_);
}
};
}

46
experimental/distributed/start_main.py
View File

@ -1,46 +0,0 @@
#!/usr/bin/env python3
# Automatically copied to the build/ directory during Makefile (configured by cmake)
import os
terminal_command = 'gnome-terminal'
terminal_flags = ' --geometry=200x50 ' # columns x rows
config_filename = 'config'
log_dir = "logs"
glog_flags = '--alsologtostderr --logbufsecs=0 --minloglevel=2 --log_dir="{}" '.format(log_dir)
def GetMainCall(my_mnid, address, port):
ret = "./main {} --my_mnid {} --address {} --port {} --config_filename={}".format(
glog_flags, my_mnid, address, port, config_filename)
print(ret)
return ret
def GetClientCall():
ret = "./main-client {} --address 127.0.0.1 --port 10000 --config_filename={}".format(
glog_flags, config_filename)
print(ret)
return ret
def NamedGnomeTab(name, command):
return " --tab -e \"bash -c 'printf \\\"\\033]0;{}\\007\\\"; {}'\" ".format(name, command)
if __name__ == "__main__":
command = "{} {}".format(terminal_command, terminal_flags)
command += NamedGnomeTab("client", GetClientCall())
f = open(config_filename, 'r')
for line in f:
data = line.strip().split(' ')
my_mnid = data[0]
address = data[1]
port = data[2]
command += NamedGnomeTab("mnid={}".format(my_mnid), GetMainCall(my_mnid, address, port))
print(command)
os.system('mkdir -p {}'.format(log_dir))
os.system(command)

35
experimental/distributed/tests/CMakeLists.txt
View File

@ -1,35 +0,0 @@
# set current directory name as a test type
get_filename_component(test_type ${CMAKE_CURRENT_SOURCE_DIR} NAME)
# get all cpp abs file names recursively starting from current directory
file(GLOB_RECURSE test_type_cpps *.cpp)
message(STATUS "Available ${test_type} cpp files are: ${test_type_cpps}")
include_directories(${GTEST_INCLUDE_DIR})
# for each cpp file build binary and register test
foreach(test_cpp ${test_type_cpps})
# get exec name (remove extension from the abs path)
get_filename_component(exec_name ${test_cpp} NAME_WE)
set(target_name memgraph__${test_type}__${exec_name})
# build exec file
add_executable(${target_name} ${test_cpp})
# OUTPUT_NAME sets the real name of a target when it is built and can be
# used to help create two targets of the same name even though CMake
# requires unique logical target names
set_target_properties(${target_name} PROPERTIES OUTPUT_NAME ${exec_name})
# link libraries
target_link_libraries(${target_name} distributed_lib memgraph_lib)
# gtest
target_link_libraries(${target_name} gtest gtest_main)
# register test
set(output_path ${CMAKE_BINARY_DIR}/test_results/unit/${target_name}.xml)
add_test(${target_name} ${exec_name} --gtest_output=xml:${output_path})
endforeach()

3
experimental/distributed/tests/config
View File

@ -1,3 +0,0 @@
0 127.0.0.1 10000
2 127.0.0.1 10001
3 127.0.0.1 10002

213
experimental/distributed/tests/distributed_test.cpp
View File

@ -1,213 +0,0 @@
#include <fstream>
#include <iostream>
#include <glog/logging.h>
#include "memgraph_config.hpp"
#include "reactors_distributed.hpp"
DEFINE_int64(my_mnid, 0, "Memgraph node id"); // TODO(zuza): this should be
// assigned by the leader once in
// the future
class MemgraphDistributed {
private:
using Location = std::pair<std::string, uint16_t>;
public:
/**
* Get the (singleton) instance of MemgraphDistributed.
*
* More info:
* https://stackoverflow.com/questions/1008019/c-singleton-design-pattern
*/
static MemgraphDistributed &GetInstance() {
static MemgraphDistributed
memgraph; // guaranteed to be destroyed, initialized on first use
return memgraph;
}
/** Register memgraph node id to the given location. */
void RegisterMemgraphNode(int64_t mnid, const std::string &address,
uint16_t port) {
std::unique_lock<std::mutex> lock(mutex_);
mnodes_[mnid] = Location(address, port);
}
EventStream *FindChannel(int64_t mnid, const std::string &reactor,
const std::string &channel) {
std::unique_lock<std::mutex> lock(mutex_);
const auto &location = mnodes_.at(mnid);
return Distributed::GetInstance().FindChannel(
location.first, location.second, reactor, channel);
}
protected:
MemgraphDistributed() {}
private:
std::mutex mutex_;
std::unordered_map<int64_t, Location> mnodes_;
MemgraphDistributed(const MemgraphDistributed &) = delete;
MemgraphDistributed(MemgraphDistributed &&) = delete;
MemgraphDistributed &operator=(const MemgraphDistributed &) = delete;
MemgraphDistributed &operator=(MemgraphDistributed &&) = delete;
};
/**
* About config file
*
* Each line contains three strings:
* memgraph node id, ip address of the worker, and port of the worker
* Data on the first line is used to start master.
* Data on the remaining lines is used to start workers.
*/
/**
* Parse config file and register processes into system.
*
* @return Pair (master mnid, list of worker's id).
*/
std::pair<int64_t, std::vector<int64_t>> ParseConfigAndRegister(
const std::string &filename) {
std::ifstream file(filename, std::ifstream::in);
assert(file.good());
int64_t master_mnid;
std::vector<int64_t> worker_mnids;
int64_t mnid;
std::string address;
uint16_t port;
file >> master_mnid >> address >> port;
MemgraphDistributed &memgraph = MemgraphDistributed::GetInstance();
memgraph.RegisterMemgraphNode(master_mnid, address, port);
while (file.good()) {
file >> mnid >> address >> port;
if (file.eof()) break;
memgraph.RegisterMemgraphNode(mnid, address, port);
worker_mnids.push_back(mnid);
}
file.close();
return std::make_pair(master_mnid, worker_mnids);
}
/**
* Sends a text message and has a return address.
*/
class TextMessage : public ReturnAddressMsg {
public:
TextMessage(std::string reactor, std::string channel, std::string s)
: ReturnAddressMsg(reactor, channel), text(s) {}
template <class Archive>
void serialize(Archive &archive) {
archive(cereal::virtual_base_class<ReturnAddressMsg>(this), text);
}
std::string text;
protected:
friend class cereal::access;
TextMessage() {} // Cereal needs access to a default constructor.
};
CEREAL_REGISTER_TYPE(TextMessage);
class Master : public Reactor {
public:
Master(std::string name, int64_t mnid, std::vector<int64_t> &&worker_mnids)
: Reactor(name), mnid_(mnid), worker_mnids_(std::move(worker_mnids)) {}
virtual void Run() {
MemgraphDistributed &memgraph = MemgraphDistributed::GetInstance();
Distributed &distributed = Distributed::GetInstance();
std::cout << "Master (" << mnid_ << ") @ "
<< distributed.network().Address() << ":"
<< distributed.network().Port() << std::endl;
auto stream = main_.first;
// wait until every worker sends a ReturnAddressMsg back, then close
stream->OnEvent<TextMessage>(
[this](const TextMessage &msg, const Subscription &subscription) {
std::cout << "Message from " << msg.Address() << ":" << msg.Port()
<< " .. " << msg.text << "\n";
++workers_seen;
if (workers_seen == static_cast<int64_t>(worker_mnids_.size())) {
subscription.Unsubscribe();
// Sleep for a while so we can read output in the terminal.
// (start_distributed runs each process in a new tab which is
// closed immediately after process has finished)
std::this_thread::sleep_for(std::chrono::seconds(4));
CloseChannel("main");
}
});
// send a TextMessage to each worker
for (auto wmnid : worker_mnids_) {
auto stream = memgraph.FindChannel(wmnid, "worker", "main");
stream->OnEventOnce().ChainOnce<ChannelResolvedMessage>(
[this, stream](const ChannelResolvedMessage &msg,
const Subscription &) {
msg.channelWriter()->Send<TextMessage>("master", "main",
"hi from master");
stream->Close();
});
}
}
protected:
int64_t workers_seen = 0;
const int64_t mnid_;
std::vector<int64_t> worker_mnids_;
};
class Worker : public Reactor {
public:
Worker(std::string name, int64_t mnid, int64_t master_mnid)
: Reactor(name), mnid_(mnid), master_mnid_(master_mnid) {}
virtual void Run() {
Distributed &distributed = Distributed::GetInstance();
std::cout << "Worker (" << mnid_ << ") @ "
<< distributed.network().Address() << ":"
<< distributed.network().Port() << std::endl;
auto stream = main_.first;
// wait until master sends us a TextMessage, then reply back and close
stream->OnEventOnce().ChainOnce<TextMessage>(
[this](const TextMessage &msg, const Subscription &) {
std::cout << "Message from " << msg.Address() << ":" << msg.Port()
<< " .. " << msg.text << "\n";
msg.GetReturnChannelWriter()->Send<TextMessage>("worker", "main",
"hi from worker");
// Sleep for a while so we can read output in the terminal.
std::this_thread::sleep_for(std::chrono::seconds(4));
CloseChannel("main");
});
}
protected:
const int64_t mnid_;
const int64_t master_mnid_;
};
int main(int argc, char *argv[]) {
google::InitGoogleLogging(argv[0]);
gflags::ParseCommandLineFlags(&argc, &argv, true);
System &system = System::GetInstance();
auto mnids = ParseConfigAndRegister(FLAGS_config_filename);
Distributed::GetInstance().StartServices();
if (FLAGS_my_mnid == mnids.first)
system.Spawn<Master>("master", FLAGS_my_mnid, std::move(mnids.second));
else
system.Spawn<Worker>("worker", FLAGS_my_mnid, mnids.first);
system.AwaitShutdown();
Distributed::GetInstance().StopServices();
return 0;
}

67
experimental/distributed/tests/graph_test.cpp
View File

@ -1,67 +0,0 @@
#include <cassert>
#include <iostream>
#include <iterator>
#include "graph.hpp"
void test_global_id() {
UniqueVid a(1, 1);
assert(a == UniqueVid(1, 1));
assert(a != UniqueVid(1, 2));
assert(a != UniqueVid(2, 1));
}
void test_global_address() {
GlobalVertAddress a(1, {1, 1});
assert(a == GlobalVertAddress(1, {1, 1}));
assert(a != GlobalVertAddress(2, {1, 1}));
assert(a != GlobalVertAddress(1, {2, 1}));
}
void test_mnode() {
ShardedStorage mnode0{0};
assert(mnode0.VertexCount() == 0);
UniqueVid n0 = mnode0.MakeVertex();
assert(mnode0.VertexCount() == 1);
ShardedStorage mnode1{1};
mnode1.PlaceVertex(n0, mnode0.GetVertex(n0));
mnode0.RemoveVertex(n0);
assert(mnode0.VertexCount() == 0);
assert(mnode1.VertexCount() == 1);
mnode1.MakeVertex();
assert(mnode1.VertexCount() == 2);
assert(std::distance(mnode1.begin(), mnode1.end()) == 2);
}
void test_distributed() {
Distributed d;
assert(d.MnodeCount() == 0);
auto w0 = d.AddMnode();
assert(d.MnodeCount() == 1);
auto w1 = d.AddMnode();
assert(d.MnodeCount() == 2);
GlobalVertAddress n0 = d.MakeVertex(w0);
assert(d.GetMnode(w0).VertexCount() == 1);
GlobalVertAddress n1 = d.MakeVertex(w1);
assert(d.GetVertex(n0).edges_out().size() == 0);
assert(d.GetVertex(n0).edges_in().size() == 0);
assert(d.GetVertex(n1).edges_out().size() == 0);
assert(d.GetVertex(n1).edges_in().size() == 0);
d.MakeEdge(n0, n1);
assert(d.GetVertex(n0).edges_out().size() == 1);
assert(d.GetVertex(n0).edges_in().size() == 0);
assert(d.GetVertex(n1).edges_out().size() == 0);
assert(d.GetVertex(n1).edges_in().size() == 1);
}
int main() {
test_global_id();
test_global_address();
test_mnode();
test_distributed();
std::cout << "All tests passed" << std::endl;
}

389
experimental/distributed/tests/local_memgraph.cpp
View File

@ -1,389 +0,0 @@
// This is a deprecated implementation! It is using the deprecated AwaitEvent, I'm changing it to use OnEvent. WIP
// #include <atomic>
// #include <chrono>
// #include <cstdlib>
// #include <iostream>
// #include <string>
// #include <thread>
// #include <vector>
// #include "reactors_distributed.hpp"
// const int NUM_WORKERS = 1;
// class Txn : public ReturnAddressMsg {
// public:
// Txn(std::string reactor, std::string channel, int64_t id) : ReturnAddressMsg(reactor, channel), id_(id) {}
// int64_t id() const { return id_; }
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<ReturnAddressMsg>(this), id_);
// }
// private:
// int64_t id_;
// };
// class CreateNodeTxn : public Txn {
// public:
// CreateNodeTxn(std::string reactor, std::string channel, int64_t id) : Txn(reactor, channel, id) {}
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<Txn>(this));
// }
// };
// class CountNodesTxn : public Txn {
// public:
// CountNodesTxn(std::string reactor, std::string channel, int64_t id) : Txn(reactor, channel, id) {}
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<Txn>(this));
// }
// };
// class CountNodesTxnResult : public Message {
// public:
// CountNodesTxnResult(int64_t count) : count_(count) {}
// int64_t count() const { return count_; }
// template <class Archive>
// void serialize(Archive &archive) {
// archive(count_);
// }
// private:
// int64_t count_;
// };
// class CommitRequest : public ReturnAddressMsg {
// public:
// CommitRequest(std::string reactor, std::string channel, int64_t worker_id)
// : ReturnAddressMsg(reactor, channel), worker_id_(worker_id) {}
// int64_t worker_id() { return worker_id_; }
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<ReturnAddressMsg>(this), worker_id_);
// }
// private:
// int64_t worker_id_;
// };
// class AbortRequest : public ReturnAddressMsg {
// public:
// AbortRequest(std::string reactor, std::string channel, int64_t worker_id)
// : ReturnAddressMsg(reactor, channel), worker_id_(worker_id) {}
// int64_t worker_id() { return worker_id_; }
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<ReturnAddressMsg>(this), worker_id_);
// }
// private:
// int64_t worker_id_;
// };
// class CommitDirective : public Message {
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<Message>(this));
// }
// };
// class AbortDirective : public Message {
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<Message>(this));
// }
// };
// class Query : public Message {
// public:
// Query(std::string query) : Message(), query_(query) {}
// std::string query() const { return query_; }
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<Message>(this), query_);
// }
// private:
// std::string query_;
// };
// class Quit : public Message {
// template <class Archive>
// void serialize(Archive &archive) {
// archive(cereal::base_class<Message>(this));
// }
// };
// class Master : public Reactor {
// public:
// Master(System *system, std::string name) : Reactor(system, name), next_xid_(1) {}
// virtual void Run() {
// auto stream = main_.first;
// FindWorkers();
// std::cout << "Master is active" << std::endl;
// while (true) {
// auto m = stream->AwaitEvent();
// if (Query *query = dynamic_cast<Query *>(m.get())) {
// ProcessQuery(query);
// break; // process only the first query
// } else if (ReturnAddressMsg *msg = dynamic_cast<ReturnAddressMsg *>(m.get())) {
// std::cout << "ReturnAddressMsg received!" << std::endl;
// std::cout << " Address: " << msg->Address() << std::endl;
// std::cout << " Port: " << msg->Port() << std::endl;
// std::cout << " Reactor: " << msg->ReactorName() << std::endl;
// std::cout << " Channel: " << msg->ChannelName() << std::endl;
// } else {
// std::cerr << "unknown message\n";
// exit(1);
// }
// }
// stream->OnEvent<Message>([this](const Message &msg, const Subscription& subscription) {
// std::cout << "Processing Query via Callback" << std::endl;
// const Query &query =
// dynamic_cast<const Query &>(msg); // exception bad_cast
// ProcessQuery(&query);
// subscription.Unsubscribe();
// });
// }
// private:
// void ProcessQuery(const Query *query) {
// if (query->query() == "create node") {
// PerformCreateNode();
// } else if (query->query() == "count nodes") {
// PerformCountNodes();
// } else {
// std::cout << "got query: " << query->query() << std::endl;
// }
// }
// void PerformCreateNode() {
// int worker_id = rand() % NUM_WORKERS;
// int64_t xid = GetTransactionId();
// std::string txn_channel_name = GetTxnName(xid);
// auto channel = Open(txn_channel_name);
// auto stream = channel.first;
// channels_[worker_id]->Send<CreateNodeTxn>("master", "main", xid);
// auto m = stream->AwaitEvent();
// if (CommitRequest *req = dynamic_cast<CommitRequest *>(m.get())) {
// req->GetReturnChannelWriter(system_)->Send<CommitDirective>();
// } else if (AbortRequest *req = dynamic_cast<AbortRequest *>(m.get())) {
// req->GetReturnChannelWriter(system_)->Send<AbortDirective>();
// } else {
// std::cerr << "unknown message\n";
// exit(1);
// }
// CloseChannel(txn_channel_name);
// }
// void PerformCountNodes() {
// int64_t xid = GetTransactionId();
// std::string txn_channel_name = GetTxnName(xid);
// auto channel = Open(txn_channel_name);
// auto stream = channel.first;
// for (int w_id = 0; w_id < NUM_WORKERS; ++w_id)
// channels_[w_id]->Send<CountNodesTxn>("master", "main", xid);
// std::vector<std::shared_ptr<Channel>> txn_channels;
// txn_channels.resize(NUM_WORKERS, nullptr);
// bool commit = true;
// for (int responds = 0; responds < NUM_WORKERS; ++responds) {
// auto m = stream->AwaitEvent();
// if (CommitRequest *req = dynamic_cast<CommitRequest *>(m.get())) {
// txn_channels[req->worker_id()] = req->GetReturnChannelWriter(system_);
// commit &= true;
// } else if (AbortRequest *req = dynamic_cast<AbortRequest *>(m.get())) {
// txn_channels[req->worker_id()] = req->GetReturnChannelWriter(system_);
// commit = false;
// } else {
// std::cerr << "unknown message\n";
// exit(1);
// }
// }
// if (commit) {
// for (int w_id = 0; w_id < NUM_WORKERS; ++w_id)
// txn_channels[w_id]->Send<CommitDirective>();
// } else {
// for (int w_id = 0; w_id < NUM_WORKERS; ++w_id)
// txn_channels[w_id]->Send<AbortDirective>();
// }
// int64_t count = 0;
// for (int responds = 0; responds < NUM_WORKERS; ++responds) {
// auto m = stream->AwaitEvent();
// if (CountNodesTxnResult *cnt =
// dynamic_cast<CountNodesTxnResult *>(m.get())) {
// count += cnt->count();
// } else {
// std::cerr << "unknown message\n";
// exit(1);
// }
// }
// CloseChannel(txn_channel_name);
// std::cout << "graph has " << count << " vertices" << std::endl;
// }
// int64_t GetTransactionId() { return next_xid_++; }
// std::string GetWorkerName(int worker_id) {
// return "worker" + std::to_string(worker_id);
// }
// std::string GetTxnName(int txn_id) { return "txn" + std::to_string(txn_id); }
// void FindWorkers() {
// channels_.resize(NUM_WORKERS, nullptr);
// int workers_found = 0;
// while (workers_found < NUM_WORKERS) {
// for (int64_t w_id = 0; w_id < NUM_WORKERS; ++w_id) {
// if (channels_[w_id] == nullptr) {
// // TODO: Resolve worker channel using the network service.
// channels_[w_id] = system_->FindChannel(GetWorkerName(w_id), "main");
// if (channels_[w_id] != nullptr) ++workers_found;
// }
// }
// if (workers_found < NUM_WORKERS)
// std::this_thread::sleep_for(std::chrono::seconds(1));
// }
// }
// // TODO: Why is master atomic, it should be unique?
// std::atomic<int64_t> next_xid_;
// std::vector<std::shared_ptr<Channel>> channels_;
// };
// class Worker : public Reactor {
// public:
// Worker(System *system, std::string name, int64_t id) : Reactor(system, name),
// worker_id_(id) {}
// virtual void Run() {
// std::cout << "worker " << worker_id_ << " is active" << std::endl;
// auto stream = main_.first;
// FindMaster();
// while (true) {
// auto m = stream->AwaitEvent();
// if (Txn *txn = dynamic_cast<Txn *>(m.get())) {
// HandleTransaction(txn);
// } else {
// std::cerr << "unknown message\n";
// exit(1);
// }
// }
// }
// private:
// void HandleTransaction(Txn *txn) {
// if (CreateNodeTxn *create_txn = dynamic_cast<CreateNodeTxn *>(txn)) {
// HandleCreateNode(create_txn);
// } else if (CountNodesTxn *cnt_txn = dynamic_cast<CountNodesTxn *>(txn)) {
// HandleCountNodes(cnt_txn);
// } else {
// std::cerr << "unknown transaction\n";
// exit(1);
// }
// }
// void HandleCreateNode(CreateNodeTxn *txn) {
// auto channel = Open(GetTxnChannelName(txn->id()));
// auto stream = channel.first;
// auto masterChannel = txn->GetReturnChannelWriter(system_);
// // TODO: Do the actual commit.
// masterChannel->Send<CommitRequest>("master", "main", worker_id_);
// auto m = stream->AwaitEvent();
// if (dynamic_cast<CommitDirective *>(m.get())) {
// // TODO: storage_.CreateNode();
// } else if (dynamic_cast<AbortDirective *>(m.get())) {
// // TODO: Rollback.
// } else {
// std::cerr << "unknown message\n";
// exit(1);
// }
// CloseChannel(GetTxnChannelName(txn->id()));
// }
// void HandleCountNodes(CountNodesTxn *txn) {
// auto channel = Open(GetTxnChannelName(txn->id()));
// auto stream = channel.first;
// auto masterChannel = txn->GetReturnChannelWriter(system_);
// // TODO: Fix this hack -- use the storage.
// int num = 123;
// masterChannel->Send<CommitRequest>("master", "main", worker_id_);
// auto m = stream->AwaitEvent();
// if (dynamic_cast<CommitDirective *>(m.get())) {
// masterChannel->Send<CountNodesTxnResult>(num);
// } else if (dynamic_cast<AbortDirective *>(m.get())) {
// // send nothing
// } else {
// std::cerr << "unknown message\n";
// exit(1);
// }
// CloseChannel(GetTxnChannelName(txn->id()));
// }
// // TODO: Don't repeat code from Master.
// std::string GetTxnChannelName(int64_t transaction_id) {
// return "txn" + std::to_string(transaction_id);
// }
// void FindMaster() {
// // TODO: Replace with network service and channel resolution.
// while (!(master_channel_ = system_->FindChannel("master", "main")))
// std::this_thread::sleep_for(std::chrono::seconds(1));
// }
// std::shared_ptr<Channel> master_channel_ = nullptr;
// int worker_id_;
// };
// void ClientMain(System *system) {
// std::shared_ptr<Channel> channel = nullptr;
// // TODO: Replace this with network channel resolution.
// while (!(channel = system->FindChannel("master", "main")))
// std::this_thread::sleep_for(std::chrono::seconds(1));
// std::cout << "I/O Client Main active" << std::endl;
// bool active = true;
// while (active) {
// std::string s;
// std::getline(std::cin, s);
// if (s == "quit") {
// active = false;
// channel->Send<Quit>();
// } else {
// channel->Send<Query>(s);
// }
// }
// }
int main(int, char **) { return 0; }
// int main(int argc, char *argv[]) {
// //google::InitGoogleLogging(argv[0]);
// gflags::ParseCommandLineFlags(&argc, &argv, true);
// System system;
// system.Spawn<Master>("master");
// std::thread client(ClientMain, &system);
// for (int i = 0; i < NUM_WORKERS; ++i)
// system.Spawn<Worker>("worker" + std::to_string(i), i);
// system.AwaitShutdown();
// return 0;
// }

102
experimental/distributed/tests/network_chat.cpp
View File

@ -1,102 +0,0 @@
// command to run:
// gnome-terminal --tab -e './network_chat --port 10000 --minloglevel 2' --tab -e './network_chat --port 10001 --minloglevel 2'
#include "reactors_distributed.hpp"
class ChatMessage : public ReturnAddressMsg {
public:
ChatMessage() : ReturnAddressMsg(), message_("") {}
ChatMessage(std::string reactor, std::string channel, std::string message)
: ReturnAddressMsg(reactor, channel), message_(message) {}
std::string Message() const { return message_; }
template <class Archive>
void serialize(Archive &ar) {
ar(cereal::base_class<ReturnAddressMsg>(this), message_);
}
private:
std::string message_;
};
CEREAL_REGISTER_TYPE(ChatMessage);
class ChatACK : public ChatMessage {
public:
ChatACK() : ChatMessage() {}
ChatACK(std::string reactor, std::string channel, std::string message)
: ChatMessage(reactor, channel, message) {}
template <class Archive>
void serialize(Archive &ar) {
ar(cereal::base_class<ChatMessage>(this));
}
};
CEREAL_REGISTER_TYPE(ChatACK);
class ChatServer : public Reactor {
public:
ChatServer(std::string name)
: Reactor(name) {}
virtual void Run() {
std::cout << "ChatServer is active" << std::endl;
auto chat = Open("chat").first;
chat->OnEvent<ChatACK>([](const ChatACK& ack, const Subscription&) {
std::cout << "Received ACK from " << ack.Address() << ":"
<< ack.Port() << " -> '" << ack.Message() << "'"
<< std::endl;
});
chat->OnEvent<ChatMessage>([this](const ChatMessage& msg, const Subscription&) {
std::cout << "Received message from " << msg.Address() << ":"
<< msg.Port() << " -> '" << msg.Message() << "'"
<< std::endl;
auto channel = msg.GetReturnChannelWriter();
if (channel != nullptr) {
channel->Send<ChatACK>("server", "chat", msg.Message());
}
});
}
};
class ChatClient : public Reactor {
public:
ChatClient(std::string name)
: Reactor(name) {}
virtual void Run() {
std::cout << "ChatClient is active" << std::endl;
std::string address, message;
uint16_t port;
while (true) {
std::cout << "Enter IP, port and message to send." << std::endl;
std::cin >> address >> port >> message;
auto channel =
Distributed::GetInstance().network().Resolve(address, port, "server", "chat");
if (channel != nullptr) {
channel->Send<ChatMessage>("server", "chat", message);
} else {
std::cerr << "Couldn't resolve that server!" << std::endl;
}
}
}
};
int main(int argc, char *argv[]) {
gflags::ParseCommandLineFlags(&argc, &argv, true);
System& system = System::GetInstance();
Distributed &distributed = Distributed::GetInstance();
distributed.StartServices();
system.Spawn<ChatServer>("server");
system.Spawn<ChatClient>("client");
system.AwaitShutdown();
distributed.StopServices();
return 0;
}

14
experimental/distributed/tests/network_client.cpp
View File

@ -1,14 +0,0 @@
#include "reactors_distributed.hpp"
int main(int argc, char *argv[]) {
google::InitGoogleLogging(argv[0]);
Distributed &distributed = Distributed::GetInstance();
distributed.network().StartClient(1);
auto channel = distributed.network().Resolve("127.0.0.1", 10000, "master", "main");
std::cout << channel << std::endl;
if (channel != nullptr) {
channel->Send<ReturnAddressMsg>("master", "main");
}
distributed.network().StopClient();
return 0;
}

85
experimental/distributed/tests/spinner_test.cpp
View File

@ -1,85 +0,0 @@
#include <cstdio>
#include <cstdlib>
#include <fstream>
#include <iostream>
#include <map>
#include <string>
#include <vector>
#include "graph.hpp"
#include "spinner.hpp"
void PrintStatistics(const Distributed &distributed) {
using std::cout;
using std::endl;
for (const ShardedStorage &mnode : distributed) {
cout << " ShardedStorage " << mnode.mnid_ << ":";
cout << " #vertices = " << mnode.VertexCount();
int64_t edge_count{0};
for (const auto &gid_vertex_pair : mnode) {
edge_count += gid_vertex_pair.second.edges_in().size();
edge_count += gid_vertex_pair.second.edges_out().size();
}
cout << ", #edges = " << edge_count;
cout << ", #cuts = " << mnode.BoundaryEdgeCount() << endl;
}
}
/**
* Reads an undirected graph from file.
* - first line of the file: vertices_count, edges_count
* - next edges_count lines contain vertices that form an edge
* example:
* https://snap.stanford.edu/data/facebook_combined.txt.gz
* add number of vertices and edges in the first line of that file
*/
Distributed ReadGraph(std::string filename, int mnode_count) {
Distributed distributed(mnode_count);
std::fstream fs;
fs.open(filename, std::fstream::in);
if (fs.fail()) return distributed;
int vertex_count, edge_count;
fs >> vertex_count >> edge_count;
// assign vertices to random mnodes
std::vector<GlobalVertAddress> vertices;
for (int i = 0; i < vertex_count; ++i)
vertices.emplace_back(distributed.MakeVertex(rand() % mnode_count));
// add edges
for (int i = 0; i < edge_count; ++i) {
size_t u, v;
fs >> u >> v;
assert(u < vertices.size() && v < vertices.size());
distributed.MakeEdge(vertices[u], vertices[v]);
}
fs.close();
return distributed;
}
int main(int argc, const char *argv[]) {
srand(time(NULL));
if (argc == 1) {
std::cout << "Usage:" << std::endl;
std::cout << argv[0] << " filename partitions iterations" << std::endl;
return 0;
}
std::cout << "Memgraph spinner test " << std::endl;
std::string filename(argv[1]);
int partitions = std::max(1, atoi(argv[2]));
int iterations = std::max(1, atoi(argv[3]));
Distributed distributed = ReadGraph(filename, partitions);
PrintStatistics(distributed);
for (int iter = 0; iter < iterations; ++iter) {
spinner::PerformSpinnerStep(distributed);
std::cout << "Iteration " << iter << std::endl;
PrintStatistics(distributed);
}
return 0;
}

24
experimental/distributed/tests/start_distributed
View File

@ -1,24 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# Automatically copied to the build/ directory during Makefile (configured by cmake)
import os
command = 'gnome-terminal'
program = './distributed_test'
config_filename = 'config'
flags = '-alsologtostderr --minloglevel=2'
f = open(config_filename, 'r')
for line in f:
data = line.strip().split(' ')
my_mnid = data[0]
address = data[1]
port = data[2]
call = "{} {} --my_mnid {} --address {} --port {} --config_filename={}".format(
program, flags, my_mnid, address, port, config_filename)
command += " --tab -e '{}'".format(call)
print(command)
os.system(command)

2
experimental/lockfree_list_sirkovic/.gitignore vendored
View File

@ -1,2 +0,0 @@
*.txt
testmain

14
experimental/lockfree_list_sirkovic/bitflags.h
View File

@ -1,14 +0,0 @@
#include "stdint.h"
inline int is_marked(long long i) {
return ((i & 0x1L)>0);
}
inline long long get_unmarked(long long i) {
return i & ~0x1L;
}
inline long long get_marked(long long i) {
return i | 0x1L;
}

3
experimental/lockfree_list_sirkovic/compile
View File

@ -1,3 +0,0 @@
#!/bin/bash
g++ -std=c++1y testmain.cpp -o testmain -lpthread

212
experimental/lockfree_list_sirkovic/list.h
View File

@ -1,212 +0,0 @@
#include <iostream>
#include <algorithm>
#include <cstdio>
#include <vector>
#include <string>
#include <cstdlib>
#include "node.h"
using namespace std;
template<typename T>
struct list_t {
node_t<T> *start_ptr;
node_t<T> *end_ptr;
atomic<int> length;
list_t<T> (const T& a,const T& b) {
end_ptr = allocate_node(b, NULL);
start_ptr = allocate_node(a, end_ptr);
length.store(0);
}
node_t<T> *allocate_node(const T& data, node_t<T>* next) {
return new node_t<T>(data,next);
}
node_t<T> *find(const T& val, node_t<T> **left){
//printf("find node %d\n",val);
node_t<T> *left_next, *right, *left_next_copy;
left_next = right = NULL;
while(1) {
node_t<T> *it = start_ptr;
node_t<T> *it_next = start_ptr->next.load();
//while (get_flag(it_next) || (it->data.load() < val)) {
while (get_flag(it_next) || (it->data < val)) {
//printf("%d\n",it->data);
if (!get_flag(it_next)) {
(*left) = it;
left_next = it_next;
}
it = get_unflagged(it_next);
if (it == end_ptr) break;
//it_next = it->next.load(memory_order_relaxed);
it_next = it->next.load();
}
right = it;left_next_copy = left_next;
if (left_next == right){
//if (!get_flag(right->next.load(memory_order_relaxed)))
if (right == end_ptr || !get_flag(right->next.load()))
return right;
}
else {
if ((*left)->next.compare_exchange_strong(left_next_copy,right) == true) {
int previous = left_next->ref_count.fetch_add(-1);
previous = right->ref_count.fetch_add(1);
//if (!get_flag(right->next.load(memory_order_relaxed))) return right;
if (!get_flag(right->next.load())) return right;
}
}
}
}
int contains(const T& val) {
//printf("search node %d\n",val);
//node_t<T> *it = get_unflagged(start_ptr->next.load(memory_order_relaxed));
node_t<T> *it = get_unflagged(start_ptr->next.load());
while(it != end_ptr) {
//if (!get_flag(it->next) && it->data.load() >= val){
if (!get_flag(it->next) && it->data >= val){
//if (it->data.load() == val) return 1;
if (it->data == val) return 1;
else return 0;
}
//it = get_unflagged(it->next.load(memory_order_relaxed));
it = get_unflagged(it->next.load());
}
return 0;
}
int size() {
return length.load();
}
int add(const T& val) {
//printf("add node %d\n",val);
node_t<T> *right, *left;
right = left = NULL;
node_t<T> *new_elem = allocate_node(val, NULL);
while(1) {
right = find(val, &left);
//if (right != end_ptr && right->data.load() == val){
if (right != end_ptr && right->data == val){
return 0;
}
new_elem->next.store(right);
if (left->next.compare_exchange_strong(right,new_elem) == true) {
length.fetch_add(1);
return 1;
}
else {
}
}
}
node_t<T>* remove(const T& val) {
//printf("remove node %d\n",val);
node_t<T>* right, *left, *right_next, *tmp;
node_t<T>* left_next, *right_copy;
right = left = right_next = tmp = NULL;
while(1) {
right = find(val, &left);
left_next = left->next.load();
right_copy = right;
//if (right == end_ptr || right->data.load() != val){
if (right == end_ptr || right->data != val){
return NULL;
}
//right_next = right->next.load(memory_order_relaxed);
right_next = right->next.load();
if (!get_flag(right_next)){
node_t<T>* right_next_marked = get_flagged(right_next);
if ((right->next).compare_exchange_strong(right_next,right_next_marked)==true) {
if((left->next).compare_exchange_strong(right_copy,right_next) == false) {
tmp = find(val,&tmp);
} else {
int previous = right->ref_count.fetch_add(-1);
previous = right_next->ref_count.fetch_add(1);
}
length.fetch_add(-1);
return right;
}
}
}
}
int get_flag(node_t<T>* ptr) {
return is_marked(reinterpret_cast<long long>(ptr));
}
void mark_flag(node_t<T>* &ptr){
ptr = get_flagged(ptr);
}
void unmark_flag(node_t<T>* &ptr){
ptr = get_unflagged(ptr);
}
inline static node_t<T>* get_flagged(node_t<T>* ptr){
return reinterpret_cast<node_t<T>*>(get_marked(reinterpret_cast<long long>(ptr)));
}
inline static node_t<T>* get_unflagged(node_t<T>* ptr){
return reinterpret_cast<node_t<T>*>(get_unmarked(reinterpret_cast<long long>(ptr)));
}
struct iterator{
node_t<T>* ptr;
iterator(node_t<T>* ptr_) : ptr(ptr_) {
ptr->ref_count.fetch_add(1);
}
~iterator() {
if(ptr != NULL) ptr->ref_count.fetch_add(-1);
}
bool operator==(const iterator& other) {
return ptr == other.ptr;
}
bool operator!=(const iterator& other) {
return ptr != other.ptr;
}
iterator& operator++() {
node_t<T>* it_next = ptr->next.load(), *it = ptr, *it_next_unflagged = list_t<T>::get_unflagged(it_next);
while(it_next_unflagged != NULL && it_next != it_next_unflagged) {
it = it_next_unflagged;
it_next = it->next.load();
it_next_unflagged = list_t<T>::get_unflagged(it_next);
}
if(it_next_unflagged == NULL) {
it->ref_count.fetch_add(1);
ptr->ref_count.fetch_add(-1);
ptr = it;
} else {
it_next->ref_count.fetch_add(1);
ptr->ref_count.fetch_add(-1);
ptr = it_next;
}
return *this;
}
T& operator*() {
return ptr->data;
}
};
iterator begin(){
while(1) {
node_t<T>* it = start_ptr->next.load();
node_t<T>* it_next = it->next.load();
while(it!=end_ptr && get_flag(it->next.load())) {
it = it_next;
it_next = it_next->next.load();
}
if(it == end_ptr) return end();
return iterator(it_next);
}
}
iterator end(){
return iterator(end_ptr);
}
};

26
experimental/lockfree_list_sirkovic/node.h
View File

@ -1,26 +0,0 @@
#include <iostream>
#include <algorithm>
#include <cstdio>
#include <vector>
#include <string>
#include <cstdlib>
#include "bitflags.h"
#include <atomic>
using namespace std;
template <typename T>
struct node_t {
//atomic<T> data;
T data;
atomic<node_t<T>* > next;
atomic<int> ref_count;
long long timestamp;
node_t<T> (const T& data_, node_t<T>* next_) {
//data.store(data_);
timestamp = -1;
data = data_;
next.store(next_);
ref_count.store(1);
}
};

3
experimental/lockfree_list_sirkovic/run
View File

@ -1,3 +0,0 @@
#!/bin/bash
./testmain

269
experimental/lockfree_list_sirkovic/testmain.cpp
View File

@ -1,269 +0,0 @@
#include <iostream>
#include <algorithm>
#include "pthread.h"
#include <cstdio>
#include <vector>
#include <string>
#include <cstdlib>
#include "list.h"
#include <ctime>
#include <sys/time.h>
#include <climits>
using namespace std;
#define FOR(i,a,b) for(int i=(a);i<(b);++i)
#define MAXP 128
#define SCAN_ITER 32
double rand_double(){
return ((double) rand())/RAND_MAX;
}
double rand_int(int max_int){
return rand_double()*max_int;
}
struct timestamping{
vector<long long> stamps;
vector<long long> finished;
atomic<long long> maxTS;
int n;
timestamping(int n_) {
n = n_;
stamps.resize(n,0);
finished.resize(n,0);
maxTS.store(0);
}
long long get_minimal(int id) {
long long sol = INT_MAX;
FOR(i,0,n)
if(finished[i] == 0 && i != id) sol = min(stamps[i],sol);
return sol;
}
long long increase_timestamp(int id) {
long long newTS = maxTS.fetch_add(1) + 1;
stamps[id] = newTS;
return newTS;
}
void mark_finished(int id) {
finished[id] = 1;
}
};
struct thread_data {
int id;
int op_cnt, max_int;
int add_cnt, remove_cnt, find_cnt, total_op;
double find_threshold, add_threshold;
int* values;
int* tasks;
timestamping* timestamps;
int max_buffer_size;
int reclaimed_cnt;
vector<node_t<int>* > buffer;
list_t<int>* list;
thread_data(list_t<int>* ptr, int id_, timestamping* timestamps_, int max_buffer_size_,
int op_cnt_, int max_int_, double find_=0.6, double add_ = 0.8) {
list = ptr;
id = id_;
timestamps = timestamps_;
max_buffer_size = max_buffer_size_;
reclaimed_cnt = 0;
op_cnt = op_cnt_;
max_int = max_int_;
add_cnt = 0;
remove_cnt = 0;
find_cnt = 0;
total_op = 0;
find_threshold = find_;
add_threshold = add_;
init_tasks();
}
void init_tasks() {
values = (int *)malloc(op_cnt*sizeof(int));
tasks = (int *)malloc(op_cnt*sizeof(int));
FOR(i,0,op_cnt) {
double x = rand_double();
int n = rand_int(max_int);
values[i] = n;
if( x < find_threshold )
tasks[i] = 0;
else if (x < add_threshold)
tasks[i] = 1;
else
tasks[i] = 2;
}
}
};
void *print_info( void* data) {
thread_data* ptr = (thread_data*)data;
cout << "Thread " << ptr->id << endl;
cout << "Read: " << ptr->find_cnt << " Add: " << ptr->add_cnt << " Remove: " << ptr->remove_cnt << endl;
cout << "Deallocated: " << ptr->reclaimed_cnt << " To be freed: " << ptr->buffer.size() << endl;
return NULL;
}
void *print_set( void* ptr ){
thread_data* data = (thread_data*)ptr;
FILE *out = fopen("out.txt","w+");
FOR(i,0,10) {
list_t<int>::iterator it = data->list->begin();
list_t<int>::iterator endit = data->list->end();
while(it!= endit) {
fprintf(out,"%d ",*it);
++it;
}
fprintf(out,"\n");
fflush(out);
}
fclose(out);
return NULL;
}
void scan_buffer(void *ptr){
thread_data* data = (thread_data*)ptr;
node_t<int>* tmp;
int min_timestamp = data->timestamps->get_minimal(data->id);
printf("Memory reclamation process %d Min timestamp %lld Size %d\n",data->id,min_timestamp,data->buffer.size());
vector<node_t<int>* > tmp_buffer;
FOR(i,0,data->buffer.size()) {
int ts = (data->buffer[i])->timestamp;
node_t<int>* next = list_t<int>::get_unflagged(data->buffer[i]->next.load());
//printf("Deleting: %d %d %d %d %lld %d\n",data->id,i,ts,data->buffer[i]->data,(long long)data->buffer[i],data->buffer[i]->ref_count.load());
if (ts < min_timestamp && data->buffer[i]->ref_count.load() == 0) {
next->ref_count.fetch_add(-1);
free(data->buffer[i]);
++(data->reclaimed_cnt);
}
else {
tmp_buffer.push_back(data->buffer[i]);
}
}
data->buffer = tmp_buffer;
}
void *test(void *ptr){
thread_data* data = (thread_data*)ptr;
int opcnt = data->op_cnt;
int maxint = data->max_int;
int id = data->id;
list_t<int>* list = data->list;
FOR(i,0,opcnt) {
/*
double x = rand_double();
int n = rand_int(maxint);
//cout << x << " " << n << endl;
if (x < data->find_threshold) {
//cout << 0 << endl;
list->contains(n);
++(data->find_cnt);
} else if(x < data->add_threshold) {
//cout << 1 << endl;
if(list->add(n)) {
++(data->add_cnt);
}
} else {
//cout << 2 << endl;
if(list->remove(n)) {
++(data->remove_cnt);
}
}
++(data->total_op);
*/
int n = data->values[i];
int op = data->tasks[i];
long long ts = data->timestamps->increase_timestamp(id);
//printf("Time: %lld Process: %d Operation count: %d Operation type: %d Value: %d\n",ts,id,i,op,n);
if (op == 0) {
list->contains(n);
++(data->find_cnt);
} else if(op == 1) {
if(list->add(n)) {
++(data->add_cnt);
}
} else {
node_t<int>* node_ptr = list->remove(n);
if(((long long) node_ptr)%4 !=0 ){
printf("oslo u pm\n"); fflush(stdout);
exit(0);
}
if(node_ptr != NULL ) {
node_ptr->timestamp = data->timestamps->maxTS.load() + 1;
data->buffer.push_back(node_ptr);
//printf("Process %d at time %d added reclamation node: %d %lld\n",id,node_ptr->timestamp,node_ptr->data,(long long)node_ptr);
++(data->remove_cnt);
}
}
fflush(stdout);
if( i % SCAN_ITER == 0 && data->buffer.size() >= data->max_buffer_size )
scan_buffer(ptr);
}
data->timestamps->mark_finished(id);
return NULL;
}
int main(int argc, char **argv){
int P = 1;
int op_cnt = 100;
if(argc > 1){
sscanf(argv[1],"%d",&P);
sscanf(argv[2],"%d",&op_cnt);
}
int max_int = 2048;
int limit = 1e9;
int initial = max_int/2;
int max_buffer_size = max_int/16;
struct timeval start,end;
timestamping timestamps(P);
vector<pthread_t> threads(P+1);
vector<thread_data> data;
list_t<int> *list = new list_t<int>(-limit,limit);
cout << "constructed list" << endl;
FOR(i,0,initial) {
list->add(i);
}
cout << "initialized list elements" << endl;
FOR(i,0,P) data.push_back(thread_data(list,i,&timestamps,max_buffer_size,op_cnt,max_int));
cout << "created thread inputs" << endl;
gettimeofday(&start,NULL);
FOR(i,0,P) pthread_create(&threads[i],NULL,test,((void *)(&data[i])));
pthread_create(&threads[P],NULL,print_set,((void *)(&data[0])));
cout << "created threads" << endl;
FOR(i,0,P+1) pthread_join(threads[i],NULL);
gettimeofday(&end,NULL);
cout << "execution finished" << endl;
FOR(i,0,P) print_info((void*)(&data[i]));
int exp_len = initial;
FOR(i,0,P)
exp_len += (data[i].add_cnt - data[i].remove_cnt);
uint64_t duration = (end.tv_sec*(uint64_t)1000000 + end.tv_usec) - (start.tv_sec*(uint64_t)1000000 + start.tv_usec);
cout << "Actual length: " << list->length.load() << endl;
cout << "Expected length: " << exp_len << endl;
cout << "Time(s): " << duration/1000000.0 << endl;
return 0;
}

1
experimental/memory_tracker/build.sh
View File

@ -1 +0,0 @@
g++ -std=c++11 macro_override.h test_macro.cc

48
experimental/memory_tracker/macro_override.h
View File

@ -1,48 +0,0 @@
#include <stdlib.h>
#include <memory>
#include <map>
#include <iostream>
size_t ALLOCATED = 0;
std::map<void*, size_t> TRACKER;
void* operator new(size_t size, const char* filename, int line_number) {
std::cout << filename << ":" << line_number << " Allocating" << size
<< "bytes." << std::endl;
void* block = malloc(size);
TRACKER[block] = size;
ALLOCATED += size;
return block;
}
void operator delete(void* block) {
TRACKER[block] = 0;
free(block);
}
void *operator new[](size_t size, const char* filename, int line_number) {
std::cout << filename << ":" << line_number << " Allocating" << size
<< "bytes." << std::endl;
void* block = malloc(size);
TRACKER[block] = size;
ALLOCATED += size;
return block;
}
void operator delete[] (void* block) {
TRACKER[block] = 0;
free(block);
}
void print_memory() {
std::cout << "Total bytes allocated: " << ALLOCATED << std::endl;
for (const auto& el : TRACKER) {
std::cout << el.first << " " << el.second << std::endl;
}
}
#define new new (__FILE__, __LINE__)

1
experimental/memory_tracker/run_with_tcmalloc.sh
View File

@ -1 +0,0 @@
MALLOCSTATS=1 HEAPPROFILE=$(pwd)/profiling_tcmalloc.hprof LD_PRELOAD=/usr/lib/libtcmalloc.so ./a.out

9
experimental/memory_tracker/tcmalloc_demo.cc
View File

@ -1,9 +0,0 @@
#include <iostream>
int main() {
int* array = new int[16];
for (int i = 0; i < 16; i++)
*(array+i) = i;
std::cout << *(array+5);
return 0;
}

16
experimental/memory_tracker/test_macro.cc
View File

@ -1,16 +0,0 @@
#include "macro_override.h"
int main () {
int* num = new int;
delete num;
print_memory();
ALLOCATED = 0;
int* nums = new int[16];
delete nums;
print_memory();
return 0;
}

2
experimental/pro_compiler/.gitignore vendored
View File

@ -1,2 +0,0 @@
target

9
experimental/pro_compiler/build.sbt
View File

@ -1,9 +0,0 @@
lazy val root = (project in file(".")).settings(
name := "cypher-compiler",
scalaVersion := "2.12.1",
version := "0.1",
libraryDependencies ++= Seq(
"org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.5"
)
)

434
experimental/pro_compiler/cypher.ebnf
View File

@ -1,434 +0,0 @@
(*
* Copyright (c) 2015-2016 "Neo Technology,"
* Network Engine for Objects in Lund AB [http://neotechnology.com]
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*)
Cypher = [SP], Statement, [[SP], ';'], [SP] ;
Statement = Query ;
Query = RegularQuery ;
RegularQuery = SingleQuery, { [SP], Union } ;
SingleQuery = Clause, { [SP], Clause } ;
Union = ((U,N,I,O,N), SP, (A,L,L), [SP], SingleQuery)
| ((U,N,I,O,N), [SP], SingleQuery)
;
Clause = Match
| Unwind
| Merge
| Create
| Set
| Delete
| Remove
| With
| Return
;
Match = [(O,P,T,I,O,N,A,L), SP], (M,A,T,C,H), [SP], Pattern, [[SP], Where] ;
Unwind = (U,N,W,I,N,D), [SP], Expression, SP, (A,S), SP, Variable ;
Merge = (M,E,R,G,E), [SP], PatternPart, { SP, MergeAction } ;
MergeAction = ((O,N), SP, (M,A,T,C,H), SP, Set)
| ((O,N), SP, (C,R,E,A,T,E), SP, Set)
;
Create = (C,R,E,A,T,E), [SP], Pattern ;
Set = (S,E,T), [SP], SetItem, { ',', SetItem } ;
SetItem = (PropertyExpression, [SP], '=', [SP], Expression)
| (Variable, [SP], '=', [SP], Expression)
| (Variable, [SP], '+=', [SP], Expression)
| (Variable, [SP], NodeLabels)
;
Delete = [(D,E,T,A,C,H), SP], (D,E,L,E,T,E), [SP], Expression, { [SP], ',', [SP], Expression } ;
Remove = (R,E,M,O,V,E), SP, RemoveItem, { [SP], ',', [SP], RemoveItem } ;
RemoveItem = (Variable, NodeLabels)
| PropertyExpression
;
With = (W,I,T,H), [[SP], (D,I,S,T,I,N,C,T)], SP, ReturnBody, [[SP], Where] ;
Return = (R,E,T,U,R,N), [[SP], (D,I,S,T,I,N,C,T)], SP, ReturnBody ;
ReturnBody = ReturnItems, [SP, Order], [SP, Skip], [SP, Limit] ;
ReturnItems = ('*', { [SP], ',', [SP], ReturnItem })
| (ReturnItem, { [SP], ',', [SP], ReturnItem })
;
ReturnItem = (Expression, SP, (A,S), SP, Variable)
| Expression
;
Order = (O,R,D,E,R), SP, (B,Y), SP, SortItem, { ',', [SP], SortItem } ;
Skip = (S,K,I,P), SP, Expression ;
Limit = (L,I,M,I,T), SP, Expression ;
SortItem = Expression, [[SP], ((A,S,C,E,N,D,I,N,G) | (A,S,C) | (D,E,S,C,E,N,D,I,N,G) | (D,E,S,C))] ;
Where = (W,H,E,R,E), SP, Expression ;
Pattern = PatternPart, { [SP], ',', [SP], PatternPart } ;
PatternPart = (Variable, [SP], '=', [SP], AnonymousPatternPart)
| AnonymousPatternPart
;
AnonymousPatternPart = PatternElement ;
PatternElement = (NodePattern, { [SP], PatternElementChain })
| ('(', PatternElement, ')')
;
NodePattern = '(', [SP], [Variable, [SP]], [NodeLabels, [SP]], [Properties, [SP]], ')' ;
PatternElementChain = RelationshipPattern, [SP], NodePattern ;
RelationshipPattern = (LeftArrowHead, [SP], Dash, [SP], [RelationshipDetail], [SP], Dash, [SP], RightArrowHead)
| (LeftArrowHead, [SP], Dash, [SP], [RelationshipDetail], [SP], Dash)
| (Dash, [SP], [RelationshipDetail], [SP], Dash, [SP], RightArrowHead)
| (Dash, [SP], [RelationshipDetail], [SP], Dash)
;
RelationshipDetail = '[', [SP], [Variable, [SP]], [RelationshipTypes, [SP]], [RangeLiteral], [Properties, [SP]], ']' ;
Properties = MapLiteral
| Parameter
;
RelationshipTypes = ':', [SP], RelTypeName, { [SP], '|', [':'], [SP], RelTypeName } ;
NodeLabels = NodeLabel, { [SP], NodeLabel } ;
NodeLabel = ':', [SP], LabelName ;
RangeLiteral = '*', [SP], [IntegerLiteral, [SP]], ['..', [SP], [IntegerLiteral, [SP]]] ;
LabelName = SymbolicName ;
RelTypeName = SymbolicName ;
Expression = Expression12 ;
Expression12 = Expression11, { SP, (O,R), SP, Expression11 } ;
Expression11 = Expression10, { SP, (X,O,R), SP, Expression10 } ;
Expression10 = Expression9, { SP, (A,N,D), SP, Expression9 } ;
Expression9 = { (N,O,T), [SP] }, Expression8 ;
Expression8 = Expression7, { [SP], PartialComparisonExpression } ;
Expression7 = Expression6, { ([SP], '+', [SP], Expression6) | ([SP], '-', [SP], Expression6) } ;
Expression6 = Expression5, { ([SP], '*', [SP], Expression5) | ([SP], '/', [SP], Expression5) | ([SP], '%', [SP], Expression5) } ;
Expression5 = Expression4, { [SP], '^', [SP], Expression4 } ;
Expression4 = { ('+' | '-'), [SP] }, Expression3 ;
Expression3 = Expression2, { ([SP], '[', Expression, ']') | ([SP], '[', [Expression], '..', [Expression], ']') | ((([SP], '=~') | (SP, (I,N)) | (SP, (S,T,A,R,T,S), SP, (W,I,T,H)) | (SP, (E,N,D,S), SP, (W,I,T,H)) | (SP, (C,O,N,T,A,I,N,S))), [SP], Expression2) | (SP, (I,S), SP, (N,U,L,L)) | (SP, (I,S), SP, (N,O,T), SP, (N,U,L,L)) } ;
Expression2 = Atom, { [SP], (PropertyLookup | NodeLabels) } ;
Atom = Literal
| Parameter
| ((C,O,U,N,T), [SP], '(', [SP], '*', [SP], ')')
| ListComprehension
| PatternComprehension
| ((F,I,L,T,E,R), [SP], '(', [SP], FilterExpression, [SP], ')')
| ((E,X,T,R,A,C,T), [SP], '(', [SP], FilterExpression, [SP], [[SP], '|', Expression], ')')
| ((A,L,L), [SP], '(', [SP], FilterExpression, [SP], ')')
| ((A,N,Y), [SP], '(', [SP], FilterExpression, [SP], ')')
| ((N,O,N,E), [SP], '(', [SP], FilterExpression, [SP], ')')
| ((S,I,N,G,L,E), [SP], '(', [SP], FilterExpression, [SP], ')')
| RelationshipsPattern
| ParenthesizedExpression
| FunctionInvocation
| Variable
;
Literal = NumberLiteral
| StringLiteral
| BooleanLiteral
| (N,U,L,L)
| MapLiteral
| ListLiteral
;
BooleanLiteral = (T,R,U,E)
| (F,A,L,S,E)
;
ListLiteral = '[', [SP], [Expression, [SP], { ',', [SP], Expression, [SP] }], ']' ;
PartialComparisonExpression = ('=', [SP], Expression7)
| ('<>', [SP], Expression7)
| ('!=', [SP], Expression7)
| ('<', [SP], Expression7)
| ('>', [SP], Expression7)
| ('<=', [SP], Expression7)
| ('>=', [SP], Expression7)
;
ParenthesizedExpression = '(', [SP], Expression, [SP], ')' ;
RelationshipsPattern = NodePattern, { [SP], PatternElementChain }- ;
FilterExpression = IdInColl, [[SP], Where] ;
IdInColl = Variable, SP, (I,N), SP, Expression ;
FunctionInvocation = FunctionName, [SP], '(', [SP], [(D,I,S,T,I,N,C,T), [SP]], [Expression, [SP], { ',', [SP], Expression, [SP] }], ')' ;
FunctionName = SymbolicName ;
ListComprehension = '[', [SP], FilterExpression, [[SP], '|', [SP], Expression], [SP], ']' ;
PatternComprehension = '[', [SP], [Variable, [SP], '=', [SP]], RelationshipsPattern, [SP], [(W,H,E,R,E), [SP], Expression, [SP]], '|', [SP], Expression, [SP], ']' ;
PropertyLookup = '.', [SP], (PropertyKeyName) ;
Variable = SymbolicName ;
StringLiteral = ('"', { ANY - ('"' | '\') | EscapedChar }, '"')
| ("'", { ANY - ("'" | '\') | EscapedChar }, "'")
;
EscapedChar = '\', ('\' | "'" | '"' | (B) | (F) | (N) | (R) | (T) | ((U), 4 * HexDigit) | ((U), 8 * HexDigit)) ;
NumberLiteral = DoubleLiteral
| IntegerLiteral
;
MapLiteral = '{', [SP], [PropertyKeyName, [SP], ':', [SP], Expression, [SP], { ',', [SP], PropertyKeyName, [SP], ':', [SP], Expression, [SP] }], '}' ;
Parameter = '$', (SymbolicName | DecimalInteger) ;
PropertyExpression = Atom, { [SP], PropertyLookup }- ;
PropertyKeyName = SymbolicName ;
IntegerLiteral = HexInteger
| OctalInteger
| DecimalInteger
;
HexInteger = '0x', { HexDigit }- ;
DecimalInteger = ZeroDigit
| (NonZeroDigit, { Digit })
;
OctalInteger = ZeroDigit, { OctDigit }- ;
HexLetter = (A)
| (B)
| (C)
| (D)
| (E)
| (F)
;
HexDigit = Digit
| HexLetter
;
Digit = ZeroDigit
| NonZeroDigit
;
NonZeroDigit = NonZeroOctDigit
| '8'
| '9'
;
NonZeroOctDigit = '1'
| '2'
| '3'
| '4'
| '5'
| '6'
| '7'
;
OctDigit = ZeroDigit
| NonZeroOctDigit
;
ZeroDigit = '0' ;
DoubleLiteral = ExponentDecimalReal
| RegularDecimalReal
;
ExponentDecimalReal = ({ Digit }- | ({ Digit }-, '.', { Digit }-) | ('.', { Digit }-)), ((E) | (E)), ['-'], { Digit }- ;
RegularDecimalReal = { Digit }, '.', { Digit }- ;
SymbolicName = UnescapedSymbolicName
| EscapedSymbolicName
;
UnescapedSymbolicName = IdentifierStart, { IdentifierPart } ;
(* Based on the unicode identifier and pattern syntax
* (http://www.unicode.org/reports/tr31/)
* And extended with a few characters.
*)IdentifierStart = ID_Start
| '_'
| '‿'
| '⁀'
| '⁔'
| '︳'
| '︴'
| '﹍'
| '﹎'
| '﹏'
| '＿'
;
(* Based on the unicode identifier and pattern syntax
* (http://www.unicode.org/reports/tr31/)
* And extended with a few characters.
*)IdentifierPart = ID_Continue
| Sc
;
(* Any character except "`", enclosed within `backticks`. Backticks are escaped with double backticks. *)EscapedSymbolicName = { '`', { ANY - ('`') }, '`' }- ;
SP = { whitespace }- ;
whitespace = SPACE
| TAB
| LF
| VT
| FF
| CR
| FS
| GS
| RS
| US
| ' '
| '᠎'
| ' '
| ' '
| ' '
| ' '
| ' '
| ' '
| ' '
| ' '
| ' '
| ' '
| '
'
| '
'
| ' '
| '　'
| ' '
| ' '
| ' '
| Comment
;
Comment = ('/*', { ANY - ('*') | ('*', ANY - ('/')) }, '*/')
| ('//', { ANY - (LF | CR) }, [CR], (LF | EOI))
;
LeftArrowHead = '<'
| '⟨'
| '〈'
| '﹤'
| '＜'
;
RightArrowHead = '>'
| '⟩'
| '〉'
| '﹥'
| '＞'
;
Dash = '-'
| '­'
| '‐'
| '‑'
| '‒'
| '–'
| '—'
| '―'
| '−'
| '﹘'
| '﹣'
| '－'
;
A = 'A' | 'a' ;
B = 'B' | 'b' ;
C = 'C' | 'c' ;
D = 'D' | 'd' ;
E = 'E' | 'e' ;
F = 'F' | 'f' ;
G = 'G' | 'g' ;
H = 'H' | 'h' ;
I = 'I' | 'i' ;
K = 'K' | 'k' ;
L = 'L' | 'l' ;
M = 'M' | 'm' ;
N = 'N' | 'n' ;
O = 'O' | 'o' ;
P = 'P' | 'p' ;
R = 'R' | 'r' ;
S = 'S' | 's' ;
T = 'T' | 't' ;
U = 'U' | 'u' ;
V = 'V' | 'v' ;
W = 'W' | 'w' ;
X = 'X' | 'x' ;
Y = 'Y' | 'y' ;

33
experimental/pro_compiler/sbt
View File

@ -1,33 +0,0 @@
#!/bin/bash
SCRIPT_PATH=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
SBT_OPTS="-Xms768M -Xmx3072M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
if [ "$JENKINS_NIGHTLY_BUILD" == "true" ]; then
SBT_ARGS="-Dsbt.log.noformat=true"
fi
if hash cygpath.exe 2>/dev/null; then
echo "Using cygpath to convert path to SBT."
SBT_CYG_JAR_PATH=`realpath "${SCRIPT_PATH}/sbt-launch.jar"`
SBT_JAR_PATH=`cygpath.exe -w "${SBT_CYG_JAR_PATH}"`
echo "Using Windows path: ${SBT_JAR_PATH}"
SBT_ARGS="-Djline.terminal=jline.UnixTerminal -Dsbt.cygwin=true ${SBT_ARGS}"
else
echo "No cygpath, apparently not using Cygwin."
SBT_JAR_PATH="${SCRIPT_PATH}/sbt-launch.jar"
fi
SBT_CMD="java ${SBT_OPTS} ${SBT_ARGS} -jar \"${SBT_JAR_PATH}\""
if hash cygpath.exe 2>/dev/null; then
stty -icanon min 1 -echo > /dev/null 2>&1
fi
echo "Running: ${SBT_CMD}"
echo "Arguments: $@"
eval ${SBT_CMD} $@
if hash cygpath.exe 2>/dev/null; then
stty icanon echo > /dev/null 2>&1
fi

BIN
experimental/pro_compiler/sbt-launch.jar
View File

Binary file not shown.

626
experimental/pro_compiler/src/main/scala/org/cypher/ast.scala
View File

@ -1,626 +0,0 @@
package org.cypher
import scala.collection._
import scala.reflect.ClassTag
import scala.util.parsing.combinator._
// Parsing
trait CharacterStream {
def asString: String
}
object CharacterStream {
case class Default(asString: String) extends CharacterStream
}
// Query trees
trait Indexed {
private var rawIndex: Int = -1
def index: Int = rawIndex
def index_=(i: Int) = rawIndex = i
}
sealed abstract class Tree extends Indexed
case class Query(
val clauses: Seq[Clause]
) extends Tree
sealed abstract class Clause extends Tree
case class Match(
optional: Boolean,
patterns: Seq[Pattern],
where: Where
) extends Clause
case class Where(
expr: Expr
) extends Tree
sealed abstract class Expr extends Tree
case class Literal(value: Value) extends Expr
case class Ident(name: Name) extends Expr
case class Equals(left: Expr, right: Expr) extends Expr
case class PatternExpr(pattern: Pattern) extends Expr
sealed trait Value
case class Bool(x: Boolean) extends Value
case class Return(expr: Expr) extends Clause
case class Pattern(name: Name, elems: Seq[PatternElement]) extends Tree
sealed abstract class PatternElement extends Tree
case class NodePatternElement(
name: Name,
labels: Seq[Name],
properties: Properties
) extends PatternElement
case class EdgePatternElement(
left: Boolean,
right: Boolean,
name: Name,
labels: Seq[Name],
range: RangeSpec,
properties: Properties
) extends PatternElement
case class RangeSpec(
left: Option[Int],
right: Option[Int]
) extends Tree
case class Properties(contents: Map[Name, Expr]) extends Tree
case class Name private[cypher] (private val raw: String) {
def asString = raw
override def toString = s"Name($raw)"
}
// Semantic analysis
class Symbol(val name: Name, val tpe: Type) {
override def toString = s"<$name.asString: $tpe>"
}
trait Type
case class NodeType()
case class EdgeType()
class Table[T <: AnyRef: ClassTag] {
private var array = new Array[T](100)
private def ensureSize(sz: Int) {
if (array.length < sz) {
val narray = new Array[T](sz)
System.arraycopy(array, 0, narray, 0, array.length)
array = narray
}
}
def apply(idx: Indexed): T = {
array(idx.index)
}
def update(idx: Indexed, v: T) = {
ensureSize(idx.index + 1)
array(idx.index) = v
}
}
class SymbolTable() {
private val rawTable = mutable.Map[Name, Symbol]()
def apply(name: Name): Symbol = rawTable(name)
def update(name: Name, sym: Symbol) = {
assert(!rawTable.contains(name))
rawTable(name) = sym
}
def getOrCreate(name: Name): Symbol = rawTable.get(name) match {
case Some(sym) =>
sym
case None =>
val sym = new Symbol(name, null)
rawTable(name) = sym
sym
}
}
case class TypecheckedTree(
tree: Tree,
symbols: SymbolTable,
types: Table[Type]
)
class Namer {
private var count = 0
def freshName(): Name = {
count += 1
return Name(s"anon-$count")
}
}
// Data model
trait Node
trait Edge
trait Index
trait Database {
def nodeIndices(label: Name): Seq[Index]
def edgeIndices(label: Name): Seq[Index]
}
object Database {
case class Default() extends Database {
def nodeIndices(label: Name) = Nil
def edgeIndices(label: Name) = Nil
}
}
// Logical plan
sealed trait LogicalPlan {
def outputs: Seq[Symbol]
def children: Seq[LogicalPlan]
def operatorName = s"${getClass.getSimpleName}"
def prettySelf: String = s"${operatorName}"
def prettyVars: String = outputs.map(_.name.asString).mkString(", ")
def pretty: String = {
def titleWidth(plan: LogicalPlan): Int = {
var w = 1 + plan.operatorName.length
for (child <- plan.children) {
val cw = titleWidth(child) + (plan.children.length - 1) * 2
if (w < cw) w = cw
}
w
}
val w = math.max(12, titleWidth(this))
var s = ""
s += " Operator " + " " * (w - 7) + "| Variables \n"
s += "-" * (w + 3) + "+" + "-" * 30 + "\n"
def print(indent: Int, plan: LogicalPlan): Unit = {
val leftspace = "| " * indent
val title = plan.prettySelf
val rightspace = " " * (w - title.length - 2 * indent)
val rightpadspace = " " * (w - 2 * indent)
val vars = plan.prettyVars
s += s""" $leftspace+$title$rightspace | $vars\n"""
if (plan.children.nonEmpty) {
s += s""" $leftspace|$rightpadspace | \n"""
}
for ((child, idx) <- plan.children.zipWithIndex.reverse) {
print(idx, child)
}
}
print(0, this)
s
}
}
object LogicalPlan {
sealed trait Source extends LogicalPlan {
}
case class ScanAll(outputs: Seq[Symbol], patternElem: NodePatternElement)
extends Source {
def children = Nil
// def evaluate(db: Database): Stream = {
// db.iterator().filter { n =>
// patternElem match {
// case NodePatternElement(name, labels, properties) =>
// if (labels.subset(n.labels) && properties.subset(n.properties)) {
// Some(Seq(n))
// } else {
// None
// }
// }
// }
// }
}
case class SeekByNodeLabelIndex(
index: Index, name: Name, outputs: Seq[Symbol], patternElem: NodePatternElement
) extends Source {
def children = Nil
}
case class ExpandAll(input: LogicalPlan, edgePattern: EdgePatternElement, nodePattern: NodePatternElement) extends LogicalPlan {
def outputs = ???
def children = ???
}
case class ExpandInto(input: LogicalPlan) extends LogicalPlan {
def outputs = ???
def children = ???
}
case class Filter(input: LogicalPlan, expr: Expr)
extends LogicalPlan {
def outputs = input.outputs
def children = Seq(input)
}
sealed trait Sink extends LogicalPlan
case class Produce(input: LogicalPlan, outputs: Seq[Symbol], expr: Expr)
extends Sink {
def children = Seq(input)
}
}
trait Emitter {
def emit[T](symbol: Symbol, value: T): Unit
}
// Physical plan
case class PhysicalPlan(val logicalPlan: LogicalPlan) {
def execute(db: Database): Stream = {
println(logicalPlan.pretty)
???
}
}
case class Cost()
trait Stream
trait Target
// Phases
trait Phase[In, Out] {
def apply(input: In): Out
}
case class Parser(ctx: Context) extends Phase[CharacterStream, Tree] {
object CypherParser extends RegexParsers {
def query: Parser[Query] = rep(clause) ^^ {
case clauses => Query(clauses)
}
def clause: Parser[Clause] = `match` | `return` ^^ {
case clause => clause
}
def `match`: Parser[Match] = opt("optional") ~ "match" ~ patterns ~ opt(where) ^^ {
case optional ~ _ ~ p ~ Some(w) =>
Match(optional.nonEmpty, p, w)
case optional ~ _ ~ p ~ None =>
Match(optional.nonEmpty, p, Where(Literal(Bool(true))))
}
def patterns: Parser[Seq[Pattern]] = nodePattern ~ rep(edgeAndNodePattern) ^^ {
case node ~ edgeNodes =>
val ps = node +: edgeNodes.map({ case (edge, node) => Seq(edge, node) }).flatten
Seq(Pattern(ctx.namer.freshName(), ps))
}
def nodePattern: Parser[NodePatternElement] = "(" ~ ident ~ ")" ^^ {
case _ ~ ident ~ _ => NodePatternElement(ident.name, Nil, Properties(Map()))
}
def edgeAndNodePattern: Parser[(EdgePatternElement, NodePatternElement)] =
edgePattern ~ nodePattern ^^ {
case edge ~ node => (edge, node)
}
def edgePattern: Parser[EdgePatternElement] =
opt("<") ~ "--" ~ opt(">") ^^ {
case left ~ _ ~ right => EdgePatternElement(
left.nonEmpty,
right.nonEmpty,
ctx.namer.freshName(),
Nil,
RangeSpec(None, None),
Properties(Map())
)
}
def where: Parser[Where] = "where" ~ expr ^^ {
case _ ~ expr => Where(expr)
}
def `return`: Parser[Return] = "return" ~ expr ^^ {
case _ ~ expr => Return(expr)
}
def expr: Parser[Expr] = literal | ident | binary
def binary: Parser[Expr] = equals
def equals: Parser[Expr] = expr ~ "=" ~ expr ^^ {
case left ~ _ ~ right => Equals(left, right)
}
def literal: Parser[Literal] = boolean ^^ {
case x => x
}
def boolean: Parser[Literal] = ("true" | "false") ^^ {
case "true" => Literal(Bool(true))
case "false" => Literal(Bool(false))
}
def ident: Parser[Ident] = "[a-z]+".r ^^ {
case s => Ident(Name(s))
}
}
def apply(tokens: CharacterStream): Tree = {
CypherParser.parseAll(CypherParser.query, tokens.asString) match {
case CypherParser.Success(tree, _) => tree
case failure: CypherParser.NoSuccess => sys.error(failure.msg)
}
}
}
case class Typechecker() extends Phase[Tree, TypecheckedTree] {
private class Instance(val symbols: SymbolTable, val types: Table[Type]) {
def traverse(tree: Tree): Unit = tree match {
case Query(clauses) =>
for (clause <- clauses) traverse(clause)
case Match(opt, patterns, where) =>
for (pattern <- patterns) traverse(pattern)
// Ignore where for now.
case Pattern(name, elems) =>
for (elem <- elems) traverse(elem)
case NodePatternElement(name, _, _) =>
symbols.getOrCreate(name)
case _ =>
// Ignore for now.
}
def typecheck(tree: Tree) = {
traverse(tree)
TypecheckedTree(tree, symbols, types)
}
}
def apply(tree: Tree): TypecheckedTree = {
val symbols = new SymbolTable()
val types = new Table[Type]
val instance = new Instance(symbols, types)
instance.typecheck(tree)
}
}
case class LogicalPlanner(val ctx: Context)
extends Phase[TypecheckedTree, LogicalPlan] {
def apply(tree: TypecheckedTree): LogicalPlan = {
ctx.config.logicalPlanGenerator.generate(tree, ctx).next()
}
}
case class PhysicalPlanner(val ctx: Context)
extends Phase[LogicalPlan, PhysicalPlan] {
def apply(plan: LogicalPlan): PhysicalPlan = {
ctx.config.physicalPlanGenerator.generate(plan).next()
}
}
trait LogicalPlanGenerator {
def generate(typedTree: TypecheckedTree, ctx: Context): Iterator[LogicalPlan]
}
object LogicalPlanGenerator {
case class Default() extends LogicalPlanGenerator {
class Instance(val typedTree: TypecheckedTree, val ctx: Context) {
private def findOutputs(pat: NodePatternElement): Seq[Symbol] = {
val sym = typedTree.symbols(pat.name)
Seq(sym)
}
private def findOutputs(expr: Expr): Seq[Symbol] = {
Seq()
}
private def genSource(pat: NodePatternElement): LogicalPlan = {
pat.labels.find(label => ctx.database.nodeIndices(label).nonEmpty) match {
case Some(label) =>
val index = ctx.database.nodeIndices(label).head
val outputs = findOutputs(pat)
LogicalPlan.SeekByNodeLabelIndex(index, label, outputs, pat)
case None =>
val outputs = findOutputs(pat)
LogicalPlan.ScanAll(outputs, pat)
}
}
private def genPattern(elems: Seq[PatternElement]): LogicalPlan = {
assert(elems.size == 1)
val source = genSource(elems.head.asInstanceOf[NodePatternElement])
source
}
private def genSourceClause(clause: Clause): LogicalPlan = clause match {
case Match(opt, patterns, where) =>
// Create source.
assert(patterns.length == 1)
val Pattern(_, elements) = patterns.head
val plan = genPattern(elements)
// Add a filter.
new LogicalPlan.Filter(plan, where.expr)
case tree =>
sys.error(s"Unsupported source clause: $tree.")
}
def genReturn(input: LogicalPlan, ret: Return): LogicalPlan.Produce = {
val outputs = findOutputs(ret.expr)
LogicalPlan.Produce(input, outputs, ret.expr)
}
def genQueryPlan(tree: Tree): LogicalPlan = tree match {
case Query(clauses) =>
var plan = genSourceClause(clauses.head)
for (clause <- clauses.tail) clause match {
case ret @ Return(_) =>
plan = genReturn(plan, ret)
case clause =>
sys.error(s"Unsupported clause: $tree.")
}
plan
case tree =>
sys.error(s"Not a valid query: $tree.")
}
}
def generate(typedTree: TypecheckedTree, ctx: Context) = {
val instance = new Instance(typedTree, ctx)
val plan = instance.genQueryPlan(typedTree.tree)
Iterator(plan)
}
}
}
trait PhysicalPlanGenerator {
def generate(tree: LogicalPlan): Iterator[PhysicalPlan]
}
object PhysicalPlanGenerator {
case class Default() extends PhysicalPlanGenerator {
def generate(plan: LogicalPlan) = {
Iterator(PhysicalPlan(plan))
}
}
}
case class Configuration(
logicalPlanGenerator: LogicalPlanGenerator,
physicalPlanGenerator: PhysicalPlanGenerator,
estimator: PhysicalPlan => Cost
)
case class Context(
config: Configuration,
namer: Namer,
database: Database
)
object Configuration {
val defaultLogicalPlanGenerator = LogicalPlanGenerator.Default()
val defaultPhysicalPlanGenerator = PhysicalPlanGenerator.Default()
val defaultEstimator = (plan: PhysicalPlan) => {
Cost()
}
def default() = Configuration(
defaultLogicalPlanGenerator,
defaultPhysicalPlanGenerator,
defaultEstimator
)
}
trait Interpreter {
def interpret(query: CharacterStream): Stream
}
class DefaultInterpreter(ctx: Context) extends Interpreter {
def interpret(query: CharacterStream): Stream = {
val tree = Parser(ctx).apply(query)
val typedTree = Typechecker().apply(tree)
val logicalPlan = LogicalPlanner(ctx).apply(typedTree)
val physicalPlan = PhysicalPlanner(ctx).apply(logicalPlan)
physicalPlan.execute(ctx.database)
}
}
trait Compiler {
def compile(query: CharacterStream): Target
}
object Main {
def main(args: Array[String]) {
val db = Database.Default()
val config = Configuration.default()
val namer = new Namer
val ctx = Context(config, namer, db)
val interpreter = new DefaultInterpreter(ctx)
val query = CharacterStream.Default("""
match (a)
return a
""")
interpreter.interpret(query)
}
}

33
poc/CMakeLists.txt
View File

@ -1,33 +0,0 @@
find_package(Threads REQUIRED)
include_directories(${BENCHMARK_INCLUDE_DIR})
include_directories(${GTEST_INCLUDE_DIR})
# get all cpp abs file names recursively starting from current directory
file(GLOB poc_cpps *.cpp)
message(STATUS "Available poc cpp files are: ${poc_cpps}")
include_directories(${CMAKE_SOURCE_DIR}/poc)
# for each cpp file build binary
foreach(poc_cpp ${poc_cpps})
# get exec name (remove extension from the abs path)
get_filename_component(exec_name ${poc_cpp} NAME_WE)
set(target_name memgraph_poc_${exec_name})
# build exe file
add_executable(${target_name} ${poc_cpp})
# OUTPUT_NAME sets the real name of a target when it is built and can be
# used to help create two targets of the same name even though CMake
# requires unique logical target names
set_target_properties(${target_name} PROPERTIES OUTPUT_NAME ${exec_name})
# link libraries
target_link_libraries(${target_name} memgraph_lib)
target_link_libraries(${target_name} gtest gtest_main)
# google-benchmark requires threads
target_link_libraries(${target_name} benchmark Threads::Threads)
endforeach()

59
poc/allocator_benchmark.cpp
View File

@ -1,59 +0,0 @@
#include <iostream>
#include <thread>
#include <vector>
#include "glog/logging.h"
#include "memory/allocator.hpp"
#include "memory/maker.hpp"
#include "utils/measure_time.hpp"
struct TestStructure {
TestStructure(int a, int b, int c, int d) : a(a), b(b), c(c), d(d) {}
int a, b, c, d;
};
void test_classic(int N) {
TestStructure** xs = new TestStructure*[N];
for (int i = 0; i < N; ++i) xs[i] = new TestStructure(i, i, i, i);
for (int i = 0; i < N; ++i) delete xs[i];
delete[] xs;
}
void test_fast(int N) {
TestStructure** xs = new TestStructure*[N];
for (int i = 0; i < N; ++i) xs[i] = makeme<TestStructure>(i, i, i, i);
for (int i = 0; i < N; ++i) delete xs[i];
delete[] xs;
}
int main(void) {
constexpr int n_threads = 32;
constexpr int N = 80000000 / n_threads;
auto elapsed_classic = utils::measure_time<std::chrono::milliseconds>([&]() {
std::vector<std::thread> threads;
for (int i = 0; i < n_threads; ++i)
threads.push_back(std::thread(test_classic, N));
for (auto& thread : threads) {
thread.join();
}
});
std::cout << "Classic (new): " << elapsed_classic << "ms" << std::endl;
auto elapsed_fast = utils::measure_time<std::chrono::milliseconds>([&]() {
std::vector<std::thread> threads;
for (int i = 0; i < n_threads; ++i)
threads.push_back(std::thread(test_fast, N));
for (auto& thread : threads) {
thread.join();
}
});
std::cout << "Fast (fast allocator): " << elapsed_fast << "ms" << std::endl;
CHECK(elapsed_fast < elapsed_classic)
<< "Custom fast allocator "
"has to perform faster on simple array allocation";
return 0;
}

65
poc/allocator_test.cpp
View File

@ -1,65 +0,0 @@
#include "gtest/gtest.h"
#include "memory/allocator.hpp"
TEST(AllocatorTest, ABlockOfIntegersCanBeAllocated) {
constexpr int N = 100;
fast_allocator<int> a;
int* xs = a.allocate(N);
for (int i = 0; i < N; ++i) xs[i] = i;
// can we read them back?
for (int i = 0; i < N; ++i) ASSERT_EQ(xs[i], i);
// we should be able to free the memory
a.deallocate(xs, N);
}
TEST(AllocatorTest, AllocatorShouldWorkWithStructures) {
struct TestObject {
TestObject(int a, int b, int c, int d) : a(a), b(b), c(c), d(d) {}
int a, b, c, d;
};
fast_allocator<TestObject> a;
// allocate a single object
{
auto* test = a.allocate(1);
*test = TestObject(1, 2, 3, 4);
ASSERT_EQ(test->a, 1);
ASSERT_EQ(test->b, 2);
ASSERT_EQ(test->c, 3);
ASSERT_EQ(test->d, 4);
a.deallocate(test, 1);
}
// Allocate a block of structures
{
constexpr int N = 8;
auto* tests = a.allocate(N);
// structures should not overlap!
for (int i = 0; i < N; ++i) tests[i] = TestObject(i, i, i, i);
for (int i = 0; i < N; ++i) {
ASSERT_EQ(tests[i].a, i);
ASSERT_EQ(tests[i].b, i);
ASSERT_EQ(tests[i].c, i);
ASSERT_EQ(tests[i].d, i);
}
a.deallocate(tests, N);
}
}
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

36
poc/basic_bloom_filter.cpp
View File

@ -1,36 +0,0 @@
#include "gtest/gtest.h"
#include <functional>
#include "bloom_filter.hpp"
#include "utils/hashing/fnv64.hpp"
using StringHashFunction = std::function<uint64_t(const std::string &)>;
TEST(BloomFilterTest, InsertContains) {
StringHashFunction hash1 = fnv64;
StringHashFunction hash2 = fnv1a64;
std::vector<StringHashFunction> funcs = {hash1, hash2};
BloomFilter<std::string, 64> bloom(funcs);
std::string test = "test";
std::string kifla = "kifla";
bool contains_test = bloom.contains(test);
ASSERT_EQ(contains_test, false);
bloom.insert(test);
contains_test = bloom.contains(test);
ASSERT_EQ(contains_test, true);
bool contains_kifla = bloom.contains(kifla);
ASSERT_EQ(contains_kifla, false);
bloom.insert(kifla);
contains_kifla = bloom.contains(kifla);
ASSERT_EQ(contains_kifla, true);
}
int main(int argc, char **argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

53
poc/basic_bloom_filter_benchmark.cpp
View File

@ -1,53 +0,0 @@
#include <random>
#include <thread>
#include <benchmark/benchmark_api.h>
#include <glog/logging.h>
#include "bloom_filter.hpp"
#include "utils/hashing/fnv64.hpp"
#include "utils/random/random_generator.hpp"
using utils::random::StringGenerator;
using StringHashFunction = std::function<uint64_t(const std::string &)>;
template <class Type, int Size>
static void TestBloom(benchmark::State &state, BloomFilter<Type, Size> *bloom,
const std::vector<Type> &elements) {
while (state.KeepRunning()) {
for (int start = 0; start < state.range(0); start++)
if (start % 2)
bloom->contains(elements[start]);
else
bloom->insert(elements[start]);
}
state.SetComplexityN(state.range(0));
}
auto BM_Bloom = [](benchmark::State &state, auto *bloom, const auto &elements) {
TestBloom(state, bloom, elements);
};
int main(int argc, char **argv) {
benchmark::Initialize(&argc, argv);
google::InitGoogleLogging(argv[0]);
StringGenerator generator(4);
auto elements = utils::random::generate_vector(generator, 1 << 16);
StringHashFunction hash1 = fnv64;
StringHashFunction hash2 = fnv1a64;
std::vector<StringHashFunction> funcs = {hash1, hash2};
BloomFilter<std::string, 128> bloom(funcs);
benchmark::RegisterBenchmark("SimpleBloomFilter Benchmark Test", BM_Bloom,
&bloom, elements)
->RangeMultiplier(2)
->Range(1, 1 << 16)
->Complexity(benchmark::oN);
benchmark::RunSpecifiedBenchmarks();
return 0;
}

21
poc/block_allocator_test.cpp
View File

@ -1,21 +0,0 @@
#include <gtest/gtest.h>
#include "memory/block_allocator.hpp"
TEST(BlockAllocatorTest, UnusedVsReleaseSize) {
BlockAllocator<64> block_allocator(10);
void *block = block_allocator.acquire();
block_allocator.release(block);
EXPECT_EQ(block_allocator.unused_size(), 9);
EXPECT_EQ(block_allocator.release_size(), 1);
}
TEST(BlockAllocatorTest, CountMallocAndFreeCalls) {
// TODO: implementation
EXPECT_EQ(true, true);
}
int main(int argc, char **argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

69
poc/bloom_filter.hpp
View File

@ -1,69 +0,0 @@
#pragma once
#include <bitset>
#include <functional>
#include <iostream>
#include <vector>
/**
* Implementation of a generic Bloom Filter.
* Read more about bloom filters here:
* http://en.wikipedia.org/wiki/Bloom_filter
* http://www.jasondavies.com/bloomfilter/
*
* Type specifies the type of data stored
*/
template <class Type, int BucketSize = 8>
class BloomFilter {
private:
using HashFunction = std::function<uint64_t(const Type &)>;
using CompresionFunction = std::function<int(uint64_t)>;
std::bitset<BucketSize> filter_;
std::vector<HashFunction> hashes_;
CompresionFunction compression_;
std::vector<int> buckets;
int default_compression(uint64_t hash) { return hash % BucketSize; }
void get_buckets(const Type &data) {
for (int i = 0; i < hashes_.size(); i++)
buckets[i] = compression_(hashes_[i](data));
}
void print_buckets(std::vector<uint64_t> &buckets) {
for (int i = 0; i < buckets.size(); i++) {
std::cout << buckets[i] << " ";
}
std::cout << std::endl;
}
public:
BloomFilter(std::vector<HashFunction> funcs,
CompresionFunction compression = {})
: hashes_(funcs) {
if (!compression)
compression_ = std::bind(&BloomFilter::default_compression, this,
std::placeholders::_1);
else
compression_ = compression;
buckets.resize(hashes_.size());
}
bool contains(const Type &data) {
get_buckets(data);
bool contains_element = true;
for (int i = 0; i < buckets.size(); i++)
contains_element &= filter_[buckets[i]];
return contains_element;
}
void insert(const Type &data) {
get_buckets(data);
for (int i = 0; i < buckets.size(); i++) filter_[buckets[i]] = true;
}
};

77
poc/bounded_spsc_queue.hpp
View File

@ -1,77 +0,0 @@
#pragma once
#include <atomic>
#include <memory>
namespace lockfree {
template <class T, size_t N>
class BoundedSpscQueue {
public:
static constexpr size_t size = N;
BoundedSpscQueue() = default;
BoundedSpscQueue(const BoundedSpscQueue&) = delete;
BoundedSpscQueue(BoundedSpscQueue&&) = delete;
BoundedSpscQueue& operator=(const BoundedSpscQueue&) = delete;
bool push(const T& item) {
// load the current tail
// [] [] [1] [2] [3] [4] [5] [$] []
// H T
auto t = tail.load(std::memory_order_relaxed);
// what will next tail be after we push
// [] [] [1] [2] [3] [4] [5] [$] [ ]
// H T T'
auto next = increment(t);
// check if queue is full and do nothing if it is
// [3] [4] [5] [6] [7] [8] [$] [ 1 ] [2]
// T T'H
if (next == head.load(std::memory_order_acquire)) return false;
// insert the item into the empty spot
// [] [] [1] [2] [3] [4] [5] [ ] []
// H T T'
items[t] = item;
// release the tail to the consumer (serialization point)
// [] [] [1] [2] [3] [4] [5] [ $ ] []
// H T T'
tail.store(next, std::memory_order_release);
return true;
}
bool pop(T& item) {
// [] [] [1] [2] [3] [4] [5] [$] []
// H T
auto h = head.load(std::memory_order_relaxed);
// [] [] [] [] [ $ ] [] [] [] []
// H T
if (h == tail.load(std::memory_order_acquire)) return false;
// move an item from the queue
item = std::move(items[h]);
// serialization point wrt producer
// [] [] [] [2] [3] [4] [5] [$] []
// H T
head.store(increment(h), std::memory_order_release);
return true;
}
private:
static constexpr size_t capacity = N + 1;
std::array<T, capacity> items;
std::atomic<size_t> head{0}, tail{0};
size_t increment(size_t idx) const { return (idx + 1) % capacity; }
};
}

6
poc/example.cpp
View File

@ -1,6 +0,0 @@
#include <iostream>
int main() {
std::cout << "Proof of concept binary example" << std::endl;
return 0;
}

123
poc/hp.hpp
View File

@ -1,123 +0,0 @@
#pragma once
#include <unistd.h>
#include <atomic>
#include <iostream>
namespace memory {
constexpr const size_t HP_SIZE = 128;
class HP {
public:
// this object can't be copied or moved
HP(HP&) = delete;
HP(HP&&) = delete;
// grabs a singleton instance
static HP& get() {
static HP hp;
return hp;
}
class reference {
friend class HP;
public:
reference(reference&) = delete;
// this type shouldn't be copyable to avoid calling its destructor
// multiple times, but should be movable
reference(reference&& other) {
this->idx = other.idx;
// set the index to a negative number to indicate that this
// index has been moved and that you should not free its
// hazard pointer
other.idx = -1;
}
// hazard pointer is cleared once reference goes out of scope
~reference() {
// TODO: remove
// std::cout << "reference destructor called: ";
// std::cout << this->idx;
// std::cout << std::endl;
// check if this reference was moved during its lifetime
if (idx < 0) return;
auto& hp = HP::get();
hp.clear(*this);
}
// TODO: ???
reference& operator=(reference&&) { return *this; }
private:
reference(int64_t idx) : idx(idx) {}
int64_t idx;
};
friend class reference;
template <class T>
reference insert(T* ptr) {
auto p = reinterpret_cast<uintptr_t>(ptr);
while (true) {
// try to find a free spot in the hazard pointer list
for (size_t i = 0; i < HP_SIZE; ++i) {
auto hazard = ptr_list[i].load();
// if this spot isn't free, continue searching
if (hazard != 0) continue;
// try to take this spot, if we fail, then another thread has
// just taken it. continue searching for a new one
if (!ptr_list[i].compare_exchange_strong(hazard, p)) continue;
// found a free spot! return a reference to this spot so it
// can be cleared later
return reference(i);
}
// we didn't find any free spots, sleep for a while and try again
// from the beginning, some other thread might have freed a spot
// while we were traversing the lsit.
usleep(250);
}
}
bool find(uintptr_t hptr) {
for (size_t i = 0; i < HP_SIZE; ++i) {
auto& hptr_i = ptr_list[i];
if (hptr_i != hptr) continue;
if (hptr_i.load() == 1) return true;
if (hptr_i.load() == 0) return false;
}
return false;
}
friend std::ostream& operator<<(std::ostream& os, const HP& hp) {
os << "Hazard pointers: ";
for (size_t i = 0; i < HP_SIZE; ++i) {
auto& hptr_i = hp.ptr_list[i];
os << hptr_i.load() << " ";
}
return os << std::endl;
}
private:
HP() {
for (size_t i = 0; i < HP_SIZE; ++i) ptr_list[i].store(0);
}
void clear(reference& ref) { ptr_list[ref.idx].store(0); }
std::atomic<uintptr_t> ptr_list[HP_SIZE];
};
}

64
poc/kdtree/build.hpp
View File

@ -1,64 +0,0 @@
#pragma once
#include <algorithm>
#include <functional>
#include <vector>
#include "kdnode.hpp"
#include "math.hpp"
namespace kd {
template <class T, class U>
using Nodes = std::vector<KdNode<T, U>*>;
template <class T, class U>
KdNode<T, U>* build(Nodes<T, U>& nodes, byte axis = 0) {
// if there are no elements left, we've completed building of this branch
if (nodes.empty()) return nullptr;
// comparison function to use for sorting the elements
auto fsort = [axis](KdNode<T, U>* a, KdNode<T, U>* b) -> bool {
return kd::math::axial_distance(a->coord, b->coord, axis) < 0;
};
size_t median = nodes.size() / 2;
// partial sort nodes vector to compute median and ensure that elements
// less than median are positioned before the median so we can slice it
// nicely
// internal implementation is O(n) worst case
// tl;dr http://en.wikipedia.org/wiki/Introselect
std::nth_element(nodes.begin(), nodes.begin() + median, nodes.end(), fsort);
// set axis for the node
auto node = nodes.at(median);
node->axis = axis;
// slice the vector into two halves
auto left = Nodes<T, U>(nodes.begin(), nodes.begin() + median);
auto right = Nodes<T, U>(nodes.begin() + median + 1, nodes.end());
// recursively build left and right branches
node->left = build(left, axis ^ 1);
node->right = build(right, axis ^ 1);
return node;
}
template <class T, class U, class It>
KdNode<T, U>* build(It first, It last) {
Nodes<T, U> kdnodes;
std::transform(first, last, std::back_inserter(kdnodes),
[&](const std::pair<Point<T>, U>& element) {
auto key = element.first;
auto data = element.second;
return new KdNode<T, U>(key, data);
});
// build the tree from the kdnodes and return the root node
return build(kdnodes);
}
}

46
poc/kdtree/kdnode.hpp
View File

@ -1,46 +0,0 @@
#pragma once
#include <memory>
#include "point.hpp"
namespace kd {
template <class T, class U>
class KdNode {
public:
KdNode(const U& data)
: axis(0),
coord(Point<T>(0, 0)),
left(nullptr),
right(nullptr),
data(data) {}
KdNode(const Point<T>& coord, const U& data)
: axis(0), coord(coord), left(nullptr), right(nullptr), data(data) {}
KdNode(unsigned char axis, const Point<T>& coord, const U& data)
: axis(axis), coord(coord), left(nullptr), right(nullptr), data(data) {}
KdNode(unsigned char axis, const Point<T>& coord, KdNode<T, U>* left,
KdNode<T, U>* right, const U& data)
: axis(axis), coord(coord), left(left), right(right), data(data) {}
~KdNode();
unsigned char axis;
Point<T> coord;
KdNode<T, U>* left;
KdNode<T, U>* right;
U data;
};
template <class T, class U>
KdNode<T, U>::~KdNode() {
delete left;
delete right;
}
}

35
poc/kdtree/kdtree.hpp
View File

@ -1,35 +0,0 @@
#pragma once
#include <vector>
#include "build.hpp"
#include "nns.hpp"
namespace kd {
template <class T, class U>
class KdTree {
public:
KdTree() {}
template <class It>
KdTree(It first, It last);
const U& lookup(const Point<T>& pk) const;
protected:
std::unique_ptr<KdNode<float, U>> root;
};
template <class T, class U>
const U& KdTree<T, U>::lookup(const Point<T>& pk) const {
// do a nearest neighbour search on the tree
return kd::nns(pk, root.get())->data;
}
template <class T, class U>
template <class It>
KdTree<T, U>::KdTree(It first, It last) {
root.reset(kd::build<T, U, It>(first, last));
}
}

34
poc/kdtree/math.hpp
View File

@ -1,34 +0,0 @@
#pragma once
#include <cmath>
#include <limits>
#include "point.hpp"
namespace kd {
namespace math {
using byte = unsigned char;
// returns the squared distance between two points
template <class T>
T distance_sq(const Point<T>& a, const Point<T>& b) {
auto dx = a.longitude - b.longitude;
auto dy = a.latitude - b.latitude;
return dx * dx + dy * dy;
}
// returns the distance between two points
template <class T>
T distance(const Point<T>& a, const Point<T>& b) {
return std::sqrt(distance_sq(a, b));
}
// returns the distance between two points looking at a specific axis
// \param axis 0 if abscissa else 1 if ordinate
template <class T>
T axial_distance(const Point<T>& a, const Point<T>& b, byte axis) {
return axis == 0 ? a.longitude - b.longitude : a.latitude - b.latitude;
}
}
}

93
poc/kdtree/nns.hpp
View File

@ -1,93 +0,0 @@
#pragma once
#include "kdnode.hpp"
#include "math.hpp"
#include "point.hpp"
namespace kd {
// helper class for calculating the nearest neighbour in a kdtree
template <class T, class U>
struct Result {
Result() : node(nullptr), distance_sq(std::numeric_limits<T>::infinity()) {}
Result(const KdNode<T, U>* node, T distance_sq)
: node(node), distance_sq(distance_sq) {}
const KdNode<T, U>* node;
T distance_sq;
};
// a recursive implementation for the kdtree nearest neighbour search
// \param p the point for which we search for the nearest neighbour
// \param node the root of the subtree during recursive descent
// \param best the place to save the best result so far
template <class T, class U>
void nns(const Point<T>& p, const KdNode<T, U>* const node,
Result<T, U>& best) {
if (node == nullptr) return;
T d = math::distance_sq(p, node->coord);
// keep record of the closest point C found so far
if (d < best.distance_sq) {
best.node = node;
best.distance_sq = d;
}
// where to traverse next?
// what to prune?
// |
// possible |
// prune *
// area | - - - - -* P
// |
//
// |----------|
// dx
//
// possible prune
// RIGHT area
//
// --------*------ ---
// | |
// LEFT |
// | | dy
// |
// | |
// * p ---
T axd = math::axial_distance(p, node->coord, node->axis);
// traverse the subtree in order that
// maximizes the probability for pruning
auto near = axd > 0 ? node->right : node->left;
auto far = axd > 0 ? node->left : node->right;
// try near first
nns(p, near, best);
// prune subtrees once their bounding boxes say
// that they can't contain any point closer than C
if (axd * axd >= best.distance_sq) return;
// try other subtree
nns(p, far, best);
}
// an implementation for the kdtree nearest neighbour search
// \param p the point for which we search for the nearest neighbour
// \param root the root of the tree
// \return the nearest neighbour for the point p
template <class T, class U>
const KdNode<T, U>* nns(const Point<T>& p, const KdNode<T, U>* root) {
Result<T, U> best;
// begin recursive search
nns(p, root, best);
return best.node;
}
}

26
poc/kdtree/point.hpp
View File

@ -1,26 +0,0 @@
#pragma once
#include <ostream>
namespace kd {
template <class T>
class Point {
public:
Point(T latitude, T longitude) : latitude(latitude), longitude(longitude) {}
// latitude
// y
// ^
// |
// 0---> x longitude
T latitude;
T longitude;
/// nice stream formatting with the standard << operator
friend std::ostream& operator<<(std::ostream& stream, const Point& p) {
return stream << "(lat: " << p.latitude << ", lng: " << p.longitude << ')';
}
};
}

57
poc/measure_time.cpp
View File

@ -1,57 +0,0 @@
#include <chrono>
#include <iostream>
#include <glog/logging.h>
#include <gtest/gtest.h>
#include "timer.hpp"
using namespace std::chrono_literals;
using namespace utils;
/**
* Creates a test timer which will log timeout message at the timeout event.
*
* @param counter how many time units the timer has to wait
*
* @return shared pointer to a timer
*/
Timer::sptr create_test_timer(int64_t counter) {
return std::make_shared<Timer>(counter,
[]() { DLOG(INFO) << "Timer timeout"; });
}
TEST(TimerSchedulerTest, TimerSchedulerExecution) {
// initialize the timer
TimerScheduler<TimerSet, std::chrono::seconds> timer_scheduler;
// run the timer
timer_scheduler.run();
// add a couple of test timers
for (int64_t i = 1; i <= 3; ++i) {
timer_scheduler.add(create_test_timer(i));
}
// wait for that timers
std::this_thread::sleep_for(4s);
ASSERT_EQ(timer_scheduler.size(), 0);
// add another test timer
timer_scheduler.add(create_test_timer(1));
// wait for another timer
std::this_thread::sleep_for(2s);
// the test is done
timer_scheduler.stop();
ASSERT_EQ(timer_scheduler.size(), 0);
}
int main(int argc, char **argv) {
::google::InitGoogleLogging(argv[0]);
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

43
poc/memory/allocator.hpp
View File

@ -1,43 +0,0 @@
#pragma once
#include <cstdlib>
#include <new>
template <class Tp>
struct fast_allocator {
typedef Tp value_type;
fast_allocator() = default;
template <class T>
fast_allocator(const fast_allocator<T>&) {}
Tp* allocate(std::size_t n);
void deallocate(Tp* p, std::size_t n);
};
template <class Tp>
Tp* fast_allocator<Tp>::allocate(std::size_t n) {
// hopefully we're using jemalloc here!
Tp* mem = static_cast<Tp*>(malloc(n * sizeof(Tp)));
if (mem != nullptr) return mem;
throw std::bad_alloc();
}
template <class Tp>
void fast_allocator<Tp>::deallocate(Tp* p, std::size_t) {
// hopefully we're using jemalloc here!
free(p);
}
template <class T, class U>
bool operator==(const fast_allocator<T>&, const fast_allocator<U>&) {
return true;
}
template <class T, class U>
bool operator!=(const fast_allocator<T>& a, const fast_allocator<U>& b) {
return !(a == b);
}

21
poc/memory/atomic_shared_ptr.hpp
View File

@ -1,21 +0,0 @@
#pragma once
#include <atomic>
#include <memory>
// I heard this is patented.
template <class T>
class atomic_shared_ptr final {
public:
atomic_shared_ptr(std::shared_ptr<T>&& ptr) : ptr(ptr) {}
std::shared_ptr<T> load() { return std::move(std::atomic_load(&ptr)); }
bool compare_exchange_weak(std::shared_ptr<T>& expected,
std::shared_ptr<T> desired) {
return atomic_compare_exchange_weak(&ptr, &expected, desired);
}
private:
std::shared_ptr<T> ptr;
};

57
poc/memory/block_allocator.hpp
View File

@ -1,57 +0,0 @@
#pragma once
#include <memory>
#include <vector>
#include "utils/on_scope_exit.hpp"
/* @brief Allocates blocks of block_size and stores
* the pointers on allocated blocks inside a vector.
*/
template <size_t block_size>
class BlockAllocator {
struct Block {
Block() { data = malloc(block_size); }
Block(void *ptr) { data = ptr; }
void *data;
};
public:
static constexpr size_t size = block_size;
BlockAllocator(size_t capacity = 0) {
for (size_t i = 0; i < capacity; ++i) unused_.emplace_back();
}
~BlockAllocator() {
for (auto block : unused_) free(block.data);
unused_.clear();
for (auto block : release_) free(block.data);
release_.clear();
}
size_t unused_size() const { return unused_.size(); }
size_t release_size() const { return release_.size(); }
// Returns nullptr on no memory.
void *acquire() {
if (unused_.size() == 0) unused_.emplace_back();
auto ptr = unused_.back().data;
// TODO is it necessary to use OnScopeExit here? ptr is copied by value, right?
utils::OnScopeExit on_exit([this]() { unused_.pop_back(); });
return ptr;
}
void release(void *ptr) { release_.emplace_back(ptr); }
private:
// TODO: try implement with just one vector
// but consecutive acquire release calls should work
// TODO: measure first!
std::vector<Block> unused_;
std::vector<Block> release_;
};

19
poc/memory/maker.hpp
View File

@ -1,19 +0,0 @@
#pragma once
#include <cstdlib>
#include "allocator.hpp"
template <class T, typename... Args, class allocator = fast_allocator<T>>
T* makeme(Args... args) {
allocator alloc;
T* mem = alloc.allocate(1);
return new (mem) T(args...);
}
template <class T, class allocator = fast_allocator<T>>
void takeme(T* mem) {
allocator alloc;
mem->~T();
alloc.deallocate(mem, 1);
}

121
poc/memory/stack_allocator.hpp
View File

@ -1,121 +0,0 @@
#pragma once
#include <cmath>
#include "memory/block_allocator.hpp"
#include "utils/exceptions.hpp"
#include "utils/likely.hpp"
// http://en.cppreference.com/w/cpp/language/new
namespace utils {
///
/// @brief Raised by @c StackAllocator when it cannot allocate the object.
///
class OutOfMemory : public StacktraceException {
public:
using StacktraceException::StacktraceException;
};
// Useful for allocating memory which can be freed with one call.
// Most performant for data which need to be present to the end.
class StackAllocator {
static constexpr size_t page_size = 64 * 1024;
public:
~StackAllocator() { free(); }
// Allocates memory for object of type T.
// Retruns pointer to memory for it.
template <class T>
inline T *allocate() {
// If size is bigger than pages_size then this do-whil will never end
// until it eats all the memory.
static_assert(sizeof(T) <= page_size,
"Cant allocate objects larger than page_size");
do {
// Mask which has log2(alignof(T)) lower bits setted to 0 and the
// rest to 1.
// example:
// alignof(T)==8 => mask=0xfffffffffffffff8;
// This will be calculated in compile time.
size_t mask = ~(((size_t)alignof(T)) - 1);
// aligned contains ptr aligned to alignof(T).
// There are two types of head ptr:
// a) aligned to alignof(T)
// b) not aligned to alignof(T)
// For a) whe want for aligned to be equal to head, and for b)
// aligned shuold be first aligned ptr greater than head.
//
// head - 1 => turns a) into b) now whe only have to get first
// aligned ptr greater than (head - 1).
//
// (head - 1) & mask => will produce first smaller than head ptr
// aligned to alignof(T).
//
// ( (head - 1) & mask ) + alignof(T) => will produce first grater
// or equal than head ptr aligned to alignof(T).
char *aligned = (char *)(((((size_t)head) - 1) & mask) + alignof(T));
// New head which points to unallocated memory points to first byte
// after space for object T.
char *new_head = aligned + sizeof(T);
// If the new_head is greater than end that means that there isn't
// enough space for object T in current block of memory.
if (LIKELY(new_head <= end)) {
// All is fine, head can become new_head
head = new_head;
// Returns aligned ptr with enough space for object T.
return (T *)aligned;
}
// There isn't enough free space so whe must allocate more.
void *alloc = blocks.acquire();
// Check if there are memory. If not throw exception rather than
// return nullptr.
if (UNLIKELY(alloc == nullptr))
throw new OutOfMemory("BlockAllocator returned nullptr");
// Remember that whee allocated memory so that whe can free-it
// after.
allocated_blocks.push_back(alloc);
// Set new head, the old one isn't needed anymore.
head = (char *)alloc;
// Update end to point to first byte after newly allocated memory.
end = head + page_size;
// After allocating new memory lets try again to "allocate" place
// for T.
} while (true);
}
template <class T, class... Args>
inline T *make(Args &&... args) {
auto ptr = allocate<T>();
new (ptr) T(std::forward<Args>(args)...);
return ptr;
}
// Relases all memory.
void free() {
while (allocated_blocks.size()) {
blocks.release(allocated_blocks.back());
allocated_blocks.pop_back();
}
}
private:
BlockAllocator<page_size> blocks;
std::vector<void *> allocated_blocks;
char *head = {nullptr};
char *end = {nullptr};
};
} // namespace utils

141
poc/mpsc_queue.hpp
View File

@ -1,141 +0,0 @@
#pragma once
#include <atomic>
#include <memory>
namespace lockfree {
/** @brief Multiple-Producer Single-Consumer Queue
* A wait-free (*) multiple-producer single-consumer queue.
*
* features:
* - wait-free
* - fast producers (only one atomic XCHG and and one atomic store with
* release semantics)
* - extremely fast consumer (only atomic loads with acquire semantics on
* the fast path and atomic loads + atomic XCHG on the slow path)
* - no need for order reversion -> pop() is always O(1)
* - ABA free
*
* great for using in loggers, garbage collectors etc.
*
* (*) there is a small window of inconsistency from the lock free design
* see the url below for details
* URL:
* http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue
*
* mine is not intrusive for better modularity, but with slightly worse
* performance because it needs to do two memory allocations instead of
* one
*
* @tparam T Type of the items to store in the queue
*/
template <class T>
class MpscQueue {
struct Node {
Node(Node* next, std::unique_ptr<T>&& item)
: next(next), item(std::forward<std::unique_ptr<T>>(item)) {}
std::atomic<Node*> next;
std::unique_ptr<T> item;
};
public:
MpscQueue() {
auto stub = new Node(nullptr, nullptr);
head.store(stub);
tail = stub;
}
~MpscQueue() {
// purge all elements from the queue
while (pop()) {
}
// we are left with a stub, delete that
delete tail;
}
MpscQueue(MpscQueue&) = delete;
MpscQueue(MpscQueue&&) = delete;
/** @brief Pushes an item into the queue.
*
* Pushes an item into the front of the queue.
*
* @param item std::unique_ptr<T> An item to push into the queue
* @return void
*/
void push(std::unique_ptr<T>&& item) {
push(new Node(nullptr, std::forward<std::unique_ptr<T>>(item)));
}
/** @brief Pops a node from the queue.
*
* Pops and returns a node from the back of the queue.
*
* @return std::unique_ptr<T> A pointer to the node popped from the
* queue, nullptr if nothing was popped
*/
std::unique_ptr<T> pop() {
auto tail = this->tail;
// serialization point wrt producers
auto next = tail->next.load(std::memory_order_acquire);
if (next) {
// remove the last stub from the queue
// make [2] the next stub and return it's data
//
// H --> [n] <- ... <- [2] <--+--[STUB] +-- T
// | |
// +-----------+
this->tail = next;
// delete the stub node
// H --> [n] <- ... <- [STUB] <-- T
delete tail;
return std::move(next->item);
}
return nullptr;
}
private:
std::atomic<Node*> head;
Node* tail;
/** @brief Pushes a new node into the queue.
*
* Pushes a new node containing the item into the front of the queue.
*
* @param node Node* A pointer to node you want to push into the queue
* @return void
*/
void push(Node* node) {
// initial state
// H --> [3] <- [2] <- [STUB] <-- T
// serialization point wrt producers, acquire-release
auto old = head.exchange(node, std::memory_order_acq_rel);
// after exchange
// H --> [4] [3] <- [2] <- [STUB] <-- T
// this is the window of inconsistency, if the producer is blocked
// here, the consumer is also blocked. but this window is extremely
// small, it's followed by a store operation which is a
// serialization point wrt consumer
// old holds a pointer to node [3] and we need to link the [3] to a
// newly created node [4] using release semantics
// serialization point wrt consumer, release
old->next.store(node, std::memory_order_release);
// finally, we have a queue like this
// H --> [4] <- [3] <- [2] <- [1] <-- T
}
};
}

26
poc/option_ptr.hpp
View File

@ -1,26 +0,0 @@
#pragma once
#include "glog/logging.h"
// Like option just for pointers. More efficent than option.
template <class T>
class OptionPtr {
public:
OptionPtr() {}
OptionPtr(T *ptr) : ptr(ptr) {}
bool is_present() { return ptr != nullptr; }
T *get() {
DCHECK(is_present()) << "Data is not present.";
return ptr;
}
private:
T *ptr = nullptr;
};
template <class T>
auto make_option_ptr(T *t) {
return OptionPtr<T>(t);
}

51
poc/ptr_int.hpp
View File

@ -1,51 +0,0 @@
#pragma once
#include <cinttypes>
constexpr std::size_t log2(std::size_t n) {
return ((n < 2) ? 0 : 1 + log2(n >> 1));
}
template <typename PtrT>
struct PointerPackTraits {
// here is a place to embed something like platform specific things
// TODO: cover more cases
constexpr static int free_bits = log2(alignof(PtrT));
static auto get_ptr(uintptr_t value) { return (PtrT)(value); }
};
template <typename PtrT, int IntBits, typename IntT = unsigned,
typename PtrTraits = PointerPackTraits<PtrT>>
class PtrInt {
private:
constexpr static int int_shift = PtrTraits::free_bits - IntBits;
constexpr static uintptr_t ptr_mask =
~(uintptr_t)(((intptr_t)1 << PtrTraits::free_bits) - 1);
constexpr static uintptr_t int_mask =
(uintptr_t)(((intptr_t)1 << IntBits) - 1);
uintptr_t value{0};
public:
PtrInt(PtrT pointer, IntT integer) {
set_ptr(pointer);
set_int(integer);
}
auto set_ptr(PtrT pointer) {
auto integer = static_cast<uintptr_t>(get_int());
auto ptr = reinterpret_cast<uintptr_t>(pointer);
value = (ptr_mask & ptr) | (integer << int_shift);
}
auto set_int(IntT integer) {
auto ptr = reinterpret_cast<uintptr_t>(get_ptr());
auto int_shifted = static_cast<uintptr_t>(integer << int_shift);
value = (int_mask & int_shifted) | ptr;
}
auto get_ptr() const { return PtrTraits::get_ptr(value & ptr_mask); }
auto get_int() const { return (IntT)((value >> int_shift) & int_mask); }
};

26
poc/ptr_int_test.cpp
View File

@ -1,26 +0,0 @@
#include <gtest/gtest.h>
#include "ptr_int.hpp"
TEST(PtrInt, SizeOf) {
ASSERT_EQ(sizeof(PtrInt<int *, 1, int>), sizeof(uintptr_t));
}
TEST(PtrInt, ConstructionAndRead) {
auto ptr1 = std::make_unique<int>(2);
PtrInt<int *, 2, int> pack1(ptr1.get(), 1);
ASSERT_EQ(pack1.get_int(), 1);
ASSERT_EQ(pack1.get_ptr(), ptr1.get());
auto ptr2 = std::make_unique<int>(2);
PtrInt<int *, 3, int> pack2(ptr2.get(), 4);
ASSERT_EQ(pack2.get_int(), 4);
ASSERT_EQ(pack2.get_ptr(), ptr2.get());
}
int main(int argc, char **argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

462
poc/rh_hashmap.hpp
View File

@ -1,462 +0,0 @@
#pragma once
#include <cstring>
#include <functional>
#include "glog/logging.h"
#include "option_ptr.hpp"
#include "utils/crtp.hpp"
// RobinHood base.
// Entries are POINTERS alligned to 8B.
// Entries must know thers key.
// D must have method K& get_key()
// K must be comparable with ==.
template <class K, class D, size_t init_size_pow2 = 2>
class RhBase {
protected:
class Combined {
public:
Combined() : data(0) {}
Combined(D *data, size_t off) { this->data = ((size_t)data) | off; }
bool valid() const { return data != 0; }
size_t off() const { return data & 0x7; }
void decrement_off_unsafe() { data--; }
bool decrement_off() {
if (off() > 0) {
data--;
return true;
}
return false;
}
bool increment_off() {
if (off() < 7) {
data++;
return true;
}
return false;
}
D *ptr() const { return (D *)(data & (~(0x7))); }
bool equal(const K &key, size_t off) {
return this->off() == off && key == ptr()->get_key();
}
friend bool operator==(const Combined &a, const Combined &b) {
return a.off() == b.off() && a.ptr()->get_key() == b.ptr()->get_key();
}
friend bool operator!=(const Combined &a, const Combined &b) {
return !(a == b);
}
private:
size_t data;
};
// Base for all iterators. It can start from any point in map.
template <class It>
class IteratorBase : public utils::Crtp<It> {
protected:
IteratorBase() : map(nullptr) { advanced = index = ~((size_t)0); }
IteratorBase(const RhBase *map) {
index = 0;
while (index < map->capacity && !map->array[index].valid()) {
index++;
}
if (index >= map->capacity) {
this->map = nullptr;
advanced = index = ~((size_t)0);
} else {
this->map = map;
advanced = index;
}
}
IteratorBase(const RhBase *map, size_t start)
: map(map), advanced(0), index(start) {}
const RhBase *map;
// How many times did whe advance.
size_t advanced;
// Current position in array
size_t index;
public:
IteratorBase(const IteratorBase &) = default;
IteratorBase(IteratorBase &&) = default;
D *operator*() {
DCHECK(index < map->capacity && map->array[index].valid())
<< "Either index is invalid or data is not valid.";
return map->array[index].ptr();
}
D *operator->() {
DCHECK(index < map->capacity && map->array[index].valid())
<< "Either index is invalid or data is not valid.";
return map->array[index].ptr();
}
It &operator++() {
DCHECK(index < map->capacity && map->array[index].valid())
<< "Either index is invalid or data is not valid.";
auto mask = map->mask();
do {
advanced++;
if (advanced >= map->capacity) {
// Whe have advanced more than the capacity of map is so whe
// are done.
map = nullptr;
advanced = index = ~((size_t)0);
break;
}
index = (index + 1) & mask;
} while (!map->array[index].valid()); // Check if there is element
// at current position.
return this->derived();
}
It &operator++(int) { return operator++(); }
friend bool operator==(const It &a, const It &b) {
return a.index == b.index && a.map == b.map;
}
friend bool operator!=(const It &a, const It &b) { return !(a == b); }
};
public:
class ConstIterator : public IteratorBase<ConstIterator> {
friend class RhBase;
protected:
ConstIterator(const RhBase *map) : IteratorBase<ConstIterator>(map) {}
ConstIterator(const RhBase *map, size_t index)
: IteratorBase<ConstIterator>(map, index) {}
public:
ConstIterator() = default;
ConstIterator(const ConstIterator &) = default;
const D *operator->() { return IteratorBase<ConstIterator>::operator->(); }
const D *operator*() { return IteratorBase<ConstIterator>::operator*(); }
};
class Iterator : public IteratorBase<Iterator> {
friend class RhBase;
protected:
Iterator(const RhBase *map) : IteratorBase<Iterator>(map) {}
Iterator(const RhBase *map, size_t index)
: IteratorBase<Iterator>(map, index) {}
public:
Iterator() = default;
Iterator(const Iterator &) = default;
};
RhBase() {}
RhBase(const RhBase &other) { copy_from(other); }
RhBase(RhBase &&other) { take_from(std::move(other)); }
~RhBase() { this->clear(); }
RhBase &operator=(const RhBase &other) {
clear();
copy_from(other);
return *this;
}
RhBase &operator=(RhBase &&other) {
clear();
take_from(std::move(other));
return *this;
}
Iterator begin() { return Iterator(this); }
ConstIterator begin() const { return ConstIterator(this); }
ConstIterator cbegin() const { return ConstIterator(this); }
Iterator end() { return Iterator(); }
ConstIterator end() const { return ConstIterator(); }
ConstIterator cend() const { return ConstIterator(); }
protected:
// Copys RAW BYTE data from other RhBase.
void copy_from(const RhBase &other) {
capacity = other.capacity;
count = other.count;
if (capacity > 0) {
size_t bytes = sizeof(Combined) * capacity;
array = (Combined *)malloc(bytes);
memcpy(array, other.array, bytes);
} else {
array = nullptr;
}
}
// Takes data from other RhBase.
void take_from(RhBase &&other) {
capacity = other.capacity;
count = other.count;
array = other.array;
other.array = nullptr;
other.count = 0;
other.capacity = 0;
}
// Initiazes array with given capacity.
void init_array(size_t capacity) {
size_t bytes = sizeof(Combined) * capacity;
array = (Combined *)malloc(bytes);
std::memset(array, 0, bytes);
this->capacity = capacity;
}
// True if before array has some values.
// Before array must be released in the caller.
bool increase_size() {
if (capacity == 0) {
// assert(array == nullptr && count == 0);
size_t new_size = 1 << init_size_pow2;
init_array(new_size);
return false;
}
size_t new_size = capacity * 2;
init_array(new_size);
count = 0;
return true;
}
Iterator create_it(size_t index) { return Iterator(this, index); }
ConstIterator create_it(size_t index) const {
return ConstIterator(this, index);
}
public:
// Cleares all data.
void clear() {
free(array);
array = nullptr;
capacity = 0;
count = 0;
}
size_t size() const { return count; }
protected:
size_t before_index(size_t now, size_t mask) {
return (now - 1) & mask; // THIS IS VALID
}
size_t index(const K &key, size_t mask) const {
return hash(std::hash<K>()(key)) & mask;
}
// NOTE: This is rather expensive but offers good distribution.
size_t hash(size_t x) const {
x = (x ^ (x >> 30)) * UINT64_C(0xbf58476d1ce4e5b9);
x = (x ^ (x >> 27)) * UINT64_C(0x94d049bb133111eb);
x = x ^ (x >> 31);
return x;
}
size_t mask() const { return capacity - 1; }
Combined *array = nullptr;
size_t capacity = 0;
size_t count = 0;
friend class IteratorBase<Iterator>;
friend class IteratorBase<ConstIterator>;
};
/**
* HashMap with RobinHood collision resolution policy.
* Single threaded.
* Entries are saved as pointers alligned to 8B.
* Entries must know thers key.
* D must have method const K & get_key()
* K must be comparable with ==.
* HashMap behaves as if it isn't owner of entries.
* BE CAREFUL - this structure assumes that the pointer to Data is 8-alligned!
*/
template <class K, class D, size_t init_size_pow2 = 2>
class RhHashMap : public RhBase<K, D, init_size_pow2> {
typedef RhBase<K, D, init_size_pow2> base;
using base::array;
using base::capacity;
using base::count;
using base::index;
using typename base::Combined;
void increase_size() {
size_t old_size = capacity;
auto a = array;
if (base::increase_size()) {
for (int i = 0; i < old_size; i++) {
if (a[i].valid()) {
insert(a[i].ptr());
}
}
}
free(a);
}
public:
using base::RhBase;
bool contains(const K &key) { return find(key).is_present(); }
OptionPtr<D> find(const K key) {
size_t mask = this->mask();
size_t now = index(key, mask);
size_t off = 0;
size_t border = 8 <= capacity ? 8 : capacity;
while (off < border) {
Combined other = array[now];
if (other.valid()) {
auto other_off = other.off();
if (other_off == off && key == other.ptr()->get_key()) {
// Found data.
return OptionPtr<D>(other.ptr());
} else if (other_off < off) { // Other is rich
break;
} // Else other has equal or greater offset, so he is poor.
} else {
// Empty slot means that there is no searched data.
break;
}
off++;
now = (now + 1) & mask;
}
return OptionPtr<D>();
}
// Inserts element. Returns true if element wasn't in the map.
bool insert(D *data) {
CHECK(!(((uint64_t) static_cast<void *>(data) & 7)))
<< "Data is not 8-alligned.";
if (count < capacity) {
size_t mask = this->mask();
auto key = std::ref(data->get_key());
size_t now = index(key, mask);
size_t off = 0;
size_t border = 8 <= capacity ? 8 : capacity;
while (off < border) {
Combined other = array[now];
if (other.valid()) {
auto other_off = other.off();
if (other_off == off && key == other.ptr()->get_key()) {
// Element already exists.
return false;
} else if (other_off < off) { // Other is rich
// Set data.
array[now] = Combined(data, off);
// Move other data to the higher indexes,
while (other.increment_off()) {
now = (now + 1) & mask;
auto tmp = array[now];
array[now] = other;
other = tmp;
if (!other.valid()) {
count++;
return true;
}
}
data = other.ptr();
break; // Cant insert removed element because it would
// be to far from his real place.
} // Else other has equal or greater offset, so he is poor.
} else {
// Data can be placed in this empty slot.
array[now] = Combined(data, off);
count++;
return true;
}
off++;
now = (now + 1) & mask;
}
}
// There isn't enough space for element pointed by data so whe must
// increase array.
increase_size();
return insert(data);
}
// Removes element. Returns removed element if it existed.
OptionPtr<D> remove(const K &key) {
size_t mask = this->mask();
size_t now = index(key, mask);
size_t off = 0;
size_t border = 8 <= capacity ? 8 : capacity;
while (off < border) {
Combined other = array[now];
if (other.valid()) {
auto other_off = other.off();
auto other_ptr = other.ptr();
if (other_off == off && key == other_ptr->get_key()) { // Found it
auto before = now;
// Whe must move other elements one slot lower.
do {
// This is alright even for off=0 on found element
// because it wont be seen.
other.decrement_off_unsafe();
array[before] = other;
before = now;
now = (now + 1) & mask;
other = array[now];
} while (other.valid() &&
other.off() > 0); // Exit if whe encounter empty
// slot or data which is exactly
// in slot which it want's to be.
array[before] = Combined();
count--;
return OptionPtr<D>(other_ptr);
} else if (other_off < off) { // Other is rich
break;
} // Else other has equal or greater offset, so he is poor.
} else {
// If the element to be removed existed in map it would be here.
break;
}
off++;
now = (now + 1) & mask;
}
return OptionPtr<D>();
}
};

169
poc/rh_hashmap_test.cpp
View File

@ -1,169 +0,0 @@
#include "gtest/gtest.h"
#include <memory>
#include "rh_hashmap.hpp"
class Data {
private:
int key;
public:
Data(int key) : key(key) {}
const int &get_key() const { return key; }
};
void cross_validate(RhHashMap<int, Data> &map, std::map<int, Data *> &s_map);
TEST(RobinHoodHashmap, BasicFunctionality) {
RhHashMap<int, Data> map;
ASSERT_EQ(map.size(), 0);
Data d0(0);
ASSERT_EQ(map.insert(&d0), true);
ASSERT_EQ(map.size(), 1);
}
TEST(RobinHoodHashmap, RemoveFunctionality) {
RhHashMap<int, Data> map;
Data d0(0);
ASSERT_EQ(map.insert(&d0), true);
ASSERT_EQ(map.remove(0).is_present(), true);
ASSERT_EQ(map.size(), 0);
ASSERT_EQ(!map.find(0).is_present(), true);
}
TEST(RobinHoodHashmap, InsertGetCheck) {
RhHashMap<int, Data> map;
ASSERT_EQ(!map.find(0).is_present(), true);
Data d0(0);
ASSERT_EQ(map.insert(&d0), true);
ASSERT_EQ(map.find(0).is_present(), true);
ASSERT_EQ(map.find(0).get(), &d0);
}
TEST(RobinHoodHashmap, DoubleInsert) {
RhHashMap<int, Data> map;
Data d0(0);
ASSERT_EQ(map.insert(&d0), true);
ASSERT_EQ(!map.insert(&d0), true);
}
TEST(RobinHoodHashmap, FindInsertFind) {
RhHashMap<int, Data> map;
std::vector<std::unique_ptr<Data>> di;
di.reserve(128);
for (int i = 0; i < 128; ++i) di.emplace_back(std::make_unique<Data>(i));
for (int i = 0; i < 128; i++) {
ASSERT_EQ(!map.find(i).is_present(), true);
ASSERT_EQ(map.insert(di[i].get()), true);
ASSERT_EQ(map.find(i).is_present(), true);
}
for (int i = 0; i < 128; i++) {
ASSERT_EQ(map.find(i).is_present(), true);
ASSERT_EQ(map.find(i).get()->get_key(), i);
}
}
TEST(RobinHoodHashmap, Iterate) {
RhHashMap<int, Data> map;
std::vector<std::unique_ptr<Data>> di;
di.reserve(128);
for (int i = 0; i < 128; ++i) di.emplace_back(std::make_unique<Data>(i));
for (int i = 0; i < 128; i++) {
ASSERT_EQ(!map.find(i).is_present(), true);
ASSERT_EQ(map.insert(di[i].get()), true);
ASSERT_EQ(map.find(i).is_present(), true);
}
bool seen[128] = {false};
for (auto e : map) {
auto key = e->get_key();
ASSERT_EQ(!seen[key], true);
seen[key] = true;
}
for (int i = 0; i < 128; i++) {
ASSERT_EQ(seen[i], true);
}
}
TEST(RobinHoodHashmap, Checked) {
RhHashMap<int, Data> map;
std::map<int, Data *> s_map;
std::vector<std::unique_ptr<Data>> di;
std::vector<int> key;
di.reserve(128);
key.reserve(128);
for (int i = 0; i < 128; ++i) {
const int curr_key = std::rand();
key.emplace_back(curr_key);
di.emplace_back(std::make_unique<Data>(curr_key));
}
for (int i = 0; i < 128; i++) {
if (map.insert(di[i].get())) {
ASSERT_EQ(s_map.find(key[i]), s_map.end());
s_map[key[i]] = di[i].get();
} else {
ASSERT_NE(s_map.find(key[i]), s_map.end());
}
}
cross_validate(map, s_map);
}
TEST(RobinHoodHashMap, CheckWithRemove) {
RhHashMap<int, Data> map;
std::map<int, Data *> s_map;
std::vector<std::unique_ptr<Data>> di;
std::vector<int> key;
di.reserve(1280);
key.reserve(1280);
for (int i = 0; i < 1280; ++i) {
const int curr_key = std::rand() % 100;
key.emplace_back(curr_key);
di.emplace_back(std::make_unique<Data>(curr_key));
}
for (int i = 0; i < 1280; i++) {
if (map.insert(di[i].get())) {
ASSERT_EQ(s_map.find(key[i]), s_map.end());
s_map[key[i]] = di[i].get();
cross_validate(map, s_map);
} else {
ASSERT_EQ(map.remove(key[i]).is_present(), true);
ASSERT_EQ(s_map.erase(key[i]), 1);
cross_validate(map, s_map);
}
}
cross_validate(map, s_map);
}
TEST(RobinhoodHashmmap, AlignmentCheck) {
RhHashMap<int, Data> map;
char *block = static_cast<char *>(std::malloc(20));
++block; // not alligned - offset 1
EXPECT_DEATH(map.insert((Data *)(block)), "not 8-alligned");
}
void cross_validate(RhHashMap<int, Data> &map, std::map<int, Data *> &s_map) {
for (auto e : map) {
ASSERT_NE(s_map.find(e->get_key()), s_map.end());
}
for (auto e : s_map) {
ASSERT_EQ(map.find(e.first).get(), e.second);
}
}
int main(int argc, char **argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

29
poc/stack_allocator_test.cpp
View File

@ -1,29 +0,0 @@
#include "gtest/gtest.h"
#include "memory/stack_allocator.hpp"
struct Object {
int a;
int b;
Object(int a, int b) : a(a), b(b) {}
};
TEST(StackAllocatorTest, AllocationAndObjectValidity) {
utils::StackAllocator allocator;
for (int i = 0; i < 64 * 1024; ++i) {
auto object = allocator.make<Object>(1, 2);
ASSERT_EQ(object->a, 1);
ASSERT_EQ(object->b, 2);
}
}
TEST(StackAllocatorTest, CountMallocAndFreeCalls) {
// TODO: implementation
EXPECT_EQ(true, true);
}
int main(int argc, char **argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}

158
poc/timer.hpp
View File

@ -1,158 +0,0 @@
#pragma once
#include <atomic>
#include <chrono>
#include <functional>
#include <memory>
#include <set>
#include <thread>
#include <glog/logging.h>
namespace utils {
/**
* @class Timer
*
* @brief The timer contains counter and handler which is executed when the time
* exceedes.
*
* With every clock interval the counter should be decreased for
* delta count. Delta count is one for now but it should be a variable in the
* near future. The handler is function that will be called when counter
* becomes zero or smaller than zero.
*/
struct Timer {
using sptr = std::shared_ptr<Timer>;
using handler_t = std::function<void(void)>;
Timer(int64_t counter, handler_t handler)
: counter(counter), handler(handler) {}
bool operator--() {
if (--counter <= 0)
return true;
else
return false;
}
int64_t counter;
handler_t handler;
};
/**
* Timer container knows how to add a new timer and remove the
* existing container from itself. Also, time container object
* has the process method whose responsibility is to iterate
* over existing timers and call the appropriate handler function.
* The handler method could be called on the same thread, on a
* separate thread or on a thread pool, that is implementation detail of
* the process method.
*/
/**
* @class TimerSet
*
* @brief Trivial timer container implementation.
*
* Internal data stucture for storage of timers is std::set. So, the
* related timer complexities are:
* insertion: O(log(n))
* deletion: O(log(n))
* process: O(n)
*/
class TimerSet {
public:
void add(Timer::sptr timer) { timers.insert(timer); }
void remove(Timer::sptr timer) { timers.erase(timer); }
uint64_t size() const { return timers.size(); }
void process() {
for (auto it = timers.begin(); it != timers.end();) {
auto timer = *it;
if (--*timer) {
timer->handler();
it = timers.erase(it);
continue;
}
++it;
}
}
private:
std::set<std::shared_ptr<Timer>> timers;
};
/**
* @class TimerScheduler
*
* @brief TimerScheduler is a manager class and its responsibility is to
* take care of the time and call the timer_container process method in the
* appropriate time.
*
* @tparam timer_container_type implements a strategy how the timers
* are processed
* @tparam delta_time_type type of a time distance between two events
* @tparam delta_time granularity between the two events, default value is 1
*/
template <typename timer_container_type, typename delta_time_type,
uint64_t delta_time = 1>
class TimerScheduler {
public:
/**
* Adds a timer.
*
* @param timer shared pointer to the timer object \ref Timer
*/
void add(Timer::sptr timer) { timer_container.add(timer); }
/**
* Removes a timer.
*
* @param timer shared pointer to the timer object \ref Timer
*/
void remove(Timer::sptr timer) { timer_container.remove(timer); }
/**
* Provides the number of pending timers. The exact number has to be
* provided by a timer_container.
*
* @return uint64_t the number of pending timers.
*/
uint64_t size() const { return timer_container.size(); }
/**
* Runs a separate thread which responsibility is to run the process method
* at the appropriate time (every delta_time from the beginning of
* processing.
*/
void run() {
is_running.store(true);
run_thread = std::thread([this]() {
while (is_running.load()) {
std::this_thread::sleep_for(delta_time_type(delta_time));
timer_container.process();
DLOG(INFO) << "timer_container.process()";
}
});
}
/**
* Stops the whole processing.
*/
void stop() { is_running.store(false); }
/**
* Joins the processing thread.
*/
~TimerScheduler() { run_thread.join(); }
private:
timer_container_type timer_container;
std::thread run_thread;
std::atomic<bool> is_running;
};
} // namespace utils

104
poc/utils/copy_on_write.hpp
View File

@ -1,104 +0,0 @@
#pragma once
#include <memory>
namespace utils {
/**
* Helper class for implementing copy-on-write member variables. Memory
* management is automatic via shared-ptr: it is allowed for the original
* owner to expire, the content will remain valid in copies.
*
* This class is generally not thread-safe. However, it is intended for use in
* Memgraph's MVCC which is concurrent, but with specific guarantees that also
* make it possible to use this. These guarantees are:
* 1. Writing can only happen when there are no copies (no parallel reads). An
* implication of this is that an obtained copy is immutable, even though in
* general CopyOnWrite does not guarantee this (generally the owner's
* modificatins will be visible to the copies).
* 2. Copies are not created in parallel. MVCC guarantees this with
* record-locking, but should be in fact legal in CopyOnWrite.
*
* @tparam TElement - type of content. Must be copy-constructable.
*/
template <typename TElement>
class CopyOnWrite {
public:
/**
* Creates a new CopyOnWrite that owns it's element.
*
* @param args - Arguments forwarded to the TElement constructor.
* @tparam TArgs - Argument types.
*/
template <typename... TArgs>
CopyOnWrite(TArgs &&... args) {
TElement *new_element = new TElement(std::forward<TArgs>(args)...);
element_ptr_.reset(new_element);
is_owner_ = true;
}
/** Creates a copy of the given CopyOnWrite object that does not copy the
* element and does not assume ownership over it. */
CopyOnWrite(const CopyOnWrite &other)
: element_ptr_{other.element_ptr_}, is_owner_{false} {}
/** Creates a copy of the given CopyOnWrite object that does not copy the
* element and does not assume ownership over it. This is a non-const
* reference accepting copy constructor. The hack is necessary to prevent the
* variadic constructor from being a better match for non-const CopyOnWrite
* argument. */
CopyOnWrite(CopyOnWrite &other)
: CopyOnWrite(const_cast<const CopyOnWrite &>(other)) {}
/** Creates a copy of the given temporary CyopOnWrite object. If the temporary
* is owner then ownership is transferred to this CopyOnWrite. Otherwise this
* CopyOnWrite does not become the owner. */
CopyOnWrite(CopyOnWrite &&other) = default;
/** Copy assignment of another CopyOnWrite. Does not transfer ownership (this
* CopyOnWrite is not the owner). */
CopyOnWrite &operator=(const CopyOnWrite &other) {
element_ptr_ = other.element_ptr_;
is_owner_ = false;
return *this;
}
/** Copy assignment of a temporary CopyOnWrite. If the temporary is owner then
* ownerships is transferred to this CopyOnWrite. Otherwise this CopyOnWrite
* does not become the owner. */
CopyOnWrite &operator=(CopyOnWrite &&other) = default;
// All the dereferencing operators are overloaded to return a const element
// reference. There is no differentiation between const and non-const member
// function behavior because an implicit copy creation on non-const
// dereferencing would most likely result in excessive copying. For that
// reason
// an explicit call to the `Write` function is required to obtain a non-const
// reference to element.
const TElement &operator*() { return *element_ptr_; }
const TElement &operator*() const { return *element_ptr_; }
const TElement &get() { return *element_ptr_; }
const TElement &get() const { return *element_ptr_; }
const TElement *operator->() { return element_ptr_.get(); }
const TElement *operator->() const { return element_ptr_.get(); }
/** Indicates if this CopyOnWrite object is the owner of it's element. */
bool is_owner() const { return is_owner_; };
/**
* If this CopyOnWrite is the owner of it's element, a non-const reference to
* is returned. If this CopyOnWrite is not the owner, then the element is
* copied and this CopyOnWrite becomes the owner.
*/
TElement &Write() {
if (is_owner_) return *element_ptr_;
element_ptr_ = std::shared_ptr<TElement>(new TElement(*element_ptr_));
is_owner_ = true;
return *element_ptr_;
};
private:
std::shared_ptr<TElement> element_ptr_;
bool is_owner_{false};
};
} // namespace utils

67
poc/utils/utils_copy_on_write.cpp
View File

@ -1,67 +0,0 @@
#include "gtest/gtest.h"
#include "utils/copy_on_write.hpp"
TEST(CopyOneWrite, Dereferencing) {
utils::CopyOnWrite<std::string> number("42");
EXPECT_EQ((*number).size(), 2);
EXPECT_EQ(number.get().size(), 2);
EXPECT_EQ(number->size(), 2);
}
TEST(CopyOneWrite, Ownership) {
utils::CopyOnWrite<std::string> number("42");
EXPECT_TRUE(number.is_owner());
utils::CopyOnWrite<std::string> copy_constructed(number);
auto copy_assigned = number;
EXPECT_FALSE(copy_constructed.is_owner());
EXPECT_FALSE(copy_assigned.is_owner());
EXPECT_TRUE(number.is_owner());
}
TEST(CopyOneWrite, OwnershipFromTemporary) {
utils::CopyOnWrite<std::string> copy_constructed(
utils::CopyOnWrite<std::string>("42"));
auto copy_assigned = utils::CopyOnWrite<std::string>("42");
EXPECT_TRUE(copy_constructed.is_owner());
EXPECT_TRUE(copy_assigned.is_owner());
}
struct DestructorCounter {
DestructorCounter(int &counter) : counter_(counter) {}
~DestructorCounter() {
counter_++;
}
private:
int &counter_;
};
TEST(CopyOnWrite, ElementDestruction) {
int counter = 0;
std::vector<utils::CopyOnWrite<DestructorCounter>> initial_owner;
initial_owner.emplace_back(counter);
{
auto copy = initial_owner[0];
EXPECT_EQ(counter, 0);
initial_owner.clear();
EXPECT_EQ(counter, 0);
}
EXPECT_EQ(counter, 1);
}
TEST(CopyOneWrite, Copy) {
utils::CopyOnWrite<std::string> number("42");
auto copy = number;
EXPECT_EQ(*copy, "42");
// modification by owner
number.Write().resize(1);
EXPECT_EQ(*number, "4");
EXPECT_EQ(*copy, "4");
// modification by copy
copy.Write().resize(2, '3');
EXPECT_EQ(*number, "4");
EXPECT_EQ(*copy, "43");
}

1
tools/apollo/apollo_runs.yaml
View File

@ -7,7 +7,6 @@
- cppcheck # cppcheck script
- ../../src # src source dir
- ../../tests # tests source dir
- ../../poc # poc source dir
- ../../.git # git directory (used to find out changed files in commit)
- ../../.clang-format # clang-format config file
outfile_paths: