From beaba0fc16145f0d8f19b91fc4214026e5319f2b Mon Sep 17 00:00:00 2001
From: Katarina Supe <61758502+katarinasupe@users.noreply.github.com>
Date: Tue, 7 Feb 2023 15:43:41 +0100
Subject: [PATCH 1/7] Update README with Cloud, Lab and import info (#768)
---
README.md | 52 +++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 41 insertions(+), 11 deletions(-)
diff --git a/README.md b/README.md
index 820728ad0..da30c2b55 100644
--- a/README.md
+++ b/README.md
@@ -56,6 +56,17 @@ to ensure that you’re getting the [best possible
performance](http://memgraph.com/benchgraph) consistently and without surprises.
It’s also ACID-compliant and highly available.
+## :zap: Features
+
+- Run Python, Rust, and C/C++ code natively, check out the
+ [MAGE](https://github.com/memgraph/mage) graph algorithm library
+- Native support for machine learning
+- Streaming support
+- Replication
+- Authentication and authorization
+- ACID compliance
+
+
## :video_game: Memgraph Playground
You don't need to install anything to try out Memgraph. Check out
@@ -91,15 +102,34 @@ You can find the binaries and Docker images on the [Download
Hub](https://memgraph.com/download) and the installation instructions in the
[official documentation](https://memgraph.com/docs/memgraph/installation).
-## :zap: Features
-- Run Python, Rust, and C/C++ code natively, check out the
- [MAGE](https://github.com/memgraph/mage) graph algorithm library
-- Native support for machine learning
-- Streaming support
-- Replication
-- Authentication and authorization
-- ACID compliance
+## :cloud: Memgraph Cloud
+
+Check out [Memgraph Cloud](https://memgraph.com/docs/memgraph-cloud) - a cloud service fully managed on AWS and available in 6 geographic regions around the world. Memgraph Cloud allows you to create projects with Enterprise instances of MemgraphDB from your browser.
+
+<p align="left">
+ <a href="https://memgraph.com/docs/memgraph-cloud">
+ <img width="450px" alt="Memgraph Cloud" src="https://public-assets.memgraph.com/memgraph-gifs%2Fcloud.gif">
+ </a>
+</p>
+
+## :link: Connect to Memgraph
+
+[Connect to the database](https://memgraph.com/docs/memgraph/connect-to-memgraph) using Memgraph Lab, mgconsole, various drivers (Python, C/C++ and others) and WebSocket.
+
+### :microscope: Memgraph Lab
+
+Visualize graphs and play with queries to understand your data. [Memgraph Lab](https://memgraph.com/docs/memgraph-lab) is a user interface that helps you explore and manipulate the data stored in Memgraph. Visualize graphs, execute ad hoc queries, and optimize their performance.
+
+<p align="left">
+ <a href="https://memgraph.com/docs/memgraph-lab">
+ <img width="450px" alt="Memgraph Cloud" src="https://public-assets.memgraph.com/memgraph-gifs%2Flab.gif">
+ </a>
+</p>
+
+## :file_folder: Import data
+
+[Import data](https://memgraph.com/docs/memgraph/import-data) into Memgraph using Kafka, RedPanda or Pulsar streams, CSV and JSON files, or Cypher commands.
## :bookmark_tabs: Documentation
@@ -143,17 +173,17 @@ Memgraph Community is available under the [BSL
license](./licenses/BSL.txt).</br> Memgraph Enterprise is available under the
[MEL license](./licenses/MEL.txt).
-## 🙋 Community
+## :busts_in_silhouette: Community
- :purple_heart: [**Discord**](https://discord.gg/memgraph)
+- :ocean: [**Stack Overflow**](https://stackoverflow.com/questions/tagged/memgraphdb)
- :busts_in_silhouette: [**Discourse forum**](https://discourse.memgraph.com/)
-- :open_file_folder: [**Memgraph GitHub**](https://github.com/memgraph)
- :bird: [**Twitter**](https://twitter.com/memgraphdb)
- :movie_camera:
[**YouTube**](https://www.youtube.com/channel/UCZ3HOJvHGxtQ_JHxOselBYg)
<p align="center">
<a href="#">
- <img src="https://img.shields.io/badge/⬆️back_to_top_⬆️-white" alt="Back to top" title="Back to top"/>
+ <img src="https://img.shields.io/badge/⬆️ back_to_top_⬆️-white" alt="Back to top" title="Back to top"/>
</a>
</p>
From 15c8662023a329ed4a92d0a123106edc8ee4ff69 Mon Sep 17 00:00:00 2001
From: Jure Bajic <jure.bajic@memgraph.com>
Date: Fri, 17 Feb 2023 10:47:36 +0100
Subject: [PATCH 2/7] Add support for Fedora 36 (#787)
---
.github/workflows/package_all.yaml | 17 ++++++++++++++
environment/os/fedora-36.sh | 7 +++++-
environment/toolchain/v4.sh | 33 ++++++++++++++++++++++++++--
init | 4 ++--
release/CMakeLists.txt | 2 +-
release/package/docker-compose.yml | 4 ++++
release/package/fedora-36/Dockerfile | 15 +++++++++++++
release/package/run.sh | 6 ++---
release/rpm/memgraph.spec.in | 6 +++--
release/rpm/rpmlintrc | 4 ++++
10 files changed, 87 insertions(+), 11 deletions(-)
create mode 100644 release/package/fedora-36/Dockerfile
create mode 100644 release/rpm/rpmlintrc
diff --git a/.github/workflows/package_all.yaml b/.github/workflows/package_all.yaml
index 0a7cd891a..6e1196e32 100644
--- a/.github/workflows/package_all.yaml
+++ b/.github/workflows/package_all.yaml
@@ -176,3 +176,20 @@ jobs:
with:
name: debian-11-arm
path: build/output/debian-11-arm/memgraph*.deb
+
+ fedora-36:
+ runs-on: [self-hosted, DockerMgBuild, X64]
+ timeout-minutes: 60
+ steps:
+ - name: "Set up repository"
+ uses: actions/checkout@v3
+ with:
+ fetch-depth: 0 # Required because of release/get_version.py
+ - name: "Build package"
+ run: |
+ ./release/package/run.sh package fedora-36
+ - name: "Upload package"
+ uses: actions/upload-artifact@v3
+ with:
+ name: fedora-36
+ path: build/output/fedora-36/memgraph*.rpm
diff --git a/environment/os/fedora-36.sh b/environment/os/fedora-36.sh
index 8161e8e52..c59881f65 100755
--- a/environment/os/fedora-36.sh
+++ b/environment/os/fedora-36.sh
@@ -26,6 +26,7 @@ TOOLCHAIN_BUILD_DEPS=(
diffutils
libipt libipt-devel # intel
patch
+ perl # for openssl
)
TOOLCHAIN_RUN_DEPS=(
@@ -36,7 +37,6 @@ TOOLCHAIN_RUN_DEPS=(
readline # for cmake and llvm
libffi libxml2 # for llvm
openssl-devel
- perl # for openssl
)
MEMGRAPH_BUILD_DEPS=(
@@ -64,6 +64,10 @@ list() {
check() {
local missing=""
+ # On Fedora yum/dnf and python10 use newer glibc which is not compatible
+ # with ours, so we need to momentarely disable env
+ local OLD_LD_LIBRARY_PATH=${LD_LIBRARY_PATH}
+ LD_LIBRARY_PATH=""
for pkg in $1; do
if ! dnf list installed "$pkg" >/dev/null 2>/dev/null; then
missing="$pkg $missing"
@@ -73,6 +77,7 @@ check() {
echo "MISSING PACKAGES: $missing"
exit 1
fi
+ LD_LIBRARY_PATH=${OLD_LD_LIBRARY_PATH}
}
install() {
diff --git a/environment/toolchain/v4.sh b/environment/toolchain/v4.sh
index 259804fa9..9dd997ad4 100755
--- a/environment/toolchain/v4.sh
+++ b/environment/toolchain/v4.sh
@@ -379,6 +379,34 @@ if [ ! -f $PREFIX/bin/gdb ]; then
--without-babeltrace \
--enable-tui \
--with-python=python3
+ elif [[ "${DISTRO}" == fedora* ]]; then
+ # Remove readline, gdb does not compile
+ env \
+ CC=gcc \
+ CXX=g++ \
+ CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" \
+ CXXFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" \
+ CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2 -fPIC" \
+ LDFLAGS="-Wl,-z,relro" \
+ PYTHON="" \
+ ../configure \
+ --build=x86_64-linux-gnu \
+ --host=x86_64-linux-gnu \
+ --prefix=$PREFIX \
+ --disable-maintainer-mode \
+ --disable-dependency-tracking \
+ --disable-silent-rules \
+ --disable-gdbtk \
+ --disable-shared \
+ --without-guile \
+ --with-system-gdbinit=$PREFIX/etc/gdb/gdbinit \
+ --with-expat \
+ --with-system-zlib \
+ --with-lzma \
+ --with-babeltrace \
+ --with-intel-pt \
+ --enable-tui \
+ --with-python=python3
else
# https://buildd.debian.org/status/fetch.php?pkg=gdb&arch=amd64&ver=8.2.1-2&stamp=1550831554&raw=0
env \
@@ -651,6 +679,7 @@ export PS1="($NAME) \$PS1"
export LD_LIBRARY_PATH=$PREFIX/lib:$PREFIX/lib64
export CXXFLAGS=-isystem\ $PREFIX/include\ \$CXXFLAGS
export CFLAGS=-isystem\ $PREFIX/include\ \$CFLAGS
+export VENV=$PREFIX
# disable root
function su () {
@@ -702,7 +731,7 @@ PROXYGEN_SHA256=5360a8ccdfb2f5a6c7b3eed331ec7ab0e2c792d579c6fff499c85c516c11fe14
SNAPPY_SHA256=75c1fbb3d618dd3a0483bff0e26d0a92b495bbe5059c8b4f1c962b478b6e06e7
SNAPPY_VERSION=1.1.9
XZ_VERSION=5.2.5 # for LZMA
-ZLIB_VERSION=1.2.12
+ZLIB_VERSION=1.2.13
ZSTD_VERSION=1.5.0
WANGLE_SHA256=1002e9c32b6f4837f6a760016e3b3e22f3509880ef3eaad191c80dc92655f23f
@@ -1226,7 +1255,7 @@ popd
# create toolchain archive
if [ ! -f $NAME-binaries-$DISTRO.tar.gz ]; then
DISTRO_FULL_NAME=${DISTRO}
- if [[ "${DISTRO}" == centos* ]]; then
+ if [[ "${DISTRO}" == centos* ]] || [[ "${DISTRO}" == fedora* ]]; then
if [[ "$for_arm" = "true" ]]; then
DISTRO_FULL_NAME="$DISTRO_FULL_NAME-aarch64"
else
diff --git a/init b/init
index adaafdeff..a4754fb3e 100755
--- a/init
+++ b/init
@@ -113,8 +113,8 @@ if [[ "$setup_libs" == "true" ]]; then
fi
# Fix for centos 7 during release
-if [ "${ARCHITECTURE}" = "centos-7" ]; then
- python3 -m pip uninstall virtualenv
+if [ "${DISTRO}" = "centos-7" ] || [ "${DISTRO}" = "debian-11" ]; then
+ python3 -m pip uninstall -y virtualenv
python3 -m pip install virtualenv
fi
diff --git a/release/CMakeLists.txt b/release/CMakeLists.txt
index 2715d39c3..489aea989 100644
--- a/release/CMakeLists.txt
+++ b/release/CMakeLists.txt
@@ -67,7 +67,7 @@ It aims to deliver developers the speed, simplicity and scale required to build
the next generation of applications driver by real-time connected data.")
# Add `openssl` package to dependencies list. Used to generate SSL certificates.
# We also depend on `python3` because we embed it in Memgraph.
-set(CPACK_RPM_PACKAGE_REQUIRES "openssl >= 1.0.0, curl >= 7.29.0, python3 >= 3.5.0, libstdc >= 6, logrotate")
+set(CPACK_RPM_PACKAGE_REQUIRES "openssl >= 1.0.0, curl >= 7.29.0, python3 >= 3.5.0, libstdc++ >= 6, logrotate")
# All variables must be set before including.
include(CPack)
diff --git a/release/package/docker-compose.yml b/release/package/docker-compose.yml
index 2f073267e..1f285bf84 100644
--- a/release/package/docker-compose.yml
+++ b/release/package/docker-compose.yml
@@ -28,3 +28,7 @@ services:
build:
context: ubuntu-22.04
container_name: "mgbuild_ubuntu-22.04"
+ mgbuild_fedora-36:
+ build:
+ context: fedora-36
+ container_name: "mgbuild_fedora-36"
diff --git a/release/package/fedora-36/Dockerfile b/release/package/fedora-36/Dockerfile
new file mode 100644
index 000000000..8e2f8a6fe
--- /dev/null
+++ b/release/package/fedora-36/Dockerfile
@@ -0,0 +1,15 @@
+FROM fedora:36
+
+ARG TOOLCHAIN_VERSION
+
+# Stops tzdata interactive configuration.
+RUN yum -y update \
+ && yum install -y wget git
+# Do NOT be smart here and clean the cache because the container is used in the
+# stateful context.
+
+RUN wget -q https://s3-eu-west-1.amazonaws.com/deps.memgraph.io/${TOOLCHAIN_VERSION}/${TOOLCHAIN_VERSION}-binaries-fedora-36-x86_64.tar.gz \
+ -O ${TOOLCHAIN_VERSION}-binaries-fedora-36-x86_64.tar.gz \
+ && tar xzvf ${TOOLCHAIN_VERSION}-binaries-fedora-36-x86_64.tar.gz -C /opt
+
+ENTRYPOINT ["sleep", "infinity"]
diff --git a/release/package/run.sh b/release/package/run.sh
index 7d0b9848c..680e7a4db 100755
--- a/release/package/run.sh
+++ b/release/package/run.sh
@@ -3,7 +3,7 @@
set -Eeuo pipefail
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-SUPPORTED_OS=(centos-7 centos-9 debian-10 debian-11 ubuntu-18.04 ubuntu-20.04 ubuntu-22.04 debian-11-arm)
+SUPPORTED_OS=(centos-7 centos-9 debian-10 debian-11 ubuntu-18.04 ubuntu-20.04 ubuntu-22.04 debian-11-arm fedora-36)
PROJECT_ROOT="$SCRIPT_DIR/../.."
TOOLCHAIN_VERSION="toolchain-v4"
ACTIVATE_TOOLCHAIN="source /opt/${TOOLCHAIN_VERSION}/activate"
@@ -23,9 +23,9 @@ make_package () {
echo "Building Memgraph for $os on $build_container..."
package_command=""
- if [[ "$os" =~ ^"centos".* ]]; then
+ if [[ "$os" =~ ^"centos".* ]] || [[ "$os" =~ ^"fedora".* ]]; then
docker exec "$build_container" bash -c "yum -y update"
- package_command=" cpack -G RPM --config ../CPackConfig.cmake && rpmlint memgraph*.rpm "
+ package_command=" cpack -G RPM --config ../CPackConfig.cmake && rpmlint --file='../../release/rpm/rpmlintrc' memgraph*.rpm "
fi
if [[ "$os" =~ ^"debian".* ]]; then
docker exec "$build_container" bash -c "apt update"
diff --git a/release/rpm/memgraph.spec.in b/release/rpm/memgraph.spec.in
index 448f884ac..d663ce953 100644
--- a/release/rpm/memgraph.spec.in
+++ b/release/rpm/memgraph.spec.in
@@ -30,7 +30,7 @@ BuildRequires: systemd
# This is needed to prevent Python compilation errors when building the RPM
# package
# https://github.com/scylladb/scylla/issues/2235
-%if 0%{?rhel} < 8
+%if 0%{?rhel} && 0%{?rhel} < 8
%global __os_install_post \
/usr/lib/rpm/redhat/brp-compress \
%{!?__debug_package:\
@@ -40,7 +40,9 @@ BuildRequires: systemd
/usr/lib/rpm/redhat/brp-strip-static-archive %{__strip} \
%{!?__jar_repack:/usr/lib/rpm/redhat/brp-java-repack-jars} \
%{nil}
-%else
+%endif
+
+%if 0%{?fedora} && 0%{?fedora} < 35
%global __os_install_post \
/usr/lib/rpm/brp-compress \
%{!?__debug_package:\
diff --git a/release/rpm/rpmlintrc b/release/rpm/rpmlintrc
new file mode 100644
index 000000000..1de91a77b
--- /dev/null
+++ b/release/rpm/rpmlintrc
@@ -0,0 +1,4 @@
+# from https://github.com/google/earthenterprise/blob/master/earth_enterprise/rpmlintrc
+
+# We are not packaging log dir
+addFilter("E: logrotate-log-dir-not-packaged")
From bbce21e78f786988bcba0cbd51a78f646ef00ed5 Mon Sep 17 00:00:00 2001
From: Antonio Filipovic <61245998+antoniofilipovic@users.noreply.github.com>
Date: Fri, 17 Feb 2023 11:50:17 +0100
Subject: [PATCH 3/7] Update pull request template (#775)
---
.github/pull_request_template.md | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index d96508555..820cc93c7 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -1,11 +1,14 @@
[master < Epic] PR
- [ ] Check, and update documentation if necessary
-- [ ] Update [changelog](https://docs.memgraph.com/memgraph/changelog)
- [ ] Write E2E tests
- [ ] Compare the [benchmarking results](https://bench-graph.memgraph.com/) between the master branch and the Epic branch
- [ ] Provide the full content or a guide for the final git message
[master < Task] PR
- [ ] Check, and update documentation if necessary
-- [ ] Update [changelog](https://docs.memgraph.com/memgraph/changelog)
- [ ] Provide the full content or a guide for the final git message
+
+
+To keep docs changelog up to date, one more thing to do:
+- [ ] Write a release note here
+- [ ] Tag someone from docs team in the comments
From 862a1afdf15838ea3b0fabfd5a8ce200be2bf2ee Mon Sep 17 00:00:00 2001
From: Antonio Filipovic <61245998+antoniofilipovic@users.noreply.github.com>
Date: Fri, 17 Feb 2023 13:09:25 +0100
Subject: [PATCH 4/7] Improve Visit performance (#774)
---
src/query/frontend/ast/ast.lcp | 25 ++++
src/query/interpret/eval.hpp | 154 ++++++++++++++++------
tests/unit/query_expression_evaluator.cpp | 44 ++++++-
3 files changed, 181 insertions(+), 42 deletions(-)
diff --git a/src/query/frontend/ast/ast.lcp b/src/query/frontend/ast/ast.lcp
index 577d15113..a1f3c8ef5 100644
--- a/src/query/frontend/ast/ast.lcp
+++ b/src/query/frontend/ast/ast.lcp
@@ -301,6 +301,7 @@ cpp<#
(lcp:define-class expression (tree "::utils::Visitable<HierarchicalTreeVisitor>"
"::utils::Visitable<ExpressionVisitor<TypedValue>>"
+ "::utils::Visitable<ExpressionVisitor<TypedValue*>>"
"::utils::Visitable<ExpressionVisitor<void>>")
()
(:abstractp t)
@@ -308,6 +309,7 @@ cpp<#
#>cpp
using utils::Visitable<HierarchicalTreeVisitor>::Accept;
using utils::Visitable<ExpressionVisitor<TypedValue>>::Accept;
+ using utils::Visitable<ExpressionVisitor<TypedValue*>>::Accept;
using utils::Visitable<ExpressionVisitor<void>>::Accept;
Expression() = default;
@@ -407,6 +409,7 @@ cpp<#
(:public
#>cpp
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -438,6 +441,7 @@ cpp<#
(:public
#>cpp
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -485,6 +489,7 @@ cpp<#
}
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -538,6 +543,7 @@ cpp<#
ListSlicingOperator() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -581,6 +587,7 @@ cpp<#
IfOperator() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -628,6 +635,7 @@ cpp<#
PrimitiveLiteral() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
DEFVISITABLE(HierarchicalTreeVisitor);
cpp<#)
@@ -656,6 +664,7 @@ cpp<#
ListLiteral() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -688,6 +697,7 @@ cpp<#
MapLiteral() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -720,6 +730,7 @@ cpp<#
Identifier() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
DEFVISITABLE(HierarchicalTreeVisitor);
@@ -757,6 +768,7 @@ cpp<#
PropertyLookup() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -797,6 +809,7 @@ cpp<#
LabelsTest() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -837,6 +850,7 @@ cpp<#
Function() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -892,6 +906,7 @@ cpp<#
Reduce() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -930,6 +945,7 @@ cpp<#
Coalesce() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -969,6 +985,7 @@ cpp<#
Extract() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1005,6 +1022,7 @@ cpp<#
All() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1046,6 +1064,7 @@ cpp<#
Single() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1087,6 +1106,7 @@ cpp<#
Any() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1128,6 +1148,7 @@ cpp<#
None() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1159,6 +1180,7 @@ cpp<#
ParameterLookup() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
DEFVISITABLE(HierarchicalTreeVisitor);
cpp<#)
@@ -1186,6 +1208,7 @@ cpp<#
RegexMatch() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
@@ -1205,6 +1228,7 @@ cpp<#
(lcp:define-class named-expression (tree "::utils::Visitable<HierarchicalTreeVisitor>"
"::utils::Visitable<ExpressionVisitor<TypedValue>>"
+ "::utils::Visitable<ExpressionVisitor<TypedValue*>>"
"::utils::Visitable<ExpressionVisitor<void>>")
((name "std::string" :scope :public)
(expression "Expression *" :initval "nullptr" :scope :public
@@ -1223,6 +1247,7 @@ cpp<#
NamedExpression() = default;
DEFVISITABLE(ExpressionVisitor<TypedValue>);
+ DEFVISITABLE(ExpressionVisitor<TypedValue*>);
DEFVISITABLE(ExpressionVisitor<void>);
bool Accept(HierarchicalTreeVisitor &visitor) override {
if (visitor.PreVisit(*this)) {
diff --git a/src/query/interpret/eval.hpp b/src/query/interpret/eval.hpp
index 1ea20846d..74df223a5 100644
--- a/src/query/interpret/eval.hpp
+++ b/src/query/interpret/eval.hpp
@@ -1,4 +1,4 @@
-// Copyright 2022 Memgraph Ltd.
+// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -31,6 +31,71 @@
namespace memgraph::query {
+class ReferenceExpressionEvaluator : public ExpressionVisitor<TypedValue *> {
+ public:
+ ReferenceExpressionEvaluator(Frame *frame, const SymbolTable *symbol_table, const EvaluationContext *ctx)
+ : frame_(frame), symbol_table_(symbol_table), ctx_(ctx) {}
+
+ using ExpressionVisitor<TypedValue *>::Visit;
+
+ utils::MemoryResource *GetMemoryResource() const { return ctx_->memory; }
+
+#define UNSUCCESSFUL_VISIT(expr_name) \
+ TypedValue *Visit(expr_name &expr) override { return nullptr; }
+
+ TypedValue *Visit(Identifier &ident) override { return &frame_->at(symbol_table_->at(ident)); }
+
+ UNSUCCESSFUL_VISIT(NamedExpression);
+ UNSUCCESSFUL_VISIT(OrOperator);
+ UNSUCCESSFUL_VISIT(XorOperator);
+ UNSUCCESSFUL_VISIT(AdditionOperator);
+ UNSUCCESSFUL_VISIT(SubtractionOperator);
+ UNSUCCESSFUL_VISIT(MultiplicationOperator);
+ UNSUCCESSFUL_VISIT(DivisionOperator);
+ UNSUCCESSFUL_VISIT(ModOperator);
+ UNSUCCESSFUL_VISIT(NotEqualOperator);
+ UNSUCCESSFUL_VISIT(EqualOperator);
+ UNSUCCESSFUL_VISIT(LessOperator);
+ UNSUCCESSFUL_VISIT(GreaterOperator);
+ UNSUCCESSFUL_VISIT(LessEqualOperator);
+ UNSUCCESSFUL_VISIT(GreaterEqualOperator);
+
+ UNSUCCESSFUL_VISIT(NotOperator);
+ UNSUCCESSFUL_VISIT(UnaryPlusOperator);
+ UNSUCCESSFUL_VISIT(UnaryMinusOperator);
+
+ UNSUCCESSFUL_VISIT(AndOperator);
+ UNSUCCESSFUL_VISIT(IfOperator);
+ UNSUCCESSFUL_VISIT(InListOperator);
+
+ UNSUCCESSFUL_VISIT(SubscriptOperator);
+
+ UNSUCCESSFUL_VISIT(ListSlicingOperator);
+ UNSUCCESSFUL_VISIT(IsNullOperator);
+ UNSUCCESSFUL_VISIT(PropertyLookup);
+ UNSUCCESSFUL_VISIT(LabelsTest);
+
+ UNSUCCESSFUL_VISIT(PrimitiveLiteral);
+ UNSUCCESSFUL_VISIT(ListLiteral);
+ UNSUCCESSFUL_VISIT(MapLiteral);
+ UNSUCCESSFUL_VISIT(Aggregation);
+ UNSUCCESSFUL_VISIT(Coalesce);
+ UNSUCCESSFUL_VISIT(Function);
+ UNSUCCESSFUL_VISIT(Reduce);
+ UNSUCCESSFUL_VISIT(Extract);
+ UNSUCCESSFUL_VISIT(All);
+ UNSUCCESSFUL_VISIT(Single);
+ UNSUCCESSFUL_VISIT(Any);
+ UNSUCCESSFUL_VISIT(None);
+ UNSUCCESSFUL_VISIT(ParameterLookup);
+ UNSUCCESSFUL_VISIT(RegexMatch);
+
+ private:
+ Frame *frame_;
+ const SymbolTable *symbol_table_;
+ const EvaluationContext *ctx_;
+};
+
class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
public:
ExpressionEvaluator(Frame *frame, const SymbolTable &symbol_table, const EvaluationContext &ctx, DbAccessor *dba,
@@ -159,50 +224,53 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
TypedValue Visit(SubscriptOperator &list_indexing) override {
- auto lhs = list_indexing.expression1_->Accept(*this);
+ ReferenceExpressionEvaluator referenceExpressionEvaluator(frame_, symbol_table_, ctx_);
+
+ TypedValue *lhs_ptr = list_indexing.expression1_->Accept(referenceExpressionEvaluator);
+ TypedValue lhs;
+ const auto referenced = nullptr != lhs_ptr;
+ if (!referenced) {
+ lhs = list_indexing.expression1_->Accept(*this);
+ lhs_ptr = &lhs;
+ }
auto index = list_indexing.expression2_->Accept(*this);
- if (!lhs.IsList() && !lhs.IsMap() && !lhs.IsVertex() && !lhs.IsEdge() && !lhs.IsNull())
+ if (!lhs_ptr->IsList() && !lhs_ptr->IsMap() && !lhs_ptr->IsVertex() && !lhs_ptr->IsEdge() && !lhs_ptr->IsNull())
throw QueryRuntimeException(
"Expected a list, a map, a node or an edge to index with '[]', got "
"{}.",
- lhs.type());
- if (lhs.IsNull() || index.IsNull()) return TypedValue(ctx_->memory);
- if (lhs.IsList()) {
+ lhs_ptr->type());
+ if (lhs_ptr->IsNull() || index.IsNull()) return TypedValue(ctx_->memory);
+ if (lhs_ptr->IsList()) {
if (!index.IsInt()) throw QueryRuntimeException("Expected an integer as a list index, got {}.", index.type());
auto index_int = index.ValueInt();
- // NOTE: Take non-const reference to list, so that we can move out the
- // indexed element as the result.
- auto &list = lhs.ValueList();
+ auto &list = lhs_ptr->ValueList();
if (index_int < 0) {
index_int += static_cast<int64_t>(list.size());
}
if (index_int >= static_cast<int64_t>(list.size()) || index_int < 0) return TypedValue(ctx_->memory);
- // NOTE: Explicit move is needed, so that we return the move constructed
- // value and preserve the correct MemoryResource.
- return std::move(list[index_int]);
+ return referenced ? TypedValue(list[index_int], ctx_->memory)
+ : TypedValue(std::move(list[index_int]), ctx_->memory);
}
- if (lhs.IsMap()) {
+ if (lhs_ptr->IsMap()) {
if (!index.IsString()) throw QueryRuntimeException("Expected a string as a map index, got {}.", index.type());
// NOTE: Take non-const reference to map, so that we can move out the
// looked-up element as the result.
- auto &map = lhs.ValueMap();
+ auto &map = lhs_ptr->ValueMap();
auto found = map.find(index.ValueString());
if (found == map.end()) return TypedValue(ctx_->memory);
- // NOTE: Explicit move is needed, so that we return the move constructed
- // value and preserve the correct MemoryResource.
- return std::move(found->second);
+ return referenced ? TypedValue(found->second, ctx_->memory) : TypedValue(std::move(found->second), ctx_->memory);
}
- if (lhs.IsVertex()) {
+ if (lhs_ptr->IsVertex()) {
if (!index.IsString()) throw QueryRuntimeException("Expected a string as a property name, got {}.", index.type());
- return TypedValue(GetProperty(lhs.ValueVertex(), index.ValueString()), ctx_->memory);
+ return {GetProperty(lhs_ptr->ValueVertex(), index.ValueString()), ctx_->memory};
}
- if (lhs.IsEdge()) {
+ if (lhs_ptr->IsEdge()) {
if (!index.IsString()) throw QueryRuntimeException("Expected a string as a property name, got {}.", index.type());
- return TypedValue(GetProperty(lhs.ValueEdge(), index.ValueString()), ctx_->memory);
- }
+ return {GetProperty(lhs_ptr->ValueEdge(), index.ValueString()), ctx_->memory};
+ };
// lhs is Null
return TypedValue(ctx_->memory);
@@ -258,7 +326,15 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
TypedValue Visit(PropertyLookup &property_lookup) override {
- auto expression_result = property_lookup.expression_->Accept(*this);
+ ReferenceExpressionEvaluator referenceExpressionEvaluator(frame_, symbol_table_, ctx_);
+
+ TypedValue *expression_result_ptr = property_lookup.expression_->Accept(referenceExpressionEvaluator);
+ TypedValue expression_result;
+
+ if (nullptr == expression_result_ptr) {
+ expression_result = property_lookup.expression_->Accept(*this);
+ expression_result_ptr = &expression_result;
+ }
auto maybe_date = [this](const auto &date, const auto &prop_name) -> std::optional<TypedValue> {
if (prop_name == "year") {
return TypedValue(date.year, ctx_->memory);
@@ -332,42 +408,38 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
return std::nullopt;
};
- switch (expression_result.type()) {
+ switch (expression_result_ptr->type()) {
case TypedValue::Type::Null:
return TypedValue(ctx_->memory);
case TypedValue::Type::Vertex:
- return TypedValue(GetProperty(expression_result.ValueVertex(), property_lookup.property_), ctx_->memory);
+ return TypedValue(GetProperty(expression_result_ptr->ValueVertex(), property_lookup.property_), ctx_->memory);
case TypedValue::Type::Edge:
- return TypedValue(GetProperty(expression_result.ValueEdge(), property_lookup.property_), ctx_->memory);
+ return TypedValue(GetProperty(expression_result_ptr->ValueEdge(), property_lookup.property_), ctx_->memory);
case TypedValue::Type::Map: {
- // NOTE: Take non-const reference to map, so that we can move out the
- // looked-up element as the result.
- auto &map = expression_result.ValueMap();
+ auto &map = expression_result_ptr->ValueMap();
auto found = map.find(property_lookup.property_.name.c_str());
if (found == map.end()) return TypedValue(ctx_->memory);
- // NOTE: Explicit move is needed, so that we return the move constructed
- // value and preserve the correct MemoryResource.
- return std::move(found->second);
+ return TypedValue(found->second, ctx_->memory);
}
case TypedValue::Type::Duration: {
const auto &prop_name = property_lookup.property_.name;
- const auto &dur = expression_result.ValueDuration();
+ const auto &dur = expression_result_ptr->ValueDuration();
if (auto dur_field = maybe_duration(dur, prop_name); dur_field) {
- return std::move(*dur_field);
+ return TypedValue(*dur_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for Duration", prop_name);
}
case TypedValue::Type::Date: {
const auto &prop_name = property_lookup.property_.name;
- const auto &date = expression_result.ValueDate();
+ const auto &date = expression_result_ptr->ValueDate();
if (auto date_field = maybe_date(date, prop_name); date_field) {
- return std::move(*date_field);
+ return TypedValue(*date_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for Date", prop_name);
}
case TypedValue::Type::LocalTime: {
const auto &prop_name = property_lookup.property_.name;
- const auto < = expression_result.ValueLocalTime();
+ const auto < = expression_result_ptr->ValueLocalTime();
if (auto lt_field = maybe_local_time(lt, prop_name); lt_field) {
return std::move(*lt_field);
}
@@ -375,20 +447,20 @@ class ExpressionEvaluator : public ExpressionVisitor<TypedValue> {
}
case TypedValue::Type::LocalDateTime: {
const auto &prop_name = property_lookup.property_.name;
- const auto &ldt = expression_result.ValueLocalDateTime();
+ const auto &ldt = expression_result_ptr->ValueLocalDateTime();
if (auto date_field = maybe_date(ldt.date, prop_name); date_field) {
return std::move(*date_field);
}
if (auto lt_field = maybe_local_time(ldt.local_time, prop_name); lt_field) {
- return std::move(*lt_field);
+ return TypedValue(*lt_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for LocalDateTime", prop_name);
}
case TypedValue::Type::Graph: {
const auto &prop_name = property_lookup.property_.name;
- const auto &graph = expression_result.ValueGraph();
+ const auto &graph = expression_result_ptr->ValueGraph();
if (auto graph_field = maybe_graph(graph, prop_name); graph_field) {
- return std::move(*graph_field);
+ return TypedValue(*graph_field, ctx_->memory);
}
throw QueryRuntimeException("Invalid property name {} for Graph", prop_name);
}
diff --git a/tests/unit/query_expression_evaluator.cpp b/tests/unit/query_expression_evaluator.cpp
index 6e61e07d6..f1ae0d652 100644
--- a/tests/unit/query_expression_evaluator.cpp
+++ b/tests/unit/query_expression_evaluator.cpp
@@ -1,4 +1,4 @@
-// Copyright 2022 Memgraph Ltd.
+// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -26,6 +26,7 @@
#include "query/interpret/eval.hpp"
#include "query/interpret/frame.hpp"
#include "query/path.hpp"
+#include "query/typed_value.hpp"
#include "storage/v2/storage.hpp"
#include "utils/exceptions.hpp"
#include "utils/string.hpp"
@@ -426,6 +427,47 @@ TEST_F(ExpressionEvaluatorTest, VertexAndEdgeIndexing) {
}
}
+TEST_F(ExpressionEvaluatorTest, TypedValueListIndexing) {
+ auto list_vector = memgraph::utils::pmr::vector<TypedValue>(ctx.memory);
+ list_vector.emplace_back("string1");
+ list_vector.emplace_back(TypedValue("string2"));
+
+ auto *identifier = storage.Create<Identifier>("n");
+ auto node_symbol = symbol_table.CreateSymbol("n", true);
+ identifier->MapTo(node_symbol);
+ frame[node_symbol] = TypedValue(list_vector, ctx.memory);
+
+ {
+ // Legal indexing.
+ auto *op = storage.Create<SubscriptOperator>(identifier, storage.Create<PrimitiveLiteral>(0));
+ auto value = Eval(op);
+ EXPECT_EQ(value.ValueString(), "string1");
+ }
+ {
+ // Out of bounds indexing
+ auto *op = storage.Create<SubscriptOperator>(identifier, storage.Create<PrimitiveLiteral>(3));
+ auto value = Eval(op);
+ EXPECT_TRUE(value.IsNull());
+ }
+ {
+ // Out of bounds indexing with negative bound.
+ auto *op = storage.Create<SubscriptOperator>(identifier, storage.Create<PrimitiveLiteral>(-100));
+ auto value = Eval(op);
+ EXPECT_TRUE(value.IsNull());
+ }
+ {
+ // Legal indexing with negative index.
+ auto *op = storage.Create<SubscriptOperator>(identifier, storage.Create<PrimitiveLiteral>(-2));
+ auto value = Eval(op);
+ EXPECT_EQ(value.ValueString(), "string1");
+ }
+ {
+ // Indexing with incompatible type.
+ auto *op = storage.Create<SubscriptOperator>(identifier, storage.Create<PrimitiveLiteral>("bla"));
+ EXPECT_THROW(Eval(op), QueryRuntimeException);
+ }
+}
+
TEST_F(ExpressionEvaluatorTest, ListSlicingOperator) {
auto *list_literal = storage.Create<ListLiteral>(
std::vector<Expression *>{storage.Create<PrimitiveLiteral>(1), storage.Create<PrimitiveLiteral>(2),
From 5e2ee6c817caee12fe40859d8ab79debb31d77fb Mon Sep 17 00:00:00 2001
From: Ante Javor <javor.ante@gmail.com>
Date: Fri, 17 Feb 2023 17:54:05 +0100
Subject: [PATCH 5/7] Improve mgbench C++ client (#760)
---
tests/mgbench/client.cpp | 145 +++++++++++++++++++++++++++++++++++++--
tests/mgbench/runners.py | 8 ++-
2 files changed, 148 insertions(+), 5 deletions(-)
diff --git a/tests/mgbench/client.cpp b/tests/mgbench/client.cpp
index 87bdda6c9..c7e002df6 100644
--- a/tests/mgbench/client.cpp
+++ b/tests/mgbench/client.cpp
@@ -1,4 +1,4 @@
-// Copyright 2022 Memgraph Ltd.
+// Copyright 2023 Memgraph Ltd.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt; by using this file, you agree to be bound by the terms of the Business Source
@@ -16,17 +16,23 @@
#include <fstream>
#include <limits>
#include <map>
+#include <numeric>
+#include <ostream>
#include <string>
#include <thread>
#include <vector>
#include <gflags/gflags.h>
+#include <math.h>
#include <json/json.hpp>
#include "communication/bolt/client.hpp"
#include "communication/bolt/v1/value.hpp"
#include "communication/init.hpp"
+#include "spdlog/formatter.h"
+#include "spdlog/spdlog.h"
#include "utils/exceptions.hpp"
+#include "utils/logging.hpp"
#include "utils/string.hpp"
#include "utils/timer.hpp"
@@ -48,6 +54,10 @@ DEFINE_bool(queries_json, false,
DEFINE_string(input, "", "Input file. By default stdin is used.");
DEFINE_string(output, "", "Output file. By default stdout is used.");
+DEFINE_bool(validation, false,
+ "Set to true to run client in validation mode."
+ "Validation mode works for singe query and returns results for validation"
+ "with metadata");
std::pair<std::map<std::string, memgraph::communication::bolt::Value>, uint64_t> ExecuteNTimesTillSuccess(
memgraph::communication::bolt::Client *client, const std::string &query,
@@ -55,6 +65,7 @@ std::pair<std::map<std::string, memgraph::communication::bolt::Value>, uint64_t>
for (uint64_t i = 0; i < max_attempts; ++i) {
try {
auto ret = client->Execute(query, params);
+
return {std::move(ret.metadata), i};
} catch (const memgraph::utils::BasicException &e) {
if (i == max_attempts - 1) {
@@ -67,6 +78,28 @@ std::pair<std::map<std::string, memgraph::communication::bolt::Value>, uint64_t>
LOG_FATAL("Could not execute query '{}' {} times!", query, max_attempts);
}
+// Validation returns results and metadata
+std::pair<std::map<std::string, memgraph::communication::bolt::Value>,
+ std::vector<std::vector<memgraph::communication::bolt::Value>>>
+ExecuteValidationNTimesTillSuccess(memgraph::communication::bolt::Client *client, const std::string &query,
+ const std::map<std::string, memgraph::communication::bolt::Value> ¶ms,
+ int max_attempts) {
+ for (uint64_t i = 0; i < max_attempts; ++i) {
+ try {
+ auto ret = client->Execute(query, params);
+
+ return {std::move(ret.metadata), std::move(ret.records)};
+ } catch (const memgraph::utils::BasicException &e) {
+ if (i == max_attempts - 1) {
+ LOG_FATAL("Could not execute query '{}' {} times! Error message: {}", query, max_attempts, e.what());
+ } else {
+ continue;
+ }
+ }
+ }
+ LOG_FATAL("Could not execute query '{}' {} times!", query, max_attempts);
+}
+
memgraph::communication::bolt::Value JsonToBoltValue(const nlohmann::json &data) {
switch (data.type()) {
case nlohmann::json::value_t::null:
@@ -158,6 +191,35 @@ class Metadata final {
std::map<std::string, Record> storage_;
};
+nlohmann::json LatencyStatistics(std::vector<std::vector<double>> &worker_query_latency) {
+ nlohmann::json statistics = nlohmann::json::object();
+ std::vector<double> query_latency;
+ for (int i = 0; i < FLAGS_num_workers; i++) {
+ for (auto &e : worker_query_latency[i]) {
+ query_latency.push_back(e);
+ }
+ }
+ auto iterations = query_latency.size();
+ const int lower_bound = 10;
+ if (iterations > lower_bound) {
+ std::sort(query_latency.begin(), query_latency.end());
+ statistics["iterations"] = iterations;
+ statistics["min"] = query_latency.front();
+ statistics["max"] = query_latency.back();
+ statistics["mean"] = std::accumulate(query_latency.begin(), query_latency.end(), 0.0) / iterations;
+ statistics["p99"] = query_latency[floor(iterations * 0.99)];
+ statistics["p95"] = query_latency[floor(iterations * 0.95)];
+ statistics["p90"] = query_latency[floor(iterations * 0.90)];
+ statistics["p75"] = query_latency[floor(iterations * 0.75)];
+ statistics["p50"] = query_latency[floor(iterations * 0.50)];
+
+ } else {
+ spdlog::info("To few iterations to calculate latency values!");
+ statistics["iterations"] = iterations;
+ }
+ return statistics;
+}
+
void Execute(
const std::vector<std::pair<std::string, std::map<std::string, memgraph::communication::bolt::Value>>> &queries,
std::ostream *stream) {
@@ -167,6 +229,7 @@ void Execute(
std::vector<uint64_t> worker_retries(FLAGS_num_workers, 0);
std::vector<Metadata> worker_metadata(FLAGS_num_workers, Metadata());
std::vector<double> worker_duration(FLAGS_num_workers, 0.0);
+ std::vector<std::vector<double>> worker_query_durations(FLAGS_num_workers);
// Start workers and execute queries.
auto size = queries.size();
@@ -187,16 +250,20 @@ void Execute(
auto &retries = worker_retries[worker];
auto &metadata = worker_metadata[worker];
auto &duration = worker_duration[worker];
- memgraph::utils::Timer timer;
+ auto &query_duration = worker_query_durations[worker];
+
+ memgraph::utils::Timer worker_timer;
while (true) {
auto pos = position.fetch_add(1, std::memory_order_acq_rel);
if (pos >= size) break;
const auto &query = queries[pos];
+ memgraph::utils::Timer query_timer;
auto ret = ExecuteNTimesTillSuccess(&client, query.first, query.second, FLAGS_max_retries);
+ query_duration.push_back(query_timer.Elapsed().count());
retries += ret.second;
metadata.Append(ret.first);
}
- duration = timer.Elapsed().count();
+ duration = worker_timer.Elapsed().count();
client.Close();
}));
}
@@ -218,6 +285,7 @@ void Execute(
final_retries += worker_retries[i];
final_duration += worker_duration[i];
}
+
final_duration /= FLAGS_num_workers;
nlohmann::json summary = nlohmann::json::object();
summary["count"] = queries.size();
@@ -226,12 +294,76 @@ void Execute(
summary["retries"] = final_retries;
summary["metadata"] = final_metadata.Export();
summary["num_workers"] = FLAGS_num_workers;
+ summary["latency_stats"] = LatencyStatistics(worker_query_durations);
+ (*stream) << summary.dump() << std::endl;
+}
+
+nlohmann::json BoltRecordsToJSONStrings(std::vector<std::vector<memgraph::communication::bolt::Value>> &results) {
+ nlohmann::json res = nlohmann::json::object();
+ std::ostringstream oss;
+ for (int i = 0; i < results.size(); i++) {
+ oss << results[i];
+ res[std::to_string(i)] = oss.str();
+ }
+ return res;
+}
+
+/// Validation mode works on single thread with 1 query.
+void ExecuteValidation(
+ const std::vector<std::pair<std::string, std::map<std::string, memgraph::communication::bolt::Value>>> &queries,
+ std::ostream *stream) {
+ spdlog::info("Running validation mode, number of workers forced to 1");
+ FLAGS_num_workers = 1;
+
+ Metadata metadata = Metadata();
+ double duration = 0.0;
+ std::vector<std::vector<memgraph::communication::bolt::Value>> results;
+
+ auto size = queries.size();
+
+ memgraph::io::network::Endpoint endpoint(FLAGS_address, FLAGS_port);
+ memgraph::communication::ClientContext context(FLAGS_use_ssl);
+ memgraph::communication::bolt::Client client(context);
+ client.Connect(endpoint, FLAGS_username, FLAGS_password);
+
+ memgraph::utils::Timer timer;
+ if (size == 1) {
+ const auto &query = queries[0];
+ auto ret = ExecuteValidationNTimesTillSuccess(&client, query.first, query.second, FLAGS_max_retries);
+ metadata.Append(ret.first);
+ results = ret.second;
+ duration = timer.Elapsed().count();
+ client.Close();
+ } else {
+ spdlog::info("Validation works with single query, pass just one query!");
+ }
+
+ nlohmann::json summary = nlohmann::json::object();
+ summary["count"] = 1;
+ summary["duration"] = duration;
+ summary["metadata"] = metadata.Export();
+ summary["results"] = BoltRecordsToJSONStrings(results);
+ summary["num_workers"] = FLAGS_num_workers;
+
(*stream) << summary.dump() << std::endl;
}
int main(int argc, char **argv) {
gflags::ParseCommandLineFlags(&argc, &argv, true);
+ spdlog::info("Running a bolt client with following settings:");
+ spdlog::info("Adress: {} ", FLAGS_address);
+ spdlog::info("Port: {} ", FLAGS_port);
+ spdlog::info("Username: {} ", FLAGS_username);
+ spdlog::info("Password: {} ", FLAGS_password);
+ spdlog::info("Usessl: {} ", FLAGS_use_ssl);
+ spdlog::info("Num of worker: {}", FLAGS_num_workers);
+ spdlog::info("Max retries: {}", FLAGS_max_retries);
+ spdlog::info("Query JSON: {}", FLAGS_queries_json);
+ spdlog::info("Input: {}", FLAGS_input);
+ spdlog::info("Output: {}", FLAGS_output);
+ spdlog::info("Validation: {}", FLAGS_validation);
+
memgraph::communication::SSLInit sslInit;
std::ifstream ifile;
@@ -291,7 +423,12 @@ int main(int argc, char **argv) {
queries.emplace_back(query, std::move(bolt_param.ValueMap()));
}
}
- Execute(queries, ostream);
+
+ if (!FLAGS_validation) {
+ Execute(queries, ostream);
+ } else {
+ ExecuteValidation(queries, ostream);
+ }
return 0;
}
diff --git a/tests/mgbench/runners.py b/tests/mgbench/runners.py
index 5eb2f84b3..3d3aa966e 100644
--- a/tests/mgbench/runners.py
+++ b/tests/mgbench/runners.py
@@ -398,7 +398,13 @@ class Client:
password=self._password,
port=self._bolt_port,
)
+
ret = subprocess.run(args, capture_output=True, check=True)
+ error = ret.stderr.decode("utf-8").strip().split("\n")
+ if error and error[0] != "":
+ print("Reported errros from client")
+ print(error)
+
data = ret.stdout.decode("utf-8").strip().split("\n")
- # data = [x for x in data if not x.startswith("[")]
+ data = [x for x in data if not x.startswith("[")]
return list(map(json.loads, data))
From b2b5a6e2a030da6cfe1eba27b7fe8dad6cb7a00e Mon Sep 17 00:00:00 2001
From: Kruno Golubic <46486712+kgolubic@users.noreply.github.com>
Date: Mon, 20 Feb 2023 17:59:57 +0100
Subject: [PATCH 6/7] Add Fedora badge to README (#795)
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index da30c2b55..ff68a0cc9 100644
--- a/README.md
+++ b/README.md
@@ -95,8 +95,8 @@ your browser.
[](https://memgraph.com/docs/memgraph/install-memgraph-on-linux-docker)
[](https://memgraph.com/docs/memgraph/install-memgraph-on-debian)
[](https://memgraph.com/docs/memgraph/install-memgraph-on-ubuntu)
-[](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
+[](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
+[](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
You can find the binaries and Docker images on the [Download
Hub](https://memgraph.com/download) and the installation instructions in the
From 2df357b0126d76e650c219510e7e558232b7d511 Mon Sep 17 00:00:00 2001
From: Kruno Golubic <46486712+kgolubic@users.noreply.github.com>
Date: Mon, 20 Feb 2023 18:18:29 +0100
Subject: [PATCH 7/7] Add RedHat badge to README (#796)
---
README.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.md b/README.md
index ff68a0cc9..012f9c62d 100644
--- a/README.md
+++ b/README.md
@@ -97,6 +97,7 @@ your browser.
[](https://memgraph.com/docs/memgraph/install-memgraph-on-ubuntu)
[](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
[](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
+[](https://memgraph.com/docs/memgraph/install-memgraph-from-rpm)
You can find the binaries and Docker images on the [Download
Hub](https://memgraph.com/download) and the installation instructions in the