Query allocator and evaluation allocator were different.
After analysis, was determined they should be the same, this will help
future development reduce TypeValue copies during queries.
Changes:
- Common allocator, PoolResource backed by MonotonicResource
- Optimized Pool, now O(1) alloc/dealloc as all chunks in Pool form a single
free list
- 2nd PoolResource, using bin sizing, not as perfect for memory usage but
O(1) bin selection
- Now have jemalloc's background thread to make sure decay and return
to OS happens
- Optimized ProperyValue to be faster at destruction/copy/move
- Less temporary memory allocations
- CSV reader now maintains a common line buffer it reuses on line reads
- Writing out bolt values, now reuses a values buffer
- Evaluating an int no longer makes temporary strings for errors it most
likely never throws
- ExpandVariable will reuse existing edge list in frame it one existed
* Update deployment files for mgbuilders because of toolchain upgrade
* Fix args parameter in builder yaml files
* Add fedora 38, 39 and rockylinux 9.3 mgbuilder Dockerfiles
* Change format of ARG TOOLCHAIN_VERSION from toolchain-vX to vX
* Add function to check supported arch, build type, os and toolchain
* Add options to init subcommand
* Add image names to mgbuilders
* Add v2 of the run.sh script
* Add testing to run2.sh
* Add option for threads --thread
* Add options for enterprise license and organization name
* Make stop mgbuild container step run always
* Add --ci flag to init script
* Move init conditionals under build-memgraph flags
* Add --community flag to build-memgraph
* Change target dir inside mgbuild container
* Add node fix to debian 11, ubuntu 20.04 and ubuntu 22.04
* rm memgraph repo after installing deps
* Add mg user in Dockerfile
* Add step to install rust on all OSs
* Chown files copied into mgbuild container
* Add e2e tests
* Add jepsen test
* Bugfix: Using reference in a callback
* Bugfix: Broad target for e2e tests
* Up db info test limit
* Disable e2e streams tests
* Fix default THREADS
* Prioretize docker compose over docker-compose
* Improve selection between docker compose and docker-compose
* Install PyYAML as mg user
* Fix doxygen install for rocky linux 9.3
* Fix rocky-9.3 environment script to properly install sbcl
* Rename all rocky-9 mentions to rocky-9.3
* Add mgdeps-cache and benchgraph-api hostnames to mgbuild images
* Add logic to pull mgbuild image if missing
* Fix build errors on toolchain-v5 (#1806)
* Rename run2 script, remove run script, add small features to mgbuild.sh
* Add --no-copy flag to build-memgraph to resolve TODO
* Add timeouts to diff jobs
* Fix asio flaky clone, try mgdeps-cache first
---------
Co-authored-by: Andreja Tonev <andreja.tonev@memgraph.io>
Co-authored-by: Ante Pušić <ante.f.pusic@gmail.com>
Co-authored-by: antoniofilipovic <filipovicantonio1998@gmail.com>
* Optimise frame change
* Optimise distinct + orderby memory usage
- dispose collections as earlier as possible
- move values rather than copy
* Better perf, ORDER BY
* Optimise RANGE and UNWIND
* ConstraintVerificationInfo only if at least one constraint
* Optimise TypeValue
* Clang-tidy fix
Make special structure for old_disk_key. std::optional<std::string> was
40B, which is the largest member of out action union. Replaced with 8B,
structure.
This makes largest member now vertex_edge at 24B, this means Delta is
now only 56B.
🥳🎉 Now less than a cacheline 🎊✨
- Only IN_MEMORY_ANALYTICAL requires unique lock during snapshot
- GC in some cases will be provide with unique lock
- This fact can be used for optimisations
- In all other cases, optimisations should be done with alternative
check. Not via getting a unique lock
Also:
- Faster property lookup
- Faster index iteration (better conditional branching)
* Add AUTH system tx deltas
* Add auth data RPC and handlers
* Support multiple system deltas in a single transaction
* Added e2e test
* Bugfix: KVStore segfault after move
---------
Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
* Split queries into system and data queries
* System queries are sequentially executed and generate separate transaction deltas
* System transaction try locks for 100ms
* last_commited_system_ts saved to DBMS durability
* Replicating CREATE/DROP DATABASE
* Sending a system snapshot if REPLICA behind
* Passing a copy of the gatekeeper::access as std::any to all functions that could call an async execution
* Removed delete_on_drop flag (we now always delete on drop)
* Using UUID as the directory name for databases
* DBMS durability update (added versioning and salient information)
* Automatic migration from previous version
* Interpreter can run some queries without a target database
* SHOW REPLICA returns the status of the currently active DB
* Returning UUID instead of db name in the RPC responses
* Using UUIDs for database specification in RPC (not name)
* FrequentCheck forces update on reconnect
* TimestampRpc will detect if a replica is behind, and will update client's state
* Safer SLK reads
* Split SHOW DATABASES in two SHOW DATABASES (list of current databases) and SHOW DATABASE a single string naming the current database
---------
Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
* Fix up REPLICA GetInfo and CreateSnapshot
Subtle bug where these actions were using the incorrect transactional
access while in REPLICA role. This casued timestamp to be incorrectly
bumped, breaking REPLICA from doing replication.
* Delay DNS resolution
Rather than resolve at endpoint creation, we will instread resolve only
on Socket connect. This allows k8s deployments to change their IP during
pod restarts.
* Minor sonarsource fixes
---------
Co-authored-by: Andreja <andreja.tonev@memgraph.io>
Co-authored-by: DavIvek <david.ivekovic@memgraph.io>
Fixes root cause of a cascade of failures in replication code:
- Replica handling of deleting an edge is now corrected. Now tolerant of multiple edges of the same relationship type.
- Improved robustness: correct exception handling around failed stream of current WAL file. This now means a REPLICA failure will no longer prevent transactions on MAIN from performing WAL writes.
- Slightly better diagnostic messages, not user friendly but helps get developer to correct root cause quicker.
- Proactively remove vertex+edges during Abort rather than defer to GC to do that work, this included fixing constraints and indexes to be safe.
Co-authored-by: Andreja Tonev <andreja.tonev@memgraph.io>
Single (instance level) connection to a replica (messages from all databases get multiplexed through it)
ReplicationClient split in two: ReplicationClient and ReplicationStorageClient
New ReplicationClient, moved under replication, handles the raw connection, owned by MainRoleData
ReplicationStorageClient handles the storage <-> replica state machine and holds to a stream
Removed epoch and storage from *Clients
rpc::Stream proactively aborts on error and sets itself to a defunct state
Removed HandleRpcFailure, instead we simply log the error and let the FrequentCheck handle re-connection
replica_state is now a synced variable
ReplicaStorageClient state machine bugfixes
Single FrequentCheck that goes through DBMS
Moved ReplicationState under DbmsHandler
Moved some replication startup logic under the DbmsHandler's constructor
Removed InMemoryReplicationClient
CreateReplicationClient has been removed from Storage
Simplified GetRecoverySteps and made safer
---------
Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
There might be a performance impect of updating the metadata store on
bulk operations. Hence this flag which is disabling the collection by
default. If the queries to obtain the information are called with this
flag disabled, the database will throw an exception.
The objects stored_node_labels_ and stored_edge_types_ can be accesses
through separate threads but it was not safe to do so. This commit
replaces the standard containers with threadsafe ones.
Before the functions that are retrieving the data from the metadata
holding datastructures were returning a std::string, and that was
propagated outward all the way through. To keep this functions
consistent with the rest of the storage/dbaccessor functions the LabelId
and EdgeTypeId will be propagated respectively and the conversion into
string will only happen at the interpreter level.
In order to get the required metadata in constant time we need to keep
track of the node labels and edge types that were ever present in the
database. This is done by the two axuiliary datastructures that are
present in the storage instances. The ability to get this metadata is
propagated to the DBAccessor class, which the query modules can interact
with.
