Commit Graph

1296 Commits

Author SHA1 Message Date
Andi
75aad72984
Improve in-memory RAFT state (#1782) 2024-03-06 09:16:46 +01:00
Antonio Filipovic
d4d4660af0
Add force sync REPLICA with MAIN (#1777) 2024-03-05 16:51:14 +00:00
Andi
1802dc93d1
Improve Raft log serialization (#1778) 2024-03-05 07:33:13 +00:00
Antonio Filipovic
33caa27161
Ensure replication works on HA cluster in different scenarios (#1743) 2024-03-01 12:32:56 +01:00
Antonio Filipovic
b561c61b64
HA: Add initial logic for choosing new replica (#1729) 2024-02-28 09:57:00 +00:00
andrejtonev
82c47ee80d
GetInfo simplification (#1621)
* Removed force dir in the GetInfo functions
2024-02-26 14:55:45 +00:00
andrejtonev
6a4ef55e90
Better auth user/role handling (#1699)
* Stop auth module from creating users
* Explicit about auth policy (check if no users defined OR auth module used)
* Role supports database access definition
* Authenticate() returns user or role
* AuthChecker generates QueryUserOrRole (can be empty)
* QueryUserOrRole actually authorizes
* Add auth cache invalidation
* Better database access queries (GRANT, DENY, REVOKE DATABASE)
2024-02-22 14:00:39 +00:00
Marko Budiselić
61b9bb0f59
Add toolchain-v5 compatibility Revert to C++20 (#587)
* Upgrade cppitertools, spdlog, fmt, rapidcheck
* Make compilation work on both v4 and v5 toolchains
2024-02-19 21:09:54 +01:00
Andi
7ec648b4ce
Add --experimental-enabled=high-availability (#1720) 2024-02-19 16:28:15 +00:00
Andi
f3574012c5
Add cpp23 support (#1726) 2024-02-19 10:36:51 +00:00
Gareth Andrew Lloyd
33c400fcc1
Fixup memory e2e tests (#1715)
- Remove the e2e that did concurrent mgp_* calls on the same transaction
  (ATM this is unsupported)
- Fix up the concurrent mgp_global_alloc test to be testing it more precisely
- Reduce the memory limit on detach delete test due to recent memory
  optimizations around deltas.
- No longer throw from hook, through jemalloc C, to our C++ on other
  side. This cause mutex unlocks to not happen.
- No longer allocate error messages while inside the hook. This caused
  recursive entry back inside jamalloc which would try to relock a
  non-recursive mutex.
2024-02-16 15:35:08 +00:00
Antonio Filipovic
bfc756c092
HA: Polish flow for replicas from coordinator (#1711) 2024-02-16 10:58:01 +01:00
Gareth Andrew Lloyd
f48151576b
System replication experimental flag (#1702)
- Remove the compile time control
- Introduce the runtime control flag

New flag `--experimental-enabled=system-replication`
2024-02-13 12:57:18 +00:00
Andi
cf80687d1d
HA: Organize Raft coordinator group (#1687) 2024-02-08 09:11:33 +00:00
Antonio Filipovic
c15b62a88d
HA: Disable replication from old main (#1674) 2024-02-07 11:20:47 +01:00
Gareth Andrew Lloyd
4ef6a1f9c3
Improve memory handling of Deltas (#1688)
- Reduce delta from 104B to 80B
- Hold and pass them around as in a deque
- Detect and deleted deltas within commit if safe to do so
2024-02-06 18:07:38 +01:00
andrejtonev
7ead00f23e
Adding authentication data replication (#1666)
* Add AUTH system tx deltas
* Add auth data RPC and handlers
* Support multiple system deltas in a single transaction
* Added e2e test
* Bugfix: KVStore segfault after move

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2024-02-05 10:37:00 +00:00
Gareth Andrew Lloyd
97b1e67d80
Fix auth durability (#1644)
* Change auth durability to store hash algorithm
* Add Salt to SHA256
2024-01-30 18:17:05 +00:00
Andi
78a88737f8
HA: Add automatic failover (#1646)
Co-authored-by: antoniofilipovic <filipovicantonio1998@gmail.com>
2024-01-29 15:34:00 +01:00
andrejtonev
ff44d68843
Simplify auth::Auth (#1663)
Moved various auth flags under a single config
Moved all regex logic under auth::Auth
2024-01-29 12:52:32 +00:00
Andi
38ade99652
HA: Add coordinator to replication cluster (#1608) 2024-01-24 13:07:51 +01:00
Gareth Andrew Lloyd
e7f6a5f4f4
Fix SkipList iterators (#1635)
Fix SkipList iterators and find methods to be as expected by normal C++ iterator usage
2024-01-23 15:31:28 +00:00
andrejtonev
071df2f439
Replication refactor part 7 (#1550)
* Split queries into system and data queries
* System queries are sequentially executed and generate separate transaction deltas
* System transaction try locks for 100ms
* last_commited_system_ts saved to DBMS durability
* Replicating CREATE/DROP DATABASE
* Sending a system snapshot if REPLICA behind
* Passing a copy of the gatekeeper::access as std::any to all functions that could call an async execution
* Removed delete_on_drop flag (we now always delete on drop)
* Using UUID as the directory name for databases
* DBMS durability update (added versioning and salient information)
* Automatic migration from previous version
* Interpreter can run some queries without a target database
* SHOW REPLICA returns the status of the currently active DB
* Returning UUID instead of db name in the RPC responses
* Using UUIDs for database specification in RPC (not name)
* FrequentCheck forces update on reconnect
* TimestampRpc will detect if a replica is behind, and will update client's state
* Safer SLK reads
* Split SHOW DATABASES in two SHOW DATABASES (list of current databases) and SHOW DATABASE a single string naming the current database

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2024-01-23 12:06:10 +01:00
Aidar Samerkhanov
c772cab766
ToString function now returns double values with precision 15 (#1576)
The DoubleToString function has been updated to handle higher precision doubles correctly. The unnecessary string length restriction has been removed, allowing the function to convert the full double value without prematurely truncating it. This change ensures that the string representation of doubles is more accurate, especially for very large or very small numbers. Unit tests have been added to verify the correct behavior for a range of double values.
2024-01-12 12:32:34 +04:00
DavIvek
31f15b3651
Fix index hints (#1606) 2024-01-11 10:10:06 +01:00
DavIvek
b3d0c2ccc2
Add query parameters support for labels (#1602) 2024-01-10 15:08:21 +01:00
DavIvek
4bb9238679
Add support for triggers in database dump (#1610) 2024-01-09 13:05:54 +01:00
Gareth Andrew Lloyd
0fb8e4116f
Fix REPLICA timestamps (#1615)
* Fix up REPLICA GetInfo and CreateSnapshot

Subtle bug where these actions were using the incorrect transactional
access while in REPLICA role. This casued timestamp to be incorrectly
bumped, breaking REPLICA from doing replication.

* Delay DNS resolution

Rather than resolve at endpoint creation, we will instread resolve only
on Socket connect. This allows k8s deployments to change their IP during
pod restarts.

* Minor sonarsource fixes

---------
Co-authored-by: Andreja <andreja.tonev@memgraph.io>
Co-authored-by: DavIvek <david.ivekovic@memgraph.io>
2024-01-05 16:42:54 +00:00
Gareth Andrew Lloyd
21bbc196ae
Cleanup filesystem after unittest (#1581) 2023-12-13 13:19:01 +00:00
Antonio Filipovic
74fa6d21f6
Implement parallel constraints recovery (#1545) 2023-12-04 21:56:05 +01:00
Aidar Samerkhanov
953a8f5340
Add handling of deleted return values for query procedures and functions ran in analytical mode (#1395)
Co-authored-by: Ante Pušić <ante.pusic@memgraph.io>
2023-12-04 08:32:59 +01:00
Ante Pušić
3ccd78ac71
Add path and weight to variable expand filter (#1434)
Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io>
2023-12-02 20:03:40 +01:00
Antonio Filipovic
72d47fc3bf
Implement short circuiting of exists evaluation (#1539) 2023-11-27 16:44:12 +01:00
andrejtonev
8b9e1fa08b
Replication refactor part 6 (#1484)
Single (instance level) connection to a replica (messages from all databases get multiplexed through it)
ReplicationClient split in two: ReplicationClient and ReplicationStorageClient
New ReplicationClient, moved under replication, handles the raw connection, owned by MainRoleData
ReplicationStorageClient handles the storage <-> replica state machine and holds to a stream
Removed epoch and storage from *Clients
rpc::Stream proactively aborts on error and sets itself to a defunct state
Removed HandleRpcFailure, instead we simply log the error and let the FrequentCheck handle re-connection
replica_state is now a synced variable
ReplicaStorageClient state machine bugfixes
Single FrequentCheck that goes through DBMS
Moved ReplicationState under DbmsHandler
Moved some replication startup logic under the DbmsHandler's constructor
Removed InMemoryReplicationClient
CreateReplicationClient has been removed from Storage
Simplified GetRecoverySteps and made safer

---------

Co-authored-by: Gareth Lloyd <gareth.lloyd@memgraph.io>
2023-11-23 11:02:35 +01:00
Gareth Andrew Lloyd
e4f94c15c6
Fixes for clang-tidy / sonar issues (#1536) 2023-11-22 13:05:02 +00:00
Andi
d03fafcef6
Aggregations return empty result when used with group by (#1531) 2023-11-20 11:52:17 +01:00
Antonio Filipovic
645568a75b
Remove default memory limit on procedures (#1506)
* remove default limit on procedures
* fix bug on GraphQL also
2023-11-16 15:01:44 +01:00
Andi
e5b2c19ea2
Empty Collect() returns nothing (#1482) 2023-11-13 11:45:09 +01:00
Antonio Filipovic
4d5ea03dfa
Use extent hooks for memory procedure limit (#1443) 2023-11-07 16:04:29 +01:00
DavIvek
ece4b0dba8
Fix cached plan not getting invalidated (#1348) 2023-11-07 13:34:03 +01:00
Andi
66487a6dce
Durability improvements (#1385) 2023-11-07 11:37:54 +01:00
Andi
f4b97fc03d
Fix missing statistics for SetPropertiesCursor (#1460) 2023-11-07 09:11:20 +01:00
andrejtonev
dbc6054689
Replication refactor (part 5) (#1378) 2023-11-06 11:50:49 +00:00
Aidar Samerkhanov
16b8c7b27c
Fix Kafka flaky unit test (#1409) 2023-11-05 20:51:56 +01:00
Antonio Filipovic
48631d1e37
Rename memory usage and memory allocated (#1426) 2023-11-03 14:40:45 +01:00
Andi
3e9f25b8e4
Support creating date and localtime from localdatetime (#1381) 2023-11-03 10:54:01 +01:00
Gareth Andrew Lloyd
157b36162b
Speedup socket unit test (#1444)
Was testing a setup that wasn't used in production, it would
unnecessarily thrash small buffers.
2023-11-01 17:24:24 +00:00
Josipmrden
5b9802bd7b
Extend property cache to the expression evaluator (#1432)
* Add support for property cache in the produce
* Fix the previous implementation in the map literal
2023-10-28 20:32:58 -07:00
Ante Pušić
989bb97514
Extend Cypher queries with the index hinting feature (#1345) 2023-10-27 14:26:19 +02:00
Ante Pušić
3158a16ffd
Add filtering details to EXPLAIN and PROFILE (#1265) 2023-10-25 21:36:20 +02:00