memgraph/docs/dev/distributed/updates.md

# Distributed updates

Operations that modify the graph state are somewhat more complex in the
distributed system, as opposed to a single-node Memgraph deployment. The
complexity arises from two factors.

First, the data being modified is not necessarily owned by the worker
performing the modification. This situation is completely valid workers
execute parts of the query plan and parts must be executed by the
master.

Second, there are less guarantees regarding multi-threaded access. In
single-node Memgraph it was guaranteed that only one transaction will be
performing database work in a single transaction. This implied that
per-version storage could be thread-unsafe. In distributed Memgraph it
is possible that multiple threads could be performing work in the same
transaction as a consequence of the query being executed at the same
time on multiple workers and those executions interacting with the
globally partitioned database state.

## Deferred state modification

Making the per-version data storage thread-safe would most likely have a
performance impact very undesirable in a transactional database intended
for high throughput.

An alternative is that state modification over unsafe structures is not
performed immediately when requested, but postponed until it is safe to
do (there is is a guarantee of no concurrent access).

Since local query plan execution is done the same way on local data as
it is in single-node Memgraph, it is not possible to deffer that part of
the modification story. What can be deferred are modifications requested
by other workers. Since local query plan execution still is
single-threaded, this approach is safe.

At the same time those workers requesting the remote update can update
local copies (caches) of the not-owned data since that cache is only
being used by the single, local-execution thread.

### Visibility

Since updates are deferred the question arises: when do the updates
become visible? The above described process offers the following
visibility guarantees:
- updates done on the local state are visible to the owner
- updates done on the local state are NOT visible to anyone else during
  the same (transaction + command)
- updates done on remote state are deferred on the owner and not
  visible to the owner until applied
- updates done on the remote state are applied immediately to the local
  caches and thus visible locally

This implies an inconsistent view of the database state. In a concurrent
execution of a single query this can hardly be avoided and is accepted
as such. It does not change the Cypher query execution semantic in any
of the well-defined scenarios. It possibly changes some of the behaviors
in which the semantic is not well defined even in single-node execution.

### Synchronization, update application

In many queries it is mandatory to observe the latest global graph state
(typically when returning it to the client). That means that before that
happens all the deferred updates need to be applied, and all the caches
to remote data invalidated. Exactly this happens when executing queries
that modify the graph state. At some point a global synchronization
point is reached. First it is waited that all workers finish the
execution of query plan parts performing state modifications. After that
all the workers are told to apply the deferred updates they received to
their graph state. Since there is no concurrent query plan execution,
this is safe. Once that is done all the local caches are cleared and the
requested data can be returned to the client.

### Command advancement

In complex queries where a read part follows a state modification part
the synchronization process after the state modification part is
followed by command advancement, like in single-node execution.

## Creation

Graph element creation is not deferred. This is practical because the
response to a creation is the global ID of the newly created element. At
the same time it is safe because no other worker (including the owner)
will be using the newly added graph element.

## Updating

Updating is deferred, as described. Note that this also means that
record locking conflicts are deferred and serialization errors
(including lock timeouts) are postponed until the deferred update
application phase. In certain scenarios it might be beneficial to force
these errors to happen earlier, when the deferred update request is
processed.

## Deletion

Deletion is also deferred. Deleting an edge implies a modification of
it's endpoint vertices, which must be deferred as those data structures
are not thread-safe. Deleting a vertex is either with detaching, in
which case an arbitrary number of updates are implied in the vertex's
neighborhood, or without detaching which relies on checking the current
state of the graph which is generally impossible in distributed.