1e0ac8ab8f
Summary: My dear fellow Memgraphians. It's friday afternoon, and I am as ready to pop as WAL is to get reviewed... What's done: - Vertices and Edges have global IDs, stored in `VersionList`. Main storage is now a concurrent map ID->vlist_ptr. - WriteAheadLog class added. It's based around buffering WAL::Op objects (elementraly DB changes) and periodically serializing and flusing them to disk. - Snapshot recovery refactored, WAL recovery added. Snapshot format changed again to include necessary info. - Durability testing completely reworked. What's not done (and should be when we decide how): - Old WAL file purging. - Config refactor (naming and organization). Will do when we discuss what we want. - Changelog and new feature documentation (both depending on the point above). - Better error handling and recovery feedback. Currently it's all returning bools, which is not fine-grained enough (neither for errors nor partial successes, also EOF is reported as a failure at the moment). - Moving the implementation of WAL stuff to .cpp where possible. - Not sure if there are transactions being created outside of `GraphDbAccessor` and it's `BuildIndex`. Need to look into. - True write-ahead logic (flag controlled): not committing a DB transaction if the WAL has not flushed it's data. We can discuss the gain/effort ratio for this feature. Reviewers: buda, mislav.bradac, teon.banek, dgleich Reviewed By: dgleich Subscribers: mtomic, pullbot Differential Revision: https://phabricator.memgraph.io/D958
76 lines
2.1 KiB
C++
76 lines
2.1 KiB
C++
#pragma once
|
|
|
|
#include <fstream>
|
|
|
|
#include "hasher.hpp"
|
|
#include "utils/bswap.hpp"
|
|
|
|
/**
|
|
* Buffer that writes data to file and calculates hash of written data.
|
|
* Implements template param Buffer interface from BaseEncoder class.
|
|
*
|
|
* All of the methods on a HashedFileWriter can throw an exception.
|
|
*/
|
|
class HashedFileWriter {
|
|
public:
|
|
/** Constructor, initialize ofstream to throw exception on fail. */
|
|
HashedFileWriter() {
|
|
output_stream_.exceptions(std::ifstream::failbit | std::ifstream::badbit);
|
|
}
|
|
|
|
/** Constructor which also takes a file path and opens it immediately. */
|
|
explicit HashedFileWriter(const std::string &path) : HashedFileWriter() {
|
|
output_stream_.open(path, std::ios::out | std::ios::binary);
|
|
}
|
|
|
|
/** Opens the writer */
|
|
void Open(const std::string &path) {
|
|
output_stream_.open(path, std::ios::out | std::ios::binary);
|
|
hasher_ = Hasher();
|
|
}
|
|
|
|
/** Closes the writer. */
|
|
void Close() { output_stream_.close(); }
|
|
|
|
/**
|
|
* Writes data to stream.
|
|
*
|
|
* @param data - Pointer to data to write.
|
|
* @param n - Data length.
|
|
* @param hash - If writing should update the hash.
|
|
* @return - True if succesful.
|
|
*/
|
|
void Write(const uint8_t *data, size_t n, bool hash = true) {
|
|
output_stream_.write(reinterpret_cast<const char *>(data), n);
|
|
if (hash) hasher_.Update(data, n);
|
|
}
|
|
|
|
/**
|
|
* Writes a TValue to the stream.
|
|
*
|
|
* @param val - The value to write.
|
|
* @param hash - If writing should update the hash.
|
|
* @return - True if succesful.
|
|
*/
|
|
template <typename TValue>
|
|
void WriteValue(const TValue &val, bool hash = true) {
|
|
TValue val_bswapped = bswap(val);
|
|
Write(reinterpret_cast<const uint8_t *>(&val_bswapped), sizeof(TValue),
|
|
hash);
|
|
}
|
|
|
|
// TODO try to remove before diff
|
|
/** Does nothing. Just for API compatibility with the bolt buffer. */
|
|
void Chunk() {}
|
|
|
|
/** Flushes data to stream. */
|
|
void Flush() { output_stream_.flush(); }
|
|
|
|
/** Returns the hash of the data written so far to the stream. */
|
|
uint64_t hash() const { return hasher_.hash(); }
|
|
|
|
private:
|
|
std::ofstream output_stream_;
|
|
Hasher hasher_;
|
|
};
|