diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ab4fa685..915ed02d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ * Dynamic graph partitioner added. * Static vertices/edges id generators exposed through the Id Cypher function. +* Properties on disk added. ### Bug Fixes and Other Changes diff --git a/docs/dev/memgraph/storage/property-storage.md b/docs/dev/memgraph/storage/property-storage.md new file mode 100644 index 000000000..897c54ce7 --- /dev/null +++ b/docs/dev/memgraph/storage/property-storage.md @@ -0,0 +1,131 @@ +# Property storage + +Although the reader is probably familiar with properties in *Memgraph*, let's +briefly recap. + +Both vertices and edges can store an arbitrary number of properties. Properties +are, in essence, ordered pairs of property names and property values. Each +property name within a single graph element (edge/node) can store a single +property value. Property names are represented as strings, while property values +must be one of the following types: + + Type | Description +-----------|------------ + `Null` | Denotes that the property has no value. This is the same as if the property does not exist. + `String` | A character string, i.e. text. + `Boolean` | A boolean value, either `true` or `false`. + `Integer` | An integer number. + `Float` | A floating-point number, i.e. a real number. + `List` | A list containing any number of property values of any supported type. It can be used to store multiple values under a single property name. + `Map` | A mapping of string keys to values of any supported type. + +Property values are modeled in a class conveniently called `PropertyValue`. + +## Mapping between property names and property keys. + +Although users think of property names in terms of descriptive strings +(e.g. "location" or "department"), *Memgraph* internally converts those names +into property keys which are, essentially, unsigned 16-bit integers. + +Property keys are modelled by a not-so-conveniently named class called +`Property` which can be found in `storage/types.hpp`. The actual conversion +between property names and property keys is done within the `ConcurrentIdMapper` +but the internals of that implementation are out of scope for understanding +property storage. + +## PropertyValueStore + +Both `Edge` and `Vertex` objects contain an instance of `PropertyValueStore` +object which is responsible for storing properties of a corresponding graph +element. + +An interface of `PropertyValueStore` is as follows: + + Method | Description +-----------|------------ + `at` | Returns the `PropertyValue` for a given `Property` (key). + `set` | Stores a given `PropertyValue` under a given `Property` (key). + `erase` | Deletes a given `Property` (key) alongside its corresponding `PropertyValue`. + `clear` | Clears the storage. + `iterator`| Provides an extension of `std::input_iterator` that iterates over storage. + +## Storage location + +By default, *Memgraph* is an in-memory database and all properties are therefore +stored in working memory unless specified otherwise by the user. User has an +option to specify via the command line which properties they wish to be stored +on disk. + +Storage location of each property is encapsulated within a `Property` object +which is ensured by the `ConcurrentIdMapper`. More precisely, the unsigned 16-bit +property key has the following format: + +``` +|---location--|------id------| +|-Memory|Disk-|-----2^15-----| +``` + +In other words, the most significant bit determines the location where the +property will be stored. + +### In-memory storage + +The underlying implementation of in-memory storage for the time being is +`std::vector<std::pair<Property, PropertyValue>>`. Implementations of`at`, `set` +and `erase` are linear in time. This implementation is arguably more efficient +than `std::map` or `std::unordered_map` when the average number of properties of +a record is relatively small (up to 10) which seems to be the case. + +### On-disk storage + +#### KVStore + +Disk storage is modeled by an abstraction of key-value storage as implemented in +`storage/kvstore.hpp'. An interface of this abstraction is as follows: + + Method | Description +----------------|------------ + `Put` | Stores the given value under the given key. + `Get` | Obtains the given value stored under the given key. + `Delete` | Deletes a given (key, value) pair from storage.. + `DeletePrefix` | Deletes all (key, value) pairs where key begins with a given prefix. + `Size` | Returns the size of the storage or, optionally, the number of stored pairs that begin with a given prefix. + `iterator` | Provides an extension of `std::input_iterator` that iterates over storage. + +Keys and values in this context are of type `std::string`. + +The actual underlying implementation of this abstraction uses +[RocksDB]{https://rocksdb.org} — a persistent key-value store for fast +storage. + +It is worthy to note that the custom iterator implementation allows the user +to iterate over a given prefix. Otherwise, the implementation follows familiar +c++ constructs and can be used as follows: + +``` +KVStore storage = ...; +for (auto it = storage.begin(); it != storage.end(); ++it) {} +for (auto kv : storage) {} +for (auto it = storage.begin("prefix"); it != storage.end("prefix"); ++it) {} +``` + +Note that it is not possible to scan over multiple prefixes. For instance, one +might assume that you can scan over all keys that fall in a certain +lexicographical range. Unfortunately, that is not the case and running the +following code will result in an infinite loop with a touch of undefined +behavior. + +``` +KVStore storage = ...; +for (auto it = storage.begin("alpha"); it != storage.end("omega"); ++it) {} +``` + +#### Data organization on disk + +Each `PropertyValueStore` instance can access a static `KVStore` object that can +store `(key, value)` pairs on disk. The key of each property on disk consists of +two parts — a unique identifier (unsigned 64-bit integer) of the current +record version (see mvcc docummentation for further clarification) and a +property key as described above. The actual value of the property is serialized +into a bytestring using bolt `BaseEncoder`. Similarly, deserialization is +performed by bolt `Decoder`. diff --git a/docs/user_technical/storage.md b/docs/user_technical/storage.md index 690390085..4bf7d12e3 100644 --- a/docs/user_technical/storage.md +++ b/docs/user_technical/storage.md @@ -75,3 +75,27 @@ However, these queries are not: MATCH (n:Node) SET n.property[0] = 0 MATCH (n:Node) SET n.property.key = "other value" + +### Cold data on disk + +Although *Memgraph* is an in-memory database by default, it offers an option +to store a certain amount of data on disk. More precisely, the user can pass +a list of properties they wish to keep stored on disk via the command line. +In certain cases, this might result in a significant performance boost due to +reduced memory usage. It is recommended to use this feature on large, +cold properties, i.e. properties that are rarely accessed. + +For example, a user of a library database might identify author biographies +and book summaries as cold properties. In that case, the user should run +*Memgraph* as follows: + +``` +/usr/lib/memgraph/memgraph --properties-on-disk biography,summary +``` + +Note that the usage of *Memgraph* has not changed, i.e. durability and +data recovery mechanisms are still in place and the query language remains +the same. It is also important to note that the user cannot change the storage +location of a property while *Memgraph* is running. Naturally, the user can +reload their database from snapshot, provide a different list of properties on +disk and rest assured that only those properties will be stored on disk. diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index c7ed82cc9..b838f8a9c 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -213,6 +213,10 @@ add_dependencies(memgraph_lib generate_capnp) add_library(kvstore_lib STATIC storage/kvstore.cpp) target_link_libraries(kvstore_lib stdc++fs mg-utils rocksdb bzip2 zlib) +# STATIC library for dummy key-value storage +add_library(kvstore_dummy_lib STATIC storage/kvstore_dummy.cpp) +target_link_libraries(kvstore_dummy_lib mg-utils) + # Generate a version.hpp file set(VERSION_STRING ${memgraph_VERSION}) configure_file(version.hpp.in version.hpp @ONLY) @@ -220,7 +224,7 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR}) # memgraph main executable add_executable(memgraph memgraph_bolt.cpp) -target_link_libraries(memgraph memgraph_lib) +target_link_libraries(memgraph memgraph_lib kvstore_lib) set_target_properties(memgraph PROPERTIES # Set the executable output name to include version information. OUTPUT_NAME "memgraph-${memgraph_VERSION}-${COMMIT_HASH}_${CMAKE_BUILD_TYPE}" diff --git a/src/database/storage.hpp b/src/database/storage.hpp index fff88ed6c..1d5d50e96 100644 --- a/src/database/storage.hpp +++ b/src/database/storage.hpp @@ -10,7 +10,7 @@ #include "mvcc/version_list.hpp" #include "storage/address.hpp" #include "storage/edge.hpp" -#include "storage/kvstore_mock.hpp" +#include "storage/kvstore.hpp" #include "storage/types.hpp" #include "storage/vertex.hpp" #include "transactions/type.hpp" diff --git a/src/storage/kvstore.cpp b/src/storage/kvstore.cpp index fd0f53e3f..1260c08fc 100644 --- a/src/storage/kvstore.cpp +++ b/src/storage/kvstore.cpp @@ -1,40 +1,137 @@ -#include "storage/kvstore.hpp" +#include <rocksdb/db.h> +#include <rocksdb/options.h> +#include "storage/kvstore.hpp" #include "utils/file.hpp" namespace storage { -KVStore::KVStore(fs::path storage) : storage_(storage) { - if (!utils::EnsureDir(storage)) - throw KVStoreError("Folder for the key-value store " + storage.string() + - " couldn't be initialized!"); - options_.create_if_missing = true; +struct KVStore::impl { + std::experimental::filesystem::path storage; + std::unique_ptr<rocksdb::DB> db; + rocksdb::Options options; +}; + +KVStore::KVStore(fs::path storage) : pimpl_(std::make_unique<impl>()) { + pimpl_->storage = storage; + if (!utils::EnsureDir(pimpl_->storage)) + throw KVStoreError("Folder for the key-value store " + + pimpl_->storage.string() + " couldn't be initialized!"); + pimpl_->options.create_if_missing = true; rocksdb::DB *db = nullptr; - auto s = rocksdb::DB::Open(options_, storage.c_str(), &db); + auto s = rocksdb::DB::Open(pimpl_->options, storage.c_str(), &db); if (!s.ok()) throw KVStoreError("RocksDB couldn't be initialized inside " + - storage.string() + "!"); - db_.reset(db); + storage.string() + " -- " + std::string(s.ToString())); + pimpl_->db.reset(db); +} + +KVStore::~KVStore() {} + +KVStore::KVStore(KVStore &&other) { pimpl_ = std::move(other.pimpl_); } + +KVStore &KVStore::operator=(KVStore &&other) { + pimpl_ = std::move(other.pimpl_); + return *this; } bool KVStore::Put(const std::string &key, const std::string &value) { - auto s = db_->Put(rocksdb::WriteOptions(), key.c_str(), value.c_str()); - if (!s.ok()) return false; - return true; + auto s = pimpl_->db->Put(rocksdb::WriteOptions(), key, value); + return s.ok(); } std::experimental::optional<std::string> KVStore::Get( const std::string &key) const noexcept { std::string value; - auto s = db_->Get(rocksdb::ReadOptions(), key.c_str(), &value); + auto s = pimpl_->db->Get(rocksdb::ReadOptions(), key, &value); if (!s.ok()) return std::experimental::nullopt; return value; } bool KVStore::Delete(const std::string &key) { - auto s = db_->SingleDelete(rocksdb::WriteOptions(), key.c_str()); - if (!s.ok()) return false; + auto s = pimpl_->db->Delete(rocksdb::WriteOptions(), key); + return s.ok(); +} + +bool KVStore::DeletePrefix(const std::string &prefix) { + std::unique_ptr<rocksdb::Iterator> iter = std::unique_ptr<rocksdb::Iterator>( + pimpl_->db->NewIterator(rocksdb::ReadOptions())); + for (iter->Seek(prefix); iter->Valid() && iter->key().starts_with(prefix); + iter->Next()) { + if (!pimpl_->db->Delete(rocksdb::WriteOptions(), iter->key()).ok()) + return false; + } return true; } +// iterator + +struct KVStore::iterator::impl { + const KVStore *kvstore; + std::string prefix; + std::unique_ptr<rocksdb::Iterator> it; + std::pair<std::string, std::string> disk_prop; +}; + +KVStore::iterator::iterator(const KVStore *kvstore, const std::string &prefix, + bool at_end) + : pimpl_(std::make_unique<impl>()) { + pimpl_->kvstore = kvstore; + pimpl_->prefix = prefix; + pimpl_->it = std::unique_ptr<rocksdb::Iterator>( + pimpl_->kvstore->pimpl_->db->NewIterator(rocksdb::ReadOptions())); + pimpl_->it->Seek(pimpl_->prefix); + if (!pimpl_->it->Valid() || !pimpl_->it->key().starts_with(pimpl_->prefix) || + at_end) + pimpl_->it = nullptr; +} + +KVStore::iterator::iterator(KVStore::iterator &&other) { + pimpl_ = std::move(other.pimpl_); +} + +KVStore::iterator::~iterator() {} + +KVStore::iterator &KVStore::iterator::operator=(KVStore::iterator &&other) { + pimpl_ = std::move(other.pimpl_); + return *this; +} + +KVStore::iterator &KVStore::iterator::operator++() { + pimpl_->it->Next(); + if (!pimpl_->it->Valid() || !pimpl_->it->key().starts_with(pimpl_->prefix)) + pimpl_->it = nullptr; + return *this; +} + +bool KVStore::iterator::operator==(const iterator &other) const { + return pimpl_->kvstore == other.pimpl_->kvstore && + pimpl_->prefix == other.pimpl_->prefix && + pimpl_->it == other.pimpl_->it; +} + +bool KVStore::iterator::operator!=(const iterator &other) const { + return !(*this == other); +} + +KVStore::iterator::reference KVStore::iterator::operator*() { + pimpl_->disk_prop = {pimpl_->it->key().ToString(), + pimpl_->it->value().ToString()}; + return pimpl_->disk_prop; +} + +KVStore::iterator::pointer KVStore::iterator::operator->() { return &**this; } + +void KVStore::iterator::SetInvalid() { pimpl_->it = nullptr; } + +bool KVStore::iterator::IsValid() { return pimpl_->it != nullptr; } + +// TODO(ipaljak) The complexity of the size function should be at most +// logarithmic. +size_t KVStore::Size(const std::string &prefix) { + size_t size = 0; + for (auto it = this->begin(prefix); it != this->end(prefix); ++it) ++size; + return size; +} + } // namespace storage diff --git a/src/storage/kvstore.hpp b/src/storage/kvstore.hpp index 2965d3f89..049a432e2 100644 --- a/src/storage/kvstore.hpp +++ b/src/storage/kvstore.hpp @@ -5,9 +5,6 @@ #include <memory> #include <string> -#include <rocksdb/db.h> -#include <rocksdb/options.h> - #include "utils/exceptions.hpp" namespace storage { @@ -23,6 +20,8 @@ class KVStoreError : public utils::BasicException { */ class KVStore final { public: + KVStore() = delete; + /** * @param storage Path to a directory where the data is persisted. * @@ -31,6 +30,14 @@ class KVStore final { */ explicit KVStore(std::experimental::filesystem::path storage); + KVStore(const KVStore &other) = delete; + KVStore(KVStore &&other); + + KVStore &operator=(const KVStore &other) = delete; + KVStore &operator=(KVStore &&other); + + ~KVStore(); + /** * Store value under the given key. * @@ -54,7 +61,7 @@ class KVStore final { noexcept; /** - * Delete value under the given key. + * Deletes the key and corresponding value from storage. * * @param key * @@ -64,10 +71,89 @@ class KVStore final { */ bool Delete(const std::string &key); + /** + * Delete all (key, value) pairs where key begins with a given prefix. + * + * @param prefix - prefix of the keys in (key, value) pairs to be deleted. + * This parameter is optional and is empty by default. + * + * @return True on success, false on error. The return value is + * true if the key doesn't exist and underlying storage + * didn't encounter any error. + */ + bool DeletePrefix(const std::string &prefix = ""); + + /** + * Returns total number of stored (key, value) pairs. The function takes an + * optional prefix parameter used for filtering keys that start with that + * prefix. + * + * @param prefix - prefix on which the keys should be filtered. This parameter + * is optional and is empty by default. + * + * @return - number of stored pairs. + */ + size_t Size(const std::string &prefix = ""); + + /** + * Custom prefix-based iterator over kvstore. + * + * It filters all (key, value) pairs where the key has a certain prefix + * and behaves as if all of those pairs are stored in a single iterable + * collection of std::pair<std::string, std::string>. + */ + class iterator final + : public std::iterator< + std::input_iterator_tag, // iterator_category + std::pair<std::string, std::string>, // value_type + long, // difference_type + const std::pair<std::string, std::string> *, // pointer + const std::pair<std::string, std::string> & // reference + > { + public: + explicit iterator(const KVStore *kvstore, const std::string &prefix = "", + bool at_end = false); + + iterator(const iterator &other) = delete; + + iterator(iterator &&other); + + ~iterator(); + + iterator &operator=(iterator &&other); + + iterator &operator=(const iterator &other) = delete; + + iterator &operator++(); + + bool operator==(const iterator &other) const; + + bool operator!=(const iterator &other) const; + + reference operator*(); + + pointer operator->(); + + void SetInvalid(); + + bool IsValid(); + + private: + struct impl; + std::unique_ptr<impl> pimpl_; + }; + + iterator begin(const std::string &prefix = "") { + return iterator(this, prefix); + } + + iterator end(const std::string &prefix = "") { + return iterator(this, prefix, true); + } + private: - std::experimental::filesystem::path storage_; - std::unique_ptr<rocksdb::DB> db_; - rocksdb::Options options_; + struct impl; + std::unique_ptr<impl> pimpl_; }; } // namespace storage diff --git a/src/storage/kvstore_dummy.cpp b/src/storage/kvstore_dummy.cpp new file mode 100644 index 000000000..e3fd5d6fa --- /dev/null +++ b/src/storage/kvstore_dummy.cpp @@ -0,0 +1,81 @@ +#include "storage/kvstore.hpp" + +#include "glog/logging.h" +#include "utils/file.hpp" + +namespace storage { + +struct KVStore::impl {}; + +KVStore::KVStore(fs::path storage) {} + +KVStore::~KVStore() {} + +bool KVStore::Put(const std::string &key, const std::string &value) { + CHECK(false) + << "Unsupported operation (KVStore::Put) -- this is a dummy kvstore"; +} + +std::experimental::optional<std::string> KVStore::Get( + const std::string &key) const noexcept { + CHECK(false) + << "Unsupported operation (KVStore::Get) -- this is a dummy kvstore"; +} + +bool KVStore::Delete(const std::string &key) { + CHECK(false) + << "Unsupported operation (KVStore::Delete) -- this is a dummy kvstore"; +} + +bool KVStore::DeletePrefix(const std::string &prefix) { + CHECK(false) << "Unsupported operation (KVStore::DeletePrefix) -- this is a " + "dummy kvstore"; +} + +// iterator + +struct KVStore::iterator::impl {}; + +KVStore::iterator::iterator(const KVStore *kvstore, const std::string &prefix, + bool at_end) + : pimpl_(new impl()) {} + +KVStore::iterator::iterator(KVStore::iterator &&other) { + pimpl_ = std::move(other.pimpl_); +} + +KVStore::iterator::~iterator() {} + +KVStore::iterator &KVStore::iterator::operator=(KVStore::iterator &&other) { + pimpl_ = std::move(other.pimpl_); + return *this; +} + +KVStore::iterator &KVStore::iterator::operator++() { + CHECK(false) << "Unsupported operation (&KVStore::iterator::operator++) -- " + "this is a dummy kvstore"; +} + +bool KVStore::iterator::operator==(const iterator &other) const { return true; } + +bool KVStore::iterator::operator!=(const iterator &other) const { + return false; +} + +KVStore::iterator::reference KVStore::iterator::operator*() { + CHECK(false) << "Unsupported operation (KVStore::iterator::operator*)-- this " + "is a dummy kvstore"; +} + +KVStore::iterator::pointer KVStore::iterator::operator->() { + CHECK(false) << "Unsupported operation (KVStore::iterator::operator->) -- " + "this is a dummy kvstore"; +} + +void KVStore::iterator::SetInvalid() {} + +bool KVStore::iterator::IsValid() { return false; } + +size_t KVStore::Size(const std::string &prefix) { return 0; } + +} // namespace storage diff --git a/src/storage/kvstore_mock.hpp b/src/storage/kvstore_mock.hpp deleted file mode 100644 index cd4ede1e4..000000000 --- a/src/storage/kvstore_mock.hpp +++ /dev/null @@ -1,40 +0,0 @@ -#pragma once - -#include <experimental/optional> -#include <map> -#include <string> - -#include <glog/logging.h> - -// TODO(ipaljak): replace with the real implementation - -namespace storage { - -class KVStore { - public: - explicit KVStore(const std::string &) {} - - bool Put(const std::string &key, const std::string &value) { - VLOG(31) << "PUT: " << key << " : " << value; - storage_[key] = value; - return true; - } - - std::experimental::optional<std::string> Get(const std::string &key) const { - VLOG(31) << "GET: " << key; - auto it = storage_.find(key); - if (it == storage_.end()) return std::experimental::nullopt; - return it->second; - } - - bool Delete(const std::string &key) { - VLOG(31) << "DELETE: " << key; - storage_.erase(key); - return true; - } - - private: - std::map<std::string, std::string> storage_; -}; - -} // namespace storage diff --git a/src/storage/pod_buffer.hpp b/src/storage/pod_buffer.hpp new file mode 100644 index 000000000..45c17203e --- /dev/null +++ b/src/storage/pod_buffer.hpp @@ -0,0 +1,46 @@ +#pragma once + +#include "communication/bolt/v1/encoder/base_encoder.hpp" + +namespace storage { + +/** + * Buffer used for serialization of disk properties. The buffer + * implements a template parameter Buffer interface from BaseEncoder + * and Decoder classes for bolt serialization. + */ +class PODBuffer { + public: + PODBuffer() = default; + explicit PODBuffer(const std::string &s) { + buffer = std::vector<uint8_t>{s.begin(), s.end()}; + } + + /** + * Writes data to buffer + * + * @param data - Pointer to data to be written. + * @param len - Data length. + */ + void Write(const uint8_t *data, size_t len) { + for (size_t i = 0; i < len; ++i) buffer.push_back(data[i]); + } + + /** + * Reads raw data from buffer. + * + * @param data - pointer to where data should be stored. + * @param len - data length + * @return - True if successful, False otherwise. + */ + bool Read(uint8_t *data, size_t len) { + if (len > buffer.size()) return false; + memcpy(data, buffer.data(), len); + buffer.erase(buffer.begin(), buffer.begin() + len); + return true; + } + + std::vector<uint8_t> buffer; +}; + +} // namespace storage diff --git a/src/storage/property_value_store.cpp b/src/storage/property_value_store.cpp index 37c5a61d3..3ad9c6965 100644 --- a/src/storage/property_value_store.cpp +++ b/src/storage/property_value_store.cpp @@ -1,10 +1,52 @@ +#include <experimental/filesystem> + +#include "gflags/gflags.h" +#include "glog/logging.h" + +#include "storage/pod_buffer.hpp" #include "storage/property_value_store.hpp" -using Property = storage::Property; -using Location = storage::Location; +namespace fs = std::experimental::filesystem; -std::atomic<uint64_t> PropertyValueStore::disk_key_cnt_ = {0}; -storage::KVStore PropertyValueStore::disk_storage_("properties"); +using namespace communication::bolt; + +std::atomic<uint64_t> PropertyValueStore::global_key_cnt_ = {0}; + +// properties on disk are stored in a directory named properties within the +// durability directory +DECLARE_string(durability_directory); +DECLARE_string(properties_on_disk); + +std::string DiskKeyPrefix(const std::string &version_key) { + return version_key + disk_key_separator; +} + +std::string DiskKey(const std::string &version_key, + const std::string &property_id) { + return DiskKeyPrefix(version_key) + property_id; +} + +PropertyValueStore::PropertyValueStore(const PropertyValueStore &old) + : props_(old.props_) { + // We need to update disk key and disk key counter when calling a copy + // constructor due to mvcc. + if (!FLAGS_properties_on_disk.empty()) { + version_key_ = global_key_cnt_++; + storage::KVStore::iterator old_disk_it( + &DiskStorage(), DiskKeyPrefix(std::to_string(old.version_key_))); + iterator it(&old, old.props_.end(), std::move(old_disk_it)); + + while (it != old.end()) { + this->set(it->first, it->second); + ++it; + } + } +} + +PropertyValueStore::~PropertyValueStore() { + if (!FLAGS_properties_on_disk.empty()) + DiskStorage().DeletePrefix(DiskKeyPrefix(std::to_string(version_key_))); +} PropertyValue PropertyValueStore::at(const Property &key) const { auto GetValue = [&key](const auto &props) { @@ -15,7 +57,11 @@ PropertyValue PropertyValueStore::at(const Property &key) const { if (key.Location() == Location::Memory) return GetValue(props_); - return GetValue(PropsOnDisk(std::to_string(disk_key_))); + std::string disk_key = + DiskKey(std::to_string(version_key_), std::to_string(key.Id())); + auto serialized_prop = DiskStorage().Get(disk_key); + if (serialized_prop) return DeserializeProp(serialized_prop.value()); + return PropertyValue::Null; } void PropertyValueStore::set(const Property &key, const char *value) { @@ -40,80 +86,63 @@ void PropertyValueStore::set(const Property &key, const PropertyValue &value) { if (key.Location() == Location::Memory) { SetValue(props_); } else { - auto props_on_disk = PropsOnDisk(std::to_string(disk_key_)); - SetValue(props_on_disk); - disk_storage_.Put(std::to_string(disk_key_), SerializeProps(props_on_disk)); + std::string disk_key = + DiskKey(std::to_string(version_key_), std::to_string(key.Id())); + DiskStorage().Put(disk_key, SerializeProp(value)); } } -size_t PropertyValueStore::erase(const Property &key) { +bool PropertyValueStore::erase(const Property &key) { auto EraseKey = [&key](auto &props) { auto found = std::find_if(props.begin(), props.end(), [&key](std::pair<Property, PropertyValue> &kv) { return kv.first == key; }); - if (found != props.end()) { - props.erase(found); - return true; - } - return false; + if (found != props.end()) props.erase(found); + return true; }; if (key.Location() == Location::Memory) return EraseKey(props_); - auto props_on_disk = PropsOnDisk(std::to_string(disk_key_)); - if (EraseKey(props_on_disk)) { - if (props_on_disk.empty()) - return disk_storage_.Delete(std::to_string(disk_key_)); - return disk_storage_.Put(std::to_string(disk_key_), - SerializeProps(props_on_disk)); - } - - return false; + std::string disk_key = + DiskKey(std::to_string(version_key_), std::to_string(key.Id())); + return DiskStorage().Delete(disk_key); } void PropertyValueStore::clear() { props_.clear(); - disk_storage_.Delete(std::to_string(disk_key_)); + if (!FLAGS_properties_on_disk.empty()) + DiskStorage().DeletePrefix(DiskKeyPrefix(std::to_string(version_key_))); } -/* TODO(ipaljak): replace serialize/deserialize with the real implementation. - * Currently supporting a only one property on disk per record and that property - * must be a string. - * */ -std::string PropertyValueStore::SerializeProps( - const std::vector<std::pair<Property, PropertyValue>> &props) const { - if (props.size() > 1) throw std::runtime_error("Unsupported operation"); - std::stringstream strstream; - strstream << props[0].first.Id() << "," << props[0].second; - return strstream.str(); +storage::KVStore &PropertyValueStore::DiskStorage() const { + static auto disk_storage = ConstructDiskStorage(); + return disk_storage; } -std::vector<std::pair<Property, PropertyValue>> PropertyValueStore::Deserialize( - const std::string &serialized_props) const { - std::istringstream strstream(serialized_props); - - std::string s; - std::getline(strstream, s, ','); - - uint16_t id; - std::istringstream ss(s); - ss >> id; - - Property key(id, Location::Disk); - - std::getline(strstream, s, ','); - PropertyValue value(s); - - std::vector<std::pair<Property, PropertyValue>> ret; - ret.emplace_back(key, value); - - return ret; +std::string PropertyValueStore::SerializeProp(const PropertyValue &prop) const { + storage::PODBuffer pod_buffer; + BaseEncoder<storage::PODBuffer> encoder{pod_buffer}; + encoder.WriteTypedValue(prop); + return std::string(reinterpret_cast<char *>(pod_buffer.buffer.data()), + pod_buffer.buffer.size()); } -std::vector<std::pair<Property, PropertyValue>> PropertyValueStore::PropsOnDisk( - const std::string &disk_key) const { - auto serialized = disk_storage_.Get(disk_key); - if (serialized) return Deserialize(disk_storage_.Get(disk_key).value()); - return {}; +PropertyValue PropertyValueStore::DeserializeProp( + const std::string &serialized_prop) const { + storage::PODBuffer pod_buffer{serialized_prop}; + Decoder<storage::PODBuffer> decoder{pod_buffer}; + + DecodedValue dv; + if (!decoder.ReadValue(&dv)) { + DLOG(WARNING) << "Unable to read property value"; + return PropertyValue::Null; + } + return dv.operator PropertyValue(); +} + +storage::KVStore PropertyValueStore::ConstructDiskStorage() const { + auto storage_path = fs::path() / FLAGS_durability_directory / "properties"; + if (fs::exists(storage_path)) fs::remove_all(storage_path); + return storage::KVStore(storage_path); } diff --git a/src/storage/property_value_store.hpp b/src/storage/property_value_store.hpp index e94c63837..6917054d3 100644 --- a/src/storage/property_value_store.hpp +++ b/src/storage/property_value_store.hpp @@ -1,32 +1,33 @@ #pragma once -#include <algorithm> + #include <atomic> #include <experimental/optional> -#include <map> -#include <memory> +#include <string> #include <vector> -#include "glog/logging.h" - -#include "storage/kvstore_mock.hpp" +#include "storage/kvstore.hpp" #include "storage/property_value.hpp" #include "storage/types.hpp" + +const std::string disk_key_separator = "_"; + +std::string DiskKey(const std::string &version_key, + const std::string &property_id); + +std::string DiskKeyPrefix(const std::string &version_key); + /** * A collection of properties accessed in a map-like way using a key of type - * Properties::Property. + * Storage::Property. * * PropertyValueStore handles storage on disk or in memory. Property key defines * where the corresponding property should be stored. Each instance of - * PropertyValueStore contains a disk_key_ member which specifies where on + * PropertyValueStore contains a version_key_ member which specifies where on * disk should the properties be stored. That key is inferred from a static - * global counter disk_key_cnt_. + * global counter global_key_cnt_. * * The underlying implementation of in-memory storage is not necessarily * std::map. - * - * TODO(ipaljak) modify on-disk storage so that each property has its own - * key for storage. Storage key should, in essence, be an ordered pair - * (global key, property key). */ class PropertyValueStore { using Property = storage::Property; @@ -38,18 +39,9 @@ class PropertyValueStore { static constexpr char IdPropertyName[] = "__id__"; PropertyValueStore() = default; + PropertyValueStore(const PropertyValueStore &old); - PropertyValueStore(const PropertyValueStore &old) { - // We need to update disk key and disk key counter when calling a copy - // constructor due to mvcc. - props_ = old.props_; - disk_key_ = disk_key_cnt_++; - auto old_value = disk_storage_.Get(std::to_string(old.disk_key_)); - if (old_value) - disk_storage_.Put(std::to_string(disk_key_), old_value.value()); - } - - ~PropertyValueStore() { disk_storage_.Delete(std::to_string(disk_key_)); } + ~PropertyValueStore(); /** * Returns a PropertyValue (by reference) at the given key. @@ -63,39 +55,6 @@ class PropertyValueStore { */ PropertyValue at(const Property &key) const; - /** - * Sets the value for the given key. A new PropertyValue instance - * is created for the given value (which is of raw type). If the property - * is to be stored on disk then that instance does not represent an additional - * memory overhead as it goes out of scope at the end of this method. - * - * @tparam TValue Type of value. It must be one of the - * predefined possible PropertyValue values (bool, string, int...) - * @param key The key for which the property is set. The previous - * value at the same key (if there was one) is replaced. - * @param value The value to set. - */ - template <typename TValue> - void set(const Property &key, const TValue &value) { - auto SetValue = [&key, &value](auto &props) { - for (auto &kv : props) - if (kv.first == key) { - kv.second = value; - return; - } - props.emplace_back(key, value); - }; - - if (key.Location() == Location::Memory) { - SetValue(props_); - } else { - auto props_on_disk = PropsOnDisk(std::to_string(disk_key_)); - SetValue(props_on_disk); - disk_storage_.Put(std::to_string(disk_key_), - SerializeProps(props_on_disk)); - } - } - /** * Set overriding for character constants. Forces conversion * to std::string, otherwise templating might cast the pointer @@ -112,52 +71,74 @@ class PropertyValueStore { /** * Removes the PropertyValue for the given key. * - * @param key The key for which to remove the property. - * @return The number of removed properties (0 or 1). + * @param key - The key for which to remove the property. + * + * @return true if the operation was successful and there is nothing stored + * under given key after this operation. */ - size_t erase(const Property &key); + bool erase(const Property &key); /** Removes all the properties (both in-mem and on-disk) from this store. */ void clear(); + /** + * Returns a static storage::kvstore instance used for storing properties on + * disk. This hack is needed due to statics that are internal to rocksdb and + * availability of durability_directory flag. + */ + storage::KVStore &DiskStorage() const; + /** * Custom PVS iterator behaves as if all properties are stored in a single * iterable collection of std::pair<Property, PropertyValue>. - * */ - class iterator : public std::iterator< - std::input_iterator_tag, // iterator_category - std::pair<Property, PropertyValue>, // value_type - long, // difference_type - const std::pair<Property, PropertyValue> *, // pointer - const std::pair<Property, PropertyValue> & // reference - > { + */ + class iterator final + : public std::iterator< + std::input_iterator_tag, // iterator_category + std::pair<Property, PropertyValue>, // value_type + long, // difference_type + const std::pair<Property, PropertyValue> *, // pointer + const std::pair<Property, PropertyValue> & // reference + > { public: - explicit iterator(const PropertyValueStore *init, int it) - : PVS_(init), it_(it) {} + iterator() = delete; + + iterator(const PropertyValueStore *PVS, + std::vector<std::pair<Property, PropertyValue>>::const_iterator + memory_it, + storage::KVStore::iterator disk_it) + : PVS_(PVS), memory_it_(memory_it), disk_it_(std::move(disk_it)) {} + + iterator(const iterator &other) = delete; + + iterator(iterator &&other) = default; + + iterator &operator=(iterator &&other) = default; + + iterator &operator=(const iterator &other) = delete; iterator &operator++() { - ++it_; + if (memory_it_ != PVS_->props_.end()) + ++memory_it_; + else + ++disk_it_; return *this; } - iterator operator++(int) { - iterator ret = *this; - ++(*this); - return ret; - } - bool operator==(const iterator &other) const { - return PVS_ == other.PVS_ && it_ == other.it_; + return PVS_ == other.PVS_ && memory_it_ == other.memory_it_ && + disk_it_ == other.disk_it_; } - bool operator!=(const iterator &other) const { - return PVS_ != other.PVS_ || it_ != other.it_; - } + bool operator!=(const iterator &other) const { return !(*this == other); } reference operator*() { - if (it_ < static_cast<int>(PVS_->props_.size())) return PVS_->props_[it_]; - auto disk_props = PVS_->PropsOnDisk(std::to_string(PVS_->disk_key_)); - disk_prop_ = disk_props[it_ - PVS_->props_.size()]; + if (memory_it_ != PVS_->props_.end()) return *memory_it_; + std::pair<std::string, std::string> kv = *disk_it_; + std::string prop_id = + kv.first.substr(kv.first.find(disk_key_separator) + 1); + disk_prop_ = {Property(std::stoi(prop_id), Location::Disk), + PVS_->DeserializeProp(kv.second)}; return disk_prop_.value(); } @@ -165,33 +146,52 @@ class PropertyValueStore { private: const PropertyValueStore *PVS_; - int32_t it_; + std::vector<std::pair<Property, PropertyValue>>::const_iterator memory_it_; + storage::KVStore::iterator disk_it_; std::experimental::optional<std::pair<Property, PropertyValue>> disk_prop_; }; size_t size() const { - size_t ram_size = props_.size(); - size_t disk_size = PropsOnDisk(std::to_string(disk_key_)).size(); - return ram_size + disk_size; + return props_.size() + + DiskStorage().Size(DiskKeyPrefix(std::to_string(version_key_))); } - auto begin() const { return iterator(this, 0); } - auto end() const { return iterator(this, size()); } + iterator begin() const { + return iterator( + this, props_.begin(), + DiskStorage().begin(DiskKeyPrefix(std::to_string(version_key_)))); + } + + iterator end() const { + return iterator( + this, props_.end(), + DiskStorage().end(DiskKeyPrefix(std::to_string(version_key_)))); + } private: - static std::atomic<uint64_t> disk_key_cnt_; - uint64_t disk_key_ = disk_key_cnt_++; - - static storage::KVStore disk_storage_; + static std::atomic<uint64_t> global_key_cnt_; + uint64_t version_key_ = global_key_cnt_++; std::vector<std::pair<Property, PropertyValue>> props_; - std::string SerializeProps( - const std::vector<std::pair<Property, PropertyValue>> &props) const; + /** + * Serializes a single PropertyValue into std::string. + * + * @param prop - Property to be serialized. + * + * @return Serialized property. + */ + std::string SerializeProp(const PropertyValue &prop) const; - std::vector<std::pair<Property, PropertyValue>> Deserialize( - const std::string &serialized_props) const; + /** + * Deserializes a single PropertyValue from std::string. + * + * @param serialized_prop - Serialized property. + * + * @return Deserialized property. + */ + PropertyValue DeserializeProp(const std::string &serialized_prop) const; - std::vector<std::pair<Property, PropertyValue>> PropsOnDisk( - const std::string &disk_key) const; + storage::KVStore ConstructDiskStorage() const; }; + diff --git a/tests/benchmark/CMakeLists.txt b/tests/benchmark/CMakeLists.txt index 37f380df8..800ea2843 100644 --- a/tests/benchmark/CMakeLists.txt +++ b/tests/benchmark/CMakeLists.txt @@ -24,7 +24,7 @@ foreach(test_cpp ${test_type_cpps}) set_target_properties(${target_name} PROPERTIES OUTPUT_NAME ${exec_name}) # link libraries - target_link_libraries(${target_name} memgraph_lib) + target_link_libraries(${target_name} memgraph_lib kvstore_dummy_lib) # google-benchmark target_link_libraries(${target_name} benchmark Threads::Threads) diff --git a/tests/distributed/raft/CMakeLists.txt b/tests/distributed/raft/CMakeLists.txt index c3624024a..f09b290c7 100644 --- a/tests/distributed/raft/CMakeLists.txt +++ b/tests/distributed/raft/CMakeLists.txt @@ -22,7 +22,7 @@ foreach(test_cpp ${test_type_cpps}) set_target_properties(${target_name} PROPERTIES OUTPUT_NAME ${exec_name}) # link libraries - target_link_libraries(${target_name} memgraph_lib) + target_link_libraries(${target_name} memgraph_lib kvstore_dummy_lib) set(output_path ${CMAKE_BINARY_DIR}/test_results/unit/${target_name}.xml) diff --git a/tests/macro_benchmark/CMakeLists.txt b/tests/macro_benchmark/CMakeLists.txt index 6e09f79a8..b6c0fb076 100644 --- a/tests/macro_benchmark/CMakeLists.txt +++ b/tests/macro_benchmark/CMakeLists.txt @@ -26,7 +26,7 @@ foreach(test_cpp ${test_type_cpps}) set_target_properties(${target_name} PROPERTIES OUTPUT_NAME ${exec_name}) # link libraries - target_link_libraries(${target_name} memgraph_lib) + target_link_libraries(${target_name} memgraph_lib kvstore_dummy_lib) # add target to dependencies add_dependencies(${all_targets_target} ${target_name}) diff --git a/tests/manual/CMakeLists.txt b/tests/manual/CMakeLists.txt index bfa70998b..be65a819b 100644 --- a/tests/manual/CMakeLists.txt +++ b/tests/manual/CMakeLists.txt @@ -28,48 +28,45 @@ add_manual_test(binomial.cpp) target_link_libraries(${test_prefix}binomial mg-utils) add_manual_test(bolt_client.cpp) -target_link_libraries(${test_prefix}bolt_client memgraph_lib) +target_link_libraries(${test_prefix}bolt_client memgraph_lib kvstore_dummy_lib) add_manual_test(card_fraud_generate_snapshot.cpp) -target_link_libraries(${test_prefix}card_fraud_generate_snapshot memgraph_lib) +target_link_libraries(${test_prefix}card_fraud_generate_snapshot memgraph_lib kvstore_dummy_lib) add_manual_test(card_fraud_local.cpp) -target_link_libraries(${test_prefix}card_fraud_local memgraph_lib) +target_link_libraries(${test_prefix}card_fraud_local memgraph_lib kvstore_dummy_lib) add_manual_test(distributed_repl.cpp) -target_link_libraries(${test_prefix}distributed_repl memgraph_lib) +target_link_libraries(${test_prefix}distributed_repl memgraph_lib kvstore_dummy_lib) add_manual_test(endinan.cpp) add_manual_test(generate_snapshot.cpp) -target_link_libraries(${test_prefix}generate_snapshot memgraph_lib) +target_link_libraries(${test_prefix}generate_snapshot memgraph_lib kvstore_dummy_lib) add_manual_test(graph_500_generate_snapshot.cpp) -target_link_libraries(${test_prefix}graph_500_generate_snapshot memgraph_lib) - -add_manual_test(kvstore.cpp) -target_link_libraries(${test_prefix}kvstore gtest gtest_main memgraph_lib kvstore_lib) +target_link_libraries(${test_prefix}graph_500_generate_snapshot memgraph_lib kvstore_dummy_lib) add_manual_test(query_hash.cpp) -target_link_libraries(${test_prefix}query_hash memgraph_lib) +target_link_libraries(${test_prefix}query_hash memgraph_lib kvstore_dummy_lib) add_manual_test(query_planner.cpp) -target_link_libraries(${test_prefix}query_planner memgraph_lib) +target_link_libraries(${test_prefix}query_planner memgraph_lib kvstore_dummy_lib) add_manual_test(raft_rpc.cpp) -target_link_libraries(${test_prefix}raft_rpc memgraph_lib) +target_link_libraries(${test_prefix}raft_rpc memgraph_lib kvstore_dummy_lib) add_manual_test(repl.cpp) -target_link_libraries(${test_prefix}repl memgraph_lib) +target_link_libraries(${test_prefix}repl memgraph_lib kvstore_dummy_lib) add_manual_test(single_query.cpp) -target_link_libraries(${test_prefix}single_query memgraph_lib) +target_link_libraries(${test_prefix}single_query memgraph_lib kvstore_dummy_lib) add_manual_test(sl_position_and_count.cpp) -target_link_libraries(${test_prefix}sl_position_and_count memgraph_lib) +target_link_libraries(${test_prefix}sl_position_and_count memgraph_lib kvstore_dummy_lib) add_manual_test(stripped_timing.cpp) -target_link_libraries(${test_prefix}stripped_timing memgraph_lib) +target_link_libraries(${test_prefix}stripped_timing memgraph_lib kvstore_dummy_lib) add_manual_test(xorshift.cpp) target_link_libraries(${test_prefix}xorshift mg-utils) diff --git a/tests/manual/kvstore.cpp b/tests/manual/kvstore.cpp deleted file mode 100644 index f8c2e7e46..000000000 --- a/tests/manual/kvstore.cpp +++ /dev/null @@ -1,48 +0,0 @@ -#include <glog/logging.h> -#include <gtest/gtest.h> - -#include "durability/paths.hpp" -#include "storage/kvstore.hpp" -#include "utils/file.hpp" - -namespace fs = std::experimental::filesystem; - -class KVStore : public ::testing::Test { - protected: - virtual void SetUp() { utils::EnsureDir(test_folder_); } - - virtual void TearDown() { fs::remove_all(test_folder_); } - - fs::path test_folder_{fs::path("kvstore_test")}; -}; - -TEST_F(KVStore, PutGet) { - storage::KVStore kvstore(test_folder_ / "PutGet"); - ASSERT_TRUE(kvstore.Put("key", "value")); - ASSERT_EQ(kvstore.Get("key").value(), "value"); -} - -TEST_F(KVStore, PutGetDeleteGet) { - storage::KVStore kvstore(test_folder_ / "PutGetDeleteGet"); - ASSERT_TRUE(kvstore.Put("key", "value")); - ASSERT_EQ(kvstore.Get("key").value(), "value"); - ASSERT_TRUE(kvstore.Delete("key")); - ASSERT_FALSE(static_cast<bool>(kvstore.Get("key"))); -} - -TEST_F(KVStore, Durability) { - { - storage::KVStore kvstore(test_folder_ / "Durability"); - ASSERT_TRUE(kvstore.Put("key", "value")); - } - { - storage::KVStore kvstore(test_folder_ / "Durability"); - ASSERT_EQ(kvstore.Get("key").value(), "value"); - } -} - -int main(int argc, char **argv) { - google::InitGoogleLogging(argv[0]); - ::testing::InitGoogleTest(&argc, argv); - return RUN_ALL_TESTS(); -} diff --git a/tests/property_based/CMakeLists.txt b/tests/property_based/CMakeLists.txt index bb1f7c478..0f18d0f55 100644 --- a/tests/property_based/CMakeLists.txt +++ b/tests/property_based/CMakeLists.txt @@ -22,7 +22,7 @@ foreach(test_cpp ${test_type_cpps}) set_target_properties(${target_name} PROPERTIES OUTPUT_NAME ${exec_name}) # link libraries - target_link_libraries(${target_name} memgraph_lib) + target_link_libraries(${target_name} memgraph_lib kvstore_dummy_lib) # gtest target_link_libraries(${target_name} gtest gtest_main) target_link_libraries(${target_name} rapidcheck rapidcheck_gtest) diff --git a/tests/qa/continuous_integration b/tests/qa/continuous_integration index 1054e7b0d..f91fe666f 100755 --- a/tests/qa/continuous_integration +++ b/tests/qa/continuous_integration @@ -54,7 +54,7 @@ suites = [ Test( name="memgraph_V1_POD", test_suite="memgraph_V1", - memgraph_params="--properties-on-disk=surname,location", + memgraph_params="--properties-on-disk=x,y,z,w,k,v,a,b,c,d,e,f,r,t,o,prop,age,name,surname,location", mandatory=True ), Test( @@ -62,7 +62,7 @@ suites = [ test_suite="openCypher_M09", memgraph_params="", mandatory=False - ) + ), ] results_folder = os.path.join("tck_engine", "results") diff --git a/tests/stress/CMakeLists.txt b/tests/stress/CMakeLists.txt index 6e09f79a8..b6c0fb076 100644 --- a/tests/stress/CMakeLists.txt +++ b/tests/stress/CMakeLists.txt @@ -26,7 +26,7 @@ foreach(test_cpp ${test_type_cpps}) set_target_properties(${target_name} PROPERTIES OUTPUT_NAME ${exec_name}) # link libraries - target_link_libraries(${target_name} memgraph_lib) + target_link_libraries(${target_name} memgraph_lib kvstore_dummy_lib) # add target to dependencies add_dependencies(${all_targets_target} ${target_name}) diff --git a/tests/stress/apollo_runs.yaml b/tests/stress/apollo_runs.yaml index 33b242245..89e64370a 100644 --- a/tests/stress/apollo_runs.yaml +++ b/tests/stress/apollo_runs.yaml @@ -6,6 +6,10 @@ - ../../config # directory with config files - ../../build_release/tests/stress # stress client binaries +- name: stress_properties_on_disk + commands: TIMEOUT=600 ./continuous_integration --properties-on-disk + infiles: *STRESS_INFILES + - name: stress_large project: release commands: TIMEOUT=43200 ./continuous_integration --large-dataset diff --git a/tests/stress/continuous_integration b/tests/stress/continuous_integration index 290661aec..124585aa1 100755 --- a/tests/stress/continuous_integration +++ b/tests/stress/continuous_integration @@ -127,6 +127,7 @@ parser.add_argument("--config", default = os.path.join(CONFIG_DIR, "stress.conf")) parser.add_argument("--log-file", default = "") parser.add_argument("--durability-directory", default = "") +parser.add_argument("--properties-on-disk", action = "store_true") parser.add_argument("--python", default = os.path.join(SCRIPT_DIR, "ve3", "bin", "python3"), type = str) parser.add_argument("--large-dataset", action = "store_const", @@ -148,6 +149,8 @@ if args.log_file: cmd += ["--log-file", args.log_file] if args.durability_directory: cmd += ["--durability-directory", args.durability_directory] +if args.properties_on_disk: + cmd += ["--properties-on-disk", "id,x"] proc_mg = subprocess.Popen(cmd, cwd = cwd, env = {"MEMGRAPH_CONFIG": args.config}) time.sleep(1.0) diff --git a/tests/unit/CMakeLists.txt b/tests/unit/CMakeLists.txt index a4f5e45f8..0a2b0f73d 100644 --- a/tests/unit/CMakeLists.txt +++ b/tests/unit/CMakeLists.txt @@ -20,223 +20,229 @@ function(add_unit_test test_cpp) endfunction(add_unit_test) add_unit_test(bolt_chunked_decoder_buffer.cpp) -target_link_libraries(${test_prefix}bolt_chunked_decoder_buffer memgraph_lib) +target_link_libraries(${test_prefix}bolt_chunked_decoder_buffer memgraph_lib kvstore_dummy_lib) add_unit_test(bolt_chunked_encoder_buffer.cpp) -target_link_libraries(${test_prefix}bolt_chunked_encoder_buffer memgraph_lib) +target_link_libraries(${test_prefix}bolt_chunked_encoder_buffer memgraph_lib kvstore_dummy_lib) add_unit_test(bolt_decoder.cpp) -target_link_libraries(${test_prefix}bolt_decoder memgraph_lib) +target_link_libraries(${test_prefix}bolt_decoder memgraph_lib kvstore_dummy_lib) add_unit_test(bolt_encoder.cpp) -target_link_libraries(${test_prefix}bolt_encoder memgraph_lib) +target_link_libraries(${test_prefix}bolt_encoder memgraph_lib kvstore_dummy_lib) add_unit_test(bolt_result_stream.cpp) -target_link_libraries(${test_prefix}bolt_result_stream memgraph_lib) +target_link_libraries(${test_prefix}bolt_result_stream memgraph_lib kvstore_dummy_lib) add_unit_test(bolt_session.cpp) -target_link_libraries(${test_prefix}bolt_session memgraph_lib) +target_link_libraries(${test_prefix}bolt_session memgraph_lib kvstore_dummy_lib) add_unit_test(communication_buffer.cpp) -target_link_libraries(${test_prefix}communication_buffer memgraph_lib) +target_link_libraries(${test_prefix}communication_buffer memgraph_lib kvstore_dummy_lib) add_unit_test(concurrent_id_mapper_distributed.cpp) -target_link_libraries(${test_prefix}concurrent_id_mapper_distributed memgraph_lib) +target_link_libraries(${test_prefix}concurrent_id_mapper_distributed memgraph_lib kvstore_dummy_lib) add_unit_test(concurrent_id_mapper_single_node.cpp) -target_link_libraries(${test_prefix}concurrent_id_mapper_single_node memgraph_lib) +target_link_libraries(${test_prefix}concurrent_id_mapper_single_node memgraph_lib kvstore_dummy_lib) add_unit_test(concurrent_map_access.cpp) -target_link_libraries(${test_prefix}concurrent_map_access memgraph_lib) +target_link_libraries(${test_prefix}concurrent_map_access memgraph_lib kvstore_dummy_lib) add_unit_test(concurrent_map.cpp) -target_link_libraries(${test_prefix}concurrent_map memgraph_lib) +target_link_libraries(${test_prefix}concurrent_map memgraph_lib kvstore_dummy_lib) add_unit_test(counters.cpp) -target_link_libraries(${test_prefix}counters memgraph_lib) +target_link_libraries(${test_prefix}counters memgraph_lib kvstore_dummy_lib) add_unit_test(cypher_main_visitor.cpp) -target_link_libraries(${test_prefix}cypher_main_visitor memgraph_lib) +target_link_libraries(${test_prefix}cypher_main_visitor memgraph_lib kvstore_dummy_lib) add_unit_test(database_key_index.cpp) -target_link_libraries(${test_prefix}database_key_index memgraph_lib) +target_link_libraries(${test_prefix}database_key_index memgraph_lib kvstore_dummy_lib) add_unit_test(database_label_property_index.cpp) -target_link_libraries(${test_prefix}database_label_property_index memgraph_lib) +target_link_libraries(${test_prefix}database_label_property_index memgraph_lib kvstore_dummy_lib) add_unit_test(database_master.cpp) -target_link_libraries(${test_prefix}database_master memgraph_lib) +target_link_libraries(${test_prefix}database_master memgraph_lib kvstore_dummy_lib) add_unit_test(database_transaction_timeout.cpp) -target_link_libraries(${test_prefix}database_transaction_timeout memgraph_lib) +target_link_libraries(${test_prefix}database_transaction_timeout memgraph_lib kvstore_dummy_lib) add_unit_test(datastructure_union_find.cpp) -target_link_libraries(${test_prefix}datastructure_union_find memgraph_lib) +target_link_libraries(${test_prefix}datastructure_union_find memgraph_lib kvstore_dummy_lib) add_unit_test(deferred_deleter.cpp) -target_link_libraries(${test_prefix}deferred_deleter memgraph_lib) +target_link_libraries(${test_prefix}deferred_deleter memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_bfs.cpp) -target_link_libraries(${test_prefix}distributed_bfs memgraph_lib) +target_link_libraries(${test_prefix}distributed_bfs memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_coordination.cpp) -target_link_libraries(${test_prefix}distributed_coordination memgraph_lib) +target_link_libraries(${test_prefix}distributed_coordination memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_data_exchange.cpp) -target_link_libraries(${test_prefix}distributed_data_exchange memgraph_lib) +target_link_libraries(${test_prefix}distributed_data_exchange memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_durability.cpp) -target_link_libraries(${test_prefix}distributed_durability memgraph_lib) +target_link_libraries(${test_prefix}distributed_durability memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_gc.cpp) -target_link_libraries(${test_prefix}distributed_gc memgraph_lib) +target_link_libraries(${test_prefix}distributed_gc memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_graph_db.cpp) -target_link_libraries(${test_prefix}distributed_graph_db memgraph_lib) +target_link_libraries(${test_prefix}distributed_graph_db memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_interpretation.cpp) -target_link_libraries(${test_prefix}distributed_interpretation memgraph_lib) +target_link_libraries(${test_prefix}distributed_interpretation memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_query_plan.cpp) -target_link_libraries(${test_prefix}distributed_query_plan memgraph_lib) +target_link_libraries(${test_prefix}distributed_query_plan memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_serialization.cpp) -target_link_libraries(${test_prefix}distributed_serialization memgraph_lib) +target_link_libraries(${test_prefix}distributed_serialization memgraph_lib kvstore_dummy_lib) add_unit_test(distributed_updates.cpp) -target_link_libraries(${test_prefix}distributed_updates memgraph_lib) +target_link_libraries(${test_prefix}distributed_updates memgraph_lib kvstore_dummy_lib) add_unit_test(durability.cpp) -target_link_libraries(${test_prefix}durability memgraph_lib) +target_link_libraries(${test_prefix}durability memgraph_lib kvstore_dummy_lib) add_unit_test(dynamic_bitset.cpp) -target_link_libraries(${test_prefix}dynamic_bitset memgraph_lib) +target_link_libraries(${test_prefix}dynamic_bitset memgraph_lib kvstore_dummy_lib) add_unit_test(gid.cpp) -target_link_libraries(${test_prefix}gid memgraph_lib) +target_link_libraries(${test_prefix}gid memgraph_lib kvstore_dummy_lib) add_unit_test(graph_db_accessor.cpp) -target_link_libraries(${test_prefix}graph_db_accessor memgraph_lib) +target_link_libraries(${test_prefix}graph_db_accessor memgraph_lib kvstore_dummy_lib) add_unit_test(graph_db_accessor_index_api.cpp) -target_link_libraries(${test_prefix}graph_db_accessor_index_api memgraph_lib) +target_link_libraries(${test_prefix}graph_db_accessor_index_api memgraph_lib kvstore_dummy_lib) add_unit_test(graph_db.cpp) -target_link_libraries(${test_prefix}graph_db memgraph_lib) +target_link_libraries(${test_prefix}graph_db memgraph_lib kvstore_dummy_lib) add_unit_test(interpreter.cpp) -target_link_libraries(${test_prefix}interpreter memgraph_lib) +target_link_libraries(${test_prefix}interpreter memgraph_lib kvstore_dummy_lib) + +add_unit_test(kvstore.cpp) +target_link_libraries(${test_prefix}kvstore gtest gtest_main memgraph_lib kvstore_lib) add_unit_test(metrics.cpp) -target_link_libraries(${test_prefix}metrics memgraph_lib) +target_link_libraries(${test_prefix}metrics memgraph_lib kvstore_dummy_lib) add_unit_test(mvcc.cpp) -target_link_libraries(${test_prefix}mvcc memgraph_lib) +target_link_libraries(${test_prefix}mvcc memgraph_lib kvstore_dummy_lib) add_unit_test(mvcc_find.cpp) -target_link_libraries(${test_prefix}mvcc_find memgraph_lib) +target_link_libraries(${test_prefix}mvcc_find memgraph_lib kvstore_dummy_lib) add_unit_test(mvcc_gc.cpp) -target_link_libraries(${test_prefix}mvcc_gc memgraph_lib) +target_link_libraries(${test_prefix}mvcc_gc memgraph_lib kvstore_dummy_lib) add_unit_test(mvcc_one_transaction.cpp) -target_link_libraries(${test_prefix}mvcc_one_transaction memgraph_lib) +target_link_libraries(${test_prefix}mvcc_one_transaction memgraph_lib kvstore_dummy_lib) add_unit_test(mvcc_parallel_update.cpp) -target_link_libraries(${test_prefix}mvcc_parallel_update memgraph_lib) +target_link_libraries(${test_prefix}mvcc_parallel_update memgraph_lib kvstore_dummy_lib) add_unit_test(network_timeouts.cpp) -target_link_libraries(${test_prefix}network_timeouts memgraph_lib) +target_link_libraries(${test_prefix}network_timeouts memgraph_lib kvstore_dummy_lib) + +add_unit_test(pod_buffer.cpp) +target_link_libraries(${test_prefix}pod_buffer memgraph_lib kvstore_dummy_lib) add_unit_test(property_value_store.cpp) -target_link_libraries(${test_prefix}property_value_store memgraph_lib) +target_link_libraries(${test_prefix}property_value_store memgraph_lib kvstore_lib) add_unit_test(query_cost_estimator.cpp) -target_link_libraries(${test_prefix}query_cost_estimator memgraph_lib) +target_link_libraries(${test_prefix}query_cost_estimator memgraph_lib kvstore_dummy_lib) add_unit_test(query_expression_evaluator.cpp) -target_link_libraries(${test_prefix}query_expression_evaluator memgraph_lib) +target_link_libraries(${test_prefix}query_expression_evaluator memgraph_lib kvstore_dummy_lib) add_unit_test(query_plan_accumulate_aggregate.cpp) -target_link_libraries(${test_prefix}query_plan_accumulate_aggregate memgraph_lib) +target_link_libraries(${test_prefix}query_plan_accumulate_aggregate memgraph_lib kvstore_dummy_lib) add_unit_test(query_plan_bag_semantics.cpp) -target_link_libraries(${test_prefix}query_plan_bag_semantics memgraph_lib) +target_link_libraries(${test_prefix}query_plan_bag_semantics memgraph_lib kvstore_dummy_lib) add_unit_test(query_plan_create_set_remove_delete.cpp) -target_link_libraries(${test_prefix}query_plan_create_set_remove_delete memgraph_lib) +target_link_libraries(${test_prefix}query_plan_create_set_remove_delete memgraph_lib kvstore_dummy_lib) add_unit_test(query_plan_edge_cases.cpp) -target_link_libraries(${test_prefix}query_plan_edge_cases memgraph_lib) +target_link_libraries(${test_prefix}query_plan_edge_cases memgraph_lib kvstore_dummy_lib) add_unit_test(query_plan_match_filter_return.cpp) -target_link_libraries(${test_prefix}query_plan_match_filter_return memgraph_lib) +target_link_libraries(${test_prefix}query_plan_match_filter_return memgraph_lib kvstore_dummy_lib) add_unit_test(query_planner.cpp) -target_link_libraries(${test_prefix}query_planner memgraph_lib) +target_link_libraries(${test_prefix}query_planner memgraph_lib kvstore_dummy_lib) add_unit_test(query_semantic.cpp) -target_link_libraries(${test_prefix}query_semantic memgraph_lib) +target_link_libraries(${test_prefix}query_semantic memgraph_lib kvstore_dummy_lib) add_unit_test(query_variable_start_planner.cpp) -target_link_libraries(${test_prefix}query_variable_start_planner memgraph_lib) +target_link_libraries(${test_prefix}query_variable_start_planner memgraph_lib kvstore_dummy_lib) add_unit_test(queue.cpp) -target_link_libraries(${test_prefix}queue memgraph_lib) +target_link_libraries(${test_prefix}queue memgraph_lib kvstore_dummy_lib) add_unit_test(raft.cpp) -target_link_libraries(${test_prefix}raft memgraph_lib) +target_link_libraries(${test_prefix}raft memgraph_lib kvstore_dummy_lib) add_unit_test(raft_storage.cpp) -target_link_libraries(${test_prefix}raft_storage memgraph_lib) +target_link_libraries(${test_prefix}raft_storage memgraph_lib kvstore_dummy_lib) add_unit_test(record_edge_vertex_accessor.cpp) -target_link_libraries(${test_prefix}record_edge_vertex_accessor memgraph_lib) +target_link_libraries(${test_prefix}record_edge_vertex_accessor memgraph_lib kvstore_dummy_lib) add_unit_test(rpc.cpp) -target_link_libraries(${test_prefix}rpc memgraph_lib) +target_link_libraries(${test_prefix}rpc memgraph_lib kvstore_dummy_lib) add_unit_test(rpc_worker_clients.cpp) -target_link_libraries(${test_prefix}rpc_worker_clients memgraph_lib) +target_link_libraries(${test_prefix}rpc_worker_clients memgraph_lib kvstore_dummy_lib) add_unit_test(serialization.cpp) -target_link_libraries(${test_prefix}serialization memgraph_lib) +target_link_libraries(${test_prefix}serialization memgraph_lib kvstore_dummy_lib) add_unit_test(skiplist_access.cpp) -target_link_libraries(${test_prefix}skiplist_access memgraph_lib) +target_link_libraries(${test_prefix}skiplist_access memgraph_lib kvstore_dummy_lib) add_unit_test(skiplist_gc.cpp) -target_link_libraries(${test_prefix}skiplist_gc memgraph_lib) +target_link_libraries(${test_prefix}skiplist_gc memgraph_lib kvstore_dummy_lib) add_unit_test(skiplist_position_and_count.cpp) -target_link_libraries(${test_prefix}skiplist_position_and_count memgraph_lib) +target_link_libraries(${test_prefix}skiplist_position_and_count memgraph_lib kvstore_dummy_lib) add_unit_test(skiplist_reverse_iteration.cpp) -target_link_libraries(${test_prefix}skiplist_reverse_iteration memgraph_lib) +target_link_libraries(${test_prefix}skiplist_reverse_iteration memgraph_lib kvstore_dummy_lib) add_unit_test(skiplist_suffix.cpp) -target_link_libraries(${test_prefix}skiplist_suffix memgraph_lib) +target_link_libraries(${test_prefix}skiplist_suffix memgraph_lib kvstore_dummy_lib) add_unit_test(state_delta.cpp) -target_link_libraries(${test_prefix}state_delta memgraph_lib) +target_link_libraries(${test_prefix}state_delta memgraph_lib kvstore_dummy_lib) add_unit_test(static_bitset.cpp) -target_link_libraries(${test_prefix}static_bitset memgraph_lib) +target_link_libraries(${test_prefix}static_bitset memgraph_lib kvstore_dummy_lib) add_unit_test(storage_address.cpp) -target_link_libraries(${test_prefix}storage_address memgraph_lib) +target_link_libraries(${test_prefix}storage_address memgraph_lib kvstore_dummy_lib) add_unit_test(stripped.cpp) -target_link_libraries(${test_prefix}stripped memgraph_lib) +target_link_libraries(${test_prefix}stripped memgraph_lib kvstore_dummy_lib) add_unit_test(transaction_engine_distributed.cpp) -target_link_libraries(${test_prefix}transaction_engine_distributed memgraph_lib) +target_link_libraries(${test_prefix}transaction_engine_distributed memgraph_lib kvstore_dummy_lib) add_unit_test(transaction_engine_single_node.cpp) -target_link_libraries(${test_prefix}transaction_engine_single_node memgraph_lib) +target_link_libraries(${test_prefix}transaction_engine_single_node memgraph_lib kvstore_dummy_lib) add_unit_test(typed_value.cpp) -target_link_libraries(${test_prefix}typed_value memgraph_lib) +target_link_libraries(${test_prefix}typed_value memgraph_lib kvstore_dummy_lib) # Test data structures diff --git a/tests/unit/database_label_property_index.cpp b/tests/unit/database_label_property_index.cpp index 544932989..c0d022d56 100644 --- a/tests/unit/database_label_property_index.cpp +++ b/tests/unit/database_label_property_index.cpp @@ -30,7 +30,7 @@ class LabelPropertyIndexComplexTest : public ::testing::Test { vertex = vlist->find(*t); ASSERT_NE(vertex, nullptr); vertex->labels_.push_back(label); - vertex->properties_.set(property, 0); + vertex->properties_.set(property, PropertyValue(0)); EXPECT_EQ(index.Count(*key), 0); } diff --git a/tests/unit/distributed_common.hpp b/tests/unit/distributed_common.hpp index a5fb688f1..63ee14166 100644 --- a/tests/unit/distributed_common.hpp +++ b/tests/unit/distributed_common.hpp @@ -2,6 +2,7 @@ #include <memory> #include <thread> +#include <gflags/gflags.h> #include <gtest/gtest.h> #include "database/graph_db.hpp" @@ -11,6 +12,8 @@ #include "storage/address_types.hpp" #include "transactions/engine_master.hpp" +DECLARE_string(durability_directory); + namespace fs = std::experimental::filesystem; class WorkerInThread { @@ -62,6 +65,9 @@ class DistributedGraphDbTest : public ::testing::Test { return config; }; + // Flag needs to be updated due to props on disk storage. + FLAGS_durability_directory = tmp_dir_; + for (int i = 0; i < kWorkerCount; ++i) { workers_.emplace_back(std::make_unique<WorkerInThread>( modify_config(worker_config(i + 1)))); diff --git a/tests/unit/durability.cpp b/tests/unit/durability.cpp index 2d99bf957..02b0fe88a 100644 --- a/tests/unit/durability.cpp +++ b/tests/unit/durability.cpp @@ -25,6 +25,8 @@ DECLARE_int32(wal_flush_interval_millis); DECLARE_int32(wal_rotate_deltas_count); +DECLARE_string(durability_directory); + namespace fs = std::experimental::filesystem; // Helper class for performing random CRUD ops on a database. @@ -309,6 +311,7 @@ class Durability : public ::testing::Test { snapshot_dir_ = durability_dir_ / durability::kSnapshotDir; wal_dir_ = durability_dir_ / durability::kWalDir; FLAGS_wal_rotate_deltas_count = 1000; + FLAGS_durability_directory = "MG_test_unit_durability"; CleanDurability(); } @@ -818,9 +821,8 @@ TEST_F(Durability, SequentialRecovery) { return threads; }; - auto make_updates = [&run_updates, this](database::GraphDb &db, - bool snapshot_during, - bool snapshot_after) { + auto make_updates = [&run_updates, this]( + database::GraphDb &db, bool snapshot_during, bool snapshot_after) { std::atomic<bool> keep_running{true}; auto update_theads = run_updates(db, keep_running); std::this_thread::sleep_for(25ms); diff --git a/tests/unit/kvstore.cpp b/tests/unit/kvstore.cpp new file mode 100644 index 000000000..418af312f --- /dev/null +++ b/tests/unit/kvstore.cpp @@ -0,0 +1,198 @@ +#include <glog/logging.h> +#include <gtest/gtest.h> + +#include "durability/paths.hpp" +#include "storage/kvstore.hpp" +#include "utils/file.hpp" + +namespace fs = std::experimental::filesystem; + +class KVStore : public ::testing::Test { + protected: + virtual void SetUp() { utils::EnsureDir(test_folder_); } + + virtual void TearDown() { fs::remove_all(test_folder_); } + + fs::path test_folder_{fs::path("kvstore_test")}; +}; + +TEST_F(KVStore, PutGet) { + storage::KVStore kvstore(test_folder_ / "PutGet"); + ASSERT_TRUE(kvstore.Put("key", "value")); + ASSERT_EQ(kvstore.Get("key").value(), "value"); +} + +TEST_F(KVStore, PutGetDeleteGet) { + storage::KVStore kvstore(test_folder_ / "PutGetDeleteGet"); + ASSERT_TRUE(kvstore.Put("key", "value")); + ASSERT_EQ(kvstore.Get("key").value(), "value"); + ASSERT_TRUE(kvstore.Delete("key")); + ASSERT_FALSE(static_cast<bool>(kvstore.Get("key"))); +} + +TEST_F(KVStore, Durability) { + { + storage::KVStore kvstore(test_folder_ / "Durability"); + ASSERT_TRUE(kvstore.Put("key", "value")); + } + { + storage::KVStore kvstore(test_folder_ / "Durability"); + ASSERT_EQ(kvstore.Get("key").value(), "value"); + } +} + +TEST_F(KVStore, Size) { + storage::KVStore kvstore(test_folder_ / "Size"); + + ASSERT_TRUE(kvstore.Put("prefix_1", "jedan")); + ASSERT_TRUE(kvstore.Put("prefix_2", "dva")); + ASSERT_TRUE(kvstore.Put("prefix_3", "tri")); + ASSERT_TRUE(kvstore.Put("prefix_4", "cetiri")); + ASSERT_TRUE(kvstore.Put("prefix_5", "pet")); + + EXPECT_EQ(kvstore.Size("a"), 0); + EXPECT_EQ(kvstore.Size(), 5); + EXPECT_EQ(kvstore.Size("prefix_"), 5); + EXPECT_EQ(kvstore.Size("prefix_1"), 1); + + ASSERT_TRUE(kvstore.Put("predmetak_1", "jedan")); + ASSERT_TRUE(kvstore.Put("predmetak_2", "dva")); + ASSERT_TRUE(kvstore.Put("predmetak_3", "tri")); + ASSERT_TRUE(kvstore.Put("predmetak_4", "cetiri")); + + EXPECT_EQ(kvstore.Size("a"), 0); + EXPECT_EQ(kvstore.Size(), 9); + EXPECT_EQ(kvstore.Size("prefix_"), 5); + EXPECT_EQ(kvstore.Size("predmetak_"), 4); + EXPECT_EQ(kvstore.Size("p"), 9); + EXPECT_EQ(kvstore.Size("pre"), 9); + EXPECT_EQ(kvstore.Size("pred"), 4); + EXPECT_EQ(kvstore.Size("pref"), 5); + EXPECT_EQ(kvstore.Size("prex"), 0); +} + +TEST_F(KVStore, DeletePrefix) { + storage::KVStore kvstore(test_folder_ / "DeletePrefix"); + + ASSERT_TRUE(kvstore.Put("prefix_1", "jedan")); + ASSERT_TRUE(kvstore.Put("prefix_2", "dva")); + ASSERT_TRUE(kvstore.Put("prefix_3", "tri")); + ASSERT_TRUE(kvstore.Put("prefix_4", "cetiri")); + ASSERT_TRUE(kvstore.Put("prefix_5", "pet")); + + EXPECT_EQ(kvstore.Size(), 5); + + ASSERT_TRUE(kvstore.DeletePrefix("prefix_5")); + + EXPECT_EQ(kvstore.Size(), 4); + EXPECT_EQ(kvstore.Size("prefix_"), 4); + EXPECT_EQ(kvstore.Size("prefix_1"), 1); + + ASSERT_TRUE(kvstore.Put("predmetak_1", "jedan")); + ASSERT_TRUE(kvstore.Put("predmetak_2", "dva")); + ASSERT_TRUE(kvstore.Put("predmetak_3", "tri")); + ASSERT_TRUE(kvstore.Put("predmetak_4", "cetiri")); + + EXPECT_EQ(kvstore.Size(), 8); + + ASSERT_TRUE(kvstore.DeletePrefix("predmetak_1")); + EXPECT_EQ(kvstore.Size(), 7); + + ASSERT_TRUE(kvstore.DeletePrefix("prefix_")); + EXPECT_EQ(kvstore.Size(), 3); + + ASSERT_TRUE(kvstore.DeletePrefix("predmetak_")); + EXPECT_EQ(kvstore.Size(), 0); + + ASSERT_TRUE(kvstore.DeletePrefix("whatever")); + EXPECT_EQ(kvstore.Size(), 0); + EXPECT_EQ(kvstore.Size("any_prefix"), 0); +} + +TEST_F(KVStore, Iterator) { + storage::KVStore kvstore(test_folder_ / "Iterator"); + + for (int i = 1; i <= 4; ++i) + ASSERT_TRUE( + kvstore.Put("key" + std::to_string(i), "value" + std::to_string(i))); + + auto it = kvstore.begin(); + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "key1"); + EXPECT_EQ((*it).second, "value1"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ((*it).first, "key2"); + EXPECT_EQ(it->second, "value2"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "key3"); + EXPECT_EQ((*it).second, "value3"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ((*it).first, "key4"); + EXPECT_EQ(it->second, "value4"); + + ++it; + ASSERT_FALSE(it.IsValid()); +} + +TEST_F(KVStore, IteratorPrefix) { + storage::KVStore kvstore(test_folder_ / "Iterator"); + + ASSERT_TRUE(kvstore.Put("a_1", "value1")); + ASSERT_TRUE(kvstore.Put("a_2", "value2")); + + ASSERT_TRUE(kvstore.Put("aa_1", "value1")); + ASSERT_TRUE(kvstore.Put("aa_2", "value2")); + + ASSERT_TRUE(kvstore.Put("b_1", "value1")); + ASSERT_TRUE(kvstore.Put("b_2", "value2")); + + auto it = kvstore.begin("a"); + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "a_1"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "a_2"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "aa_1"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "aa_2"); + + ++it; + ASSERT_FALSE(it.IsValid()); + + it = kvstore.begin("aa_"); + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "aa_1"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "aa_2"); + + ++it; + ASSERT_FALSE(it.IsValid()); + + it = kvstore.begin("b_"); + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "b_1"); + + ++it; + ASSERT_TRUE(it.IsValid()); + EXPECT_EQ(it->first, "b_2"); + + ++it; + ASSERT_FALSE(it.IsValid()); + + it = kvstore.begin("unexisting_prefix"); + ASSERT_FALSE(it.IsValid()); +} diff --git a/tests/unit/pod_buffer.cpp b/tests/unit/pod_buffer.cpp new file mode 100644 index 000000000..77e07bfc6 --- /dev/null +++ b/tests/unit/pod_buffer.cpp @@ -0,0 +1,60 @@ +#include <glog/logging.h> +#include <gtest/gtest.h> + +#include "storage/pod_buffer.hpp" + +class PODBufferTest : public ::testing::Test { + protected: + storage::PODBuffer buffer_; + + void SetUp() override { buffer_ = storage::PODBuffer(""); } + + void Write(const uint8_t *data, size_t len) { buffer_.Write(data, len); } + + bool Read(uint8_t *data, size_t len) { return buffer_.Read(data, len); } +}; + +TEST_F(PODBufferTest, ReadEmpty) { + uint8_t data[10]; + ASSERT_TRUE(Read(data, 0)); + for (int i = 1; i <= 5; ++i) ASSERT_FALSE(Read(data, i)); +} + +TEST_F(PODBufferTest, ReadNonEmpty) { + uint8_t input_data[10]; + uint8_t output_data[10]; + + for (int i = 0; i < 10; ++i) input_data[i] = i; + + Write(input_data, 10); + ASSERT_TRUE(Read(output_data, 10)); + + for (int i = 0; i < 10; ++i) ASSERT_EQ(output_data[i], i); + + ASSERT_FALSE(Read(output_data, 1)); +} + +TEST_F(PODBufferTest, WriteRead) { + uint8_t input_data[10]; + uint8_t output_data[10]; + + for (int i = 0; i < 10; ++i) input_data[i] = i; + + Write(input_data, 10); + ASSERT_TRUE(Read(output_data, 5)); + + for (int i = 0; i < 5; ++i) ASSERT_EQ(output_data[i], i); + + ASSERT_TRUE(Read(output_data, 5)); + + for (int i = 0; i < 5; ++i) ASSERT_EQ(output_data[i], i + 5); + + ASSERT_FALSE(Read(output_data, 1)); + + Write(input_data + 5, 5); + ASSERT_TRUE(Read(output_data, 5)); + + for (int i = 0; i < 5; ++i) ASSERT_EQ(output_data[i], i + 5); + + ASSERT_FALSE(Read(output_data, 1)); +} diff --git a/tests/unit/property_value_store.cpp b/tests/unit/property_value_store.cpp index 3aa5b0cb6..98e6da526 100644 --- a/tests/unit/property_value_store.cpp +++ b/tests/unit/property_value_store.cpp @@ -1,5 +1,6 @@ #include <vector> +#include "gflags/gflags.h" #include "gtest/gtest.h" #include "storage/property_value.hpp" @@ -8,10 +9,17 @@ using std::string; using Location = storage::Location; +DECLARE_string(properties_on_disk); + class PropertyValueStoreTest : public ::testing::Test { protected: PropertyValueStore props_; + void SetUp() override { + // we need this to test the copy constructor + FLAGS_properties_on_disk = "not empty"; + } + void Set(int key, Location location, PropertyValue value) { props_.set(storage::Property(key, location), value); } @@ -24,12 +32,15 @@ class PropertyValueStoreTest : public ::testing::Test { return props_.erase(storage::Property(key, location)); } + auto Begin() { return props_.begin(); } + + auto End() { return props_.end(); } + void TearDown() override { props_.clear(); } }; -TEST_F(PropertyValueStoreTest, At) { +TEST_F(PropertyValueStoreTest, AtMemory) { std::string some_string = "something"; - std::string other_string = "something completely different"; EXPECT_EQ(PropertyValue(At(0, Location::Memory)).type(), PropertyValue::Type::Null); @@ -38,10 +49,17 @@ TEST_F(PropertyValueStoreTest, At) { some_string); Set(120, Location::Memory, 42); EXPECT_EQ(PropertyValue(At(120, Location::Memory)).Value<int64_t>(), 42); +} - Set(100, Location::Disk, other_string); - EXPECT_EQ(PropertyValue(At(100, Location::Disk)).Value<string>(), - other_string); +TEST_F(PropertyValueStoreTest, AtDisk) { + std::string some_string = "something"; + + EXPECT_EQ(PropertyValue(At(0, Location::Disk)).type(), + PropertyValue::Type::Null); + Set(0, Location::Disk, some_string); + EXPECT_EQ(PropertyValue(At(0, Location::Disk)).Value<string>(), some_string); + Set(120, Location::Disk, 42); + EXPECT_EQ(PropertyValue(At(120, Location::Disk)).Value<int64_t>(), 42); } TEST_F(PropertyValueStoreTest, AtNull) { @@ -69,7 +87,7 @@ TEST_F(PropertyValueStoreTest, SetNull) { EXPECT_EQ(0, props_.size()); } -TEST_F(PropertyValueStoreTest, Remove) { +TEST_F(PropertyValueStoreTest, RemoveMemory) { // set some props Set(11, Location::Memory, "a"); Set(30, Location::Memory, "b"); @@ -85,48 +103,59 @@ TEST_F(PropertyValueStoreTest, Remove) { EXPECT_EQ(props_.size(), 0); EXPECT_EQ(At(30, Location::Memory).type(), PropertyValue::Type::Null); - EXPECT_EQ(Erase(1000, Location::Memory), 0); - - props_.clear(); - - Set(110, Location::Disk, "a"); - EXPECT_NE(At(110, Location::Disk).type(), PropertyValue::Type::Null); - EXPECT_EQ(props_.size(), 1); - - Erase(110, Location::Disk); - EXPECT_EQ(props_.size(), 0); - EXPECT_EQ(At(110, Location::Disk).type(), PropertyValue::Type::Null); - EXPECT_EQ(Erase(1000, Location::Disk), 0); + EXPECT_EQ(Erase(1000, Location::Memory), 1); } -TEST_F(PropertyValueStoreTest, Clear) { +TEST_F(PropertyValueStoreTest, RemoveDisk) { + // set some props + Set(11, Location::Disk, "a"); + Set(30, Location::Disk, "b"); + EXPECT_NE(At(11, Location::Disk).type(), PropertyValue::Type::Null); + EXPECT_NE(At(30, Location::Disk).type(), PropertyValue::Type::Null); + EXPECT_EQ(props_.size(), 2); + + Erase(11, Location::Disk); + EXPECT_EQ(props_.size(), 1); + EXPECT_EQ(At(11, Location::Disk).type(), PropertyValue::Type::Null); + + EXPECT_EQ(Erase(30, Location::Disk), 1); + EXPECT_EQ(props_.size(), 0); + EXPECT_EQ(At(30, Location::Disk).type(), PropertyValue::Type::Null); + + EXPECT_EQ(Erase(1000, Location::Disk), 1); +} + +TEST_F(PropertyValueStoreTest, ClearMemory) { EXPECT_EQ(props_.size(), 0); Set(11, Location::Memory, "a"); Set(30, Location::Memory, "b"); EXPECT_EQ(props_.size(), 2); - props_.clear(); - EXPECT_EQ(props_.size(), 0); - - Set(11, Location::Disk, "a"); - EXPECT_EQ(props_.size(), 1); - props_.clear(); - EXPECT_EQ(props_.size(), 0); } -TEST_F(PropertyValueStoreTest, Replace) { +TEST_F(PropertyValueStoreTest, ClearDisk) { + EXPECT_EQ(props_.size(), 0); + Set(11, Location::Disk, "a"); + Set(30, Location::Disk, "b"); + EXPECT_EQ(props_.size(), 2); +} + +TEST_F(PropertyValueStoreTest, ReplaceMemory) { Set(10, Location::Memory, 42); EXPECT_EQ(At(10, Location::Memory).Value<int64_t>(), 42); Set(10, Location::Memory, 0.25f); EXPECT_EQ(At(10, Location::Memory).type(), PropertyValue::Type::Double); EXPECT_FLOAT_EQ(At(10, Location::Memory).Value<double>(), 0.25); - - Set(100, Location::Disk, "some text"); - EXPECT_EQ(At(100, Location::Disk).Value<string>(), "some text"); - Set(100, Location::Disk, "some other text"); - EXPECT_EQ(At(100, Location::Disk).Value<string>(), "some other text"); } -TEST_F(PropertyValueStoreTest, Size) { +TEST_F(PropertyValueStoreTest, ReplaceDisk) { + Set(10, Location::Disk, 42); + EXPECT_EQ(At(10, Location::Disk).Value<int64_t>(), 42); + Set(10, Location::Disk, 0.25f); + EXPECT_EQ(At(10, Location::Disk).type(), PropertyValue::Type::Double); + EXPECT_FLOAT_EQ(At(10, Location::Disk).Value<double>(), 0.25); +} + +TEST_F(PropertyValueStoreTest, SizeMemory) { EXPECT_EQ(props_.size(), 0); Set(0, Location::Memory, "something"); @@ -145,14 +174,49 @@ TEST_F(PropertyValueStoreTest, Size) { EXPECT_EQ(props_.size(), 101); Erase(1, Location::Memory); EXPECT_EQ(props_.size(), 100); +} - Set(101, Location::Disk, "dalmatians"); +TEST_F(PropertyValueStoreTest, SizeDisk) { + EXPECT_EQ(props_.size(), 0); + + Set(0, Location::Disk, "something"); + EXPECT_EQ(props_.size(), 1); + Set(0, Location::Disk, true); + EXPECT_EQ(props_.size(), 1); + Set(1, Location::Disk, true); + EXPECT_EQ(props_.size(), 2); + + for (int i = 0; i < 100; ++i) Set(i + 20, Location::Disk, true); + EXPECT_EQ(props_.size(), 102); + + Erase(0, Location::Disk); EXPECT_EQ(props_.size(), 101); - Erase(101, Location::Disk); + Erase(0, Location::Disk); + EXPECT_EQ(props_.size(), 101); + Erase(1, Location::Disk); EXPECT_EQ(props_.size(), 100); } -TEST_F(PropertyValueStoreTest, InsertRetrieveList) { +TEST_F(PropertyValueStoreTest, Size) { + EXPECT_EQ(props_.size(), 0); + + for (int i = 0; i < 100; ++i) Set(i, Location::Disk, true); + EXPECT_EQ(props_.size(), 100); + + for (int i = 0; i < 200; ++i) Set(i + 100, Location::Memory, true); + EXPECT_EQ(props_.size(), 300); + + Erase(0, Location::Disk); + EXPECT_EQ(props_.size(), 299); + Erase(99, Location::Disk); + EXPECT_EQ(props_.size(), 298); + Erase(100, Location::Memory); + EXPECT_EQ(props_.size(), 297); + Erase(299, Location::Memory); + EXPECT_EQ(props_.size(), 296); +} + +TEST_F(PropertyValueStoreTest, InsertRetrieveListMemory) { Set(0, Location::Memory, std::vector<PropertyValue>{1, true, 2.5, "something", PropertyValue::Null}); auto p = At(0, Location::Memory); @@ -171,6 +235,25 @@ TEST_F(PropertyValueStoreTest, InsertRetrieveList) { EXPECT_EQ(l[4].type(), PropertyValue::Type::Null); } +TEST_F(PropertyValueStoreTest, InsertRetrieveListDisk) { + Set(0, Location::Disk, std::vector<PropertyValue>{1, true, 2.5, "something", + PropertyValue::Null}); + auto p = At(0, Location::Disk); + + EXPECT_EQ(p.type(), PropertyValue::Type::List); + auto l = p.Value<std::vector<PropertyValue>>(); + EXPECT_EQ(l.size(), 5); + EXPECT_EQ(l[0].type(), PropertyValue::Type::Int); + EXPECT_EQ(l[0].Value<int64_t>(), 1); + EXPECT_EQ(l[1].type(), PropertyValue::Type::Bool); + EXPECT_EQ(l[1].Value<bool>(), true); + EXPECT_EQ(l[2].type(), PropertyValue::Type::Double); + EXPECT_EQ(l[2].Value<double>(), 2.5); + EXPECT_EQ(l[3].type(), PropertyValue::Type::String); + EXPECT_EQ(l[3].Value<std::string>(), "something"); + EXPECT_EQ(l[4].type(), PropertyValue::Type::Null); +} + TEST_F(PropertyValueStoreTest, InsertRetrieveMap) { Set(0, Location::Memory, std::map<std::string, PropertyValue>{ {"a", 1}, {"b", true}, {"c", "something"}}); @@ -189,3 +272,89 @@ TEST_F(PropertyValueStoreTest, InsertRetrieveMap) { EXPECT_EQ(get("c").type(), PropertyValue::Type::String); EXPECT_EQ(get("c").Value<std::string>(), "something"); } + +TEST_F(PropertyValueStoreTest, InsertRetrieveMapDisk) { + Set(0, Location::Disk, std::map<std::string, PropertyValue>{ + {"a", 1}, {"b", true}, {"c", "something"}}); + + auto p = At(0, Location::Disk); + EXPECT_EQ(p.type(), PropertyValue::Type::Map); + auto m = p.Value<std::map<std::string, PropertyValue>>(); + EXPECT_EQ(m.size(), 3); + auto get = [&m](const std::string &prop_name) { + return m.find(prop_name)->second; + }; + EXPECT_EQ(get("a").type(), PropertyValue::Type::Int); + EXPECT_EQ(get("a").Value<int64_t>(), 1); + EXPECT_EQ(get("b").type(), PropertyValue::Type::Bool); + EXPECT_EQ(get("b").Value<bool>(), true); + EXPECT_EQ(get("c").type(), PropertyValue::Type::String); + EXPECT_EQ(get("c").Value<std::string>(), "something"); +} + +TEST_F(PropertyValueStoreTest, Iterator) { + Set(0, Location::Memory, "a"); + Set(1, Location::Memory, 1); + Set(2, Location::Disk, "b"); + Set(3, Location::Disk, 2); + + auto it = Begin(); + ASSERT_TRUE(it != End()); + EXPECT_EQ(it->first.Id(), 0); + EXPECT_EQ((*it).second.Value<std::string>(), "a"); + + ++it; + ASSERT_TRUE(it != End()); + EXPECT_EQ((*it).first.Id(), 1); + EXPECT_EQ(it->second.Value<int64_t>(), 1); + + ++it; + ASSERT_TRUE(it != End()); + EXPECT_EQ(it->first.Id(), 2); + EXPECT_EQ((*it).second.Value<std::string>(), "b"); + + ++it; + ASSERT_TRUE(it != End()); + EXPECT_EQ((*it).first.Id(), 3); + EXPECT_EQ(it->second.Value<int64_t>(), 2); + + ++it; + ASSERT_TRUE(it == End()); +} + +TEST_F(PropertyValueStoreTest, CopyConstructor) { + PropertyValueStore props; + for (int i = 1; i <= 3; ++i) + props.set(storage::Property(i, Location::Memory), + "mem_" + std::to_string(i)); + for (int i = 4; i <= 5; ++i) + props.set(storage::Property(i, Location::Disk), + "disk_" + std::to_string(i)); + + PropertyValueStore new_props = props; + for (int i = 1; i <= 3; ++i) + EXPECT_EQ( + new_props.at(storage::Property(i, Location::Memory)).Value<string>(), + "mem_" + std::to_string(i)); + for (int i = 4; i <= 5; ++i) + EXPECT_EQ( + new_props.at(storage::Property(i, Location::Disk)).Value<string>(), + "disk_" + std::to_string(i)); + + props.set(storage::Property(1, Location::Memory), "mem_1_update"); + EXPECT_EQ( + new_props.at(storage::Property(1, Location::Memory)).Value<string>(), + "mem_1"); + + new_props.set(storage::Property(2, Location::Memory), "mem_2_update"); + EXPECT_EQ(props.at(storage::Property(2, Location::Memory)).Value<string>(), + "mem_2"); + + props.set(storage::Property(4, Location::Disk), "disk_4_update"); + EXPECT_EQ(new_props.at(storage::Property(4, Location::Disk)).Value<string>(), + "disk_4"); + + new_props.set(storage::Property(5, Location::Disk), "disk_5_update"); + EXPECT_EQ(props.at(storage::Property(5, Location::Disk)).Value<string>(), + "disk_5"); +} diff --git a/tools/src/CMakeLists.txt b/tools/src/CMakeLists.txt index 6de12f1fb..8087c990d 100644 --- a/tools/src/CMakeLists.txt +++ b/tools/src/CMakeLists.txt @@ -1,10 +1,10 @@ # CSV Import Tool Target add_executable(mg_import_csv mg_import_csv/main.cpp) -target_link_libraries(mg_import_csv memgraph_lib) +target_link_libraries(mg_import_csv memgraph_lib kvstore_dummy_lib) # StatsD Target add_executable(mg_statsd mg_statsd/main.cpp) -target_link_libraries(mg_statsd memgraph_lib) +target_link_libraries(mg_statsd memgraph_lib kvstore_dummy_lib) # Strip the executable in release build. string(TOLOWER ${CMAKE_BUILD_TYPE} lower_build_type) diff --git a/tools/src/mg_import_csv/main.cpp b/tools/src/mg_import_csv/main.cpp index f197c0fae..d6776a382 100644 --- a/tools/src/mg_import_csv/main.cpp +++ b/tools/src/mg_import_csv/main.cpp @@ -446,7 +446,7 @@ static const char *usage = "Create a Memgraph recovery snapshot file from CSV.\n"; // Used only to get the value from memgraph's configuration files. -DEFINE_HIDDEN_string(durability_directory, "", "Durability directory"); +DECLARE_string(durability_directory); std::string GetOutputPath() { // If we have the 'out' flag, use that. diff --git a/tools/tests/CMakeLists.txt b/tools/tests/CMakeLists.txt index 4c6302039..46c515f78 100644 --- a/tools/tests/CMakeLists.txt +++ b/tools/tests/CMakeLists.txt @@ -1,10 +1,10 @@ include_directories(SYSTEM ${GTEST_INCLUDE_DIR}) add_executable(mg_recovery_check mg_recovery_check.cpp) -target_link_libraries(mg_recovery_check memgraph_lib gtest gtest_main) +target_link_libraries(mg_recovery_check memgraph_lib gtest gtest_main kvstore_dummy_lib) add_executable(mg_statsd_client statsd/mg_statsd_client.cpp) -target_link_libraries(mg_statsd_client memgraph_lib) +target_link_libraries(mg_statsd_client memgraph_lib kvstore_dummy_lib) # Copy CSV data to CMake build dir configure_file(csv/comment_nodes.csv csv/comment_nodes.csv COPYONLY)