tursom/memgraph
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5db39ac501
memgraph/docs/feature_specs/kafka/extractor.md
Matija Santl 64f189cc8a Kafka stream import feature spec 
Summary:
First version of the feature spec for importing streams of data using
kafka in memgraph.

Reviewers: buda, teon.banek, dgleich, ipaljak

Reviewed By: buda

Subscribers: lion, mculinovic

Differential Revision: https://phabricator.memgraph.io/D1415
2018-06-20 10:48:53 +02:00

882 B
Raw Blame History

Kafka - data extractor

The data extractor is responsible for loading data from Kafka. In order to do so, it needs to know the URI of the Kafka leader broker. Once the extractor connects to Kafka, it starts importing data.

Data extractor depends on cppkafka which makes message consumption just a few API calls, as seen here.

There are also other metadata that can be passed to data extractor that are defined with our extension of openCypher.

A full list of configurable metadata can be found here.

Memgraph supports customizing the following:

  • metadata.broker.list which is a required parameter, set by KAFKA 'URI'
  • queue.buffering.max.ms set by BATCH INTERVAL