Crate etl_kafka 

Kafka source for the etl-rs framework.

Built on rdkafka with a single consumer per process: partitions are split into per-partition queues (split_partition_queue) and fanned across pipeline threads as SourceLanes, keeping payload borrows local to the polling thread with zero copies. Offsets are stored when checkpoint watermarks advance and committed on an interval; rebalances and shutdown share the framework’s drain choreography, with completion deferred until draining and a synchronous commit have finished (see KafkaSource docs).

One topic per pipeline: the framework’s PartitionId is the Kafka partition number. Kafka tombstones (null payloads) surface as empty payload slices.

Configuration is read from the pipeline’s opaque source: { kafka: ... } section — see KafkaSourceConfig for the schema and the raw rdkafka: passthrough (framework-owned properties are validated and rejected with explanations).

source:
  kafka:
    brokers: ${KAFKA_BROKERS:-localhost:9092}
    topic: orders
    group_id: orders-etl
    commit_interval: 5s
    rdkafka:
      fetch.message.max.bytes: "1048576"

This crate deliberately re-exports nothing from rdkafka: its types stay out of public signatures so rdkafka major bumps are not breaking changes here (see docs/DESIGN.md § Dependency policy).

Structs

KafkaBatch

One poll’s worth of messages, borrowed from the lane.

KafkaLane

One assigned partition’s pollable queue.

KafkaSource

Kafka source: one consumer-group member per process, partitions split into per-lane queues polled by pipeline threads. Constructed from config (KafkaSource::new) or a pipeline component section (KafkaSource::from_component_config).

KafkaSourceConfig

Configuration of a KafkaSource, deserialized from the pipeline’s opaque source: { kafka: ... } section.