etl_kafka/lib.rs
1//! Kafka source for the `etl-rs` framework.
2//!
3//! Built on `rdkafka` with a **single consumer per process**: partitions
4//! are split into per-partition queues (`split_partition_queue`) and fanned
5//! across pipeline threads as [`SourceLane`](etl_core::source::SourceLane)s,
6//! keeping payload borrows local to the polling thread with zero copies.
7//! Offsets are stored when checkpoint watermarks advance and committed on
8//! an interval; rebalances and shutdown share the framework's drain
9//! choreography, with completion deferred until draining and a synchronous
10//! commit have finished (see [`KafkaSource`] docs).
11//!
12//! One topic per pipeline: the framework's `PartitionId` is the Kafka
13//! partition number. Kafka tombstones (null payloads) surface as empty
14//! payload slices.
15//!
16//! Configuration is read from the pipeline's opaque `source: { kafka: ... }`
17//! section — see [`KafkaSourceConfig`] for the schema and the raw
18//! `rdkafka:` passthrough (framework-owned properties are validated and
19//! rejected with explanations).
20//!
21//! ```yaml
22//! source:
23//! kafka:
24//! brokers: ${KAFKA_BROKERS:-localhost:9092}
25//! topic: orders
26//! group_id: orders-etl
27//! commit_interval: 5s
28//! rdkafka:
29//! fetch.message.max.bytes: "1048576"
30//! ```
31//!
32//! This crate deliberately re-exports nothing from `rdkafka`: its types
33//! stay out of public signatures so `rdkafka` major bumps are not breaking
34//! changes here (see `docs/DESIGN.md` § Dependency policy).
35
36mod config;
37mod context;
38mod error;
39mod lane;
40mod source;
41
42pub use config::KafkaSourceConfig;
43pub use lane::{KafkaBatch, KafkaLane};
44pub use source::KafkaSource;