Skip to main content

Concepts

The mental model behind etl-rs. These four pages explain the contracts the framework holds — and expects you to hold — with the reasoning behind each. The canonical, exhaustive version lives in docs/DESIGN.md; these pages are the working subset a pipeline author needs.

1. Architecture​

The anatomy of a running pipeline process: pinned pipeline threads (poll → deserialize → operator chain → route), the single shared tokio I/O runtime (sink workers, admin server, async fetchers), and the control plane. Why one process runs exactly one pipeline.

2. Delivery guarantees​

How at-least-once actually works: refcounted acknowledgements per source batch, the contiguity tracker, and watermarks that commit only after durable sink acks — across rebalances and shutdown. Also the honest limits: what ClickHouse dedup tokens do and do not cover, and how to design tables for replay.

3. Backpressure​

Flow control without ever blocking the source: try_send, pause, keep polling. The in-flight byte budget, its watermarks, and the sizing rule that keeps a saturated pipeline out of pause duty-cycling.

4. Error handling​

The error taxonomy (retryable, record-level, fatal), the two record-level policies (Skip and Fail — there is no dead-letter queue), and the metrics that make every drop and every error visible.

  • Guides — task-oriented how-tos built on these concepts.
  • Monitoring and docs/METRICS.md — observing the contracts at runtime.
  • Glossary — the vocabulary (lane, shard, watermark, epoch, sealed batch, ...).