Module prelude 

Curated imports for pipeline assemblies: one use etl::prelude::*; brings in the builder, chain entry points, and the types every assembly touches.

Connector constructors are deliberately excluded — import those from their feature-gated modules (kafka, [clickhouse], [avro]). Additions to this module are semver-additive; nothing is ever removed.

Structs

ChainCtx

Per-thread wiring handed to the chain factory — everything the terminal .sink(...) stage needs, so assemblies stop threading queues, budget, and the pipeline name by hand.

ChunkConfig

Tuning for the terminal stage’s per-shard chunking.

ExitReport

Outcome of PipelineRuntime::run.

KeyHashRouter

Default router: key hash modulo shards, falling back to the source partition for keyless records (keeps a partition’s keyless records together and the distribution stable).

Owned

Family for record types that do not borrow (owned payloads).

PartitionId

Identifier of a source partition (dense, source-assigned).

Pipeline

The pipeline builder — see the module docs for the full picture.

PipelineConfig

Root of a pipeline’s configuration file.

PipelineRuntime

One pipeline process: source, per-thread chains, and a sink, assembled per docs/DESIGN.md § Process anatomy.

RuntimeOptions

Knobs that are not part of the user-facing YAML (loop granularities, test hooks). The defaults suit production; tests shrink the timings.

ShutdownHandle

Triggers a graceful drain from anywhere (tests, custom signal wiring).

SinkOptions

Sink wiring knobs that live outside connector config.

SinkParts

The decomposed sink. Construct with SinkParts::new and refine with the with_* methods (the struct is #[non_exhaustive]; fields may be added without breaking implementors).

SinkPoolConfig

Complete sink worker-pool configuration.

Enums

BuildError

Error assembling a pipeline (cold path, before anything runs).

ErrorPolicy

What to do when a record fails in a stage.

ExitState

Terminal state of a pipeline run.

LogFormat

Output format for logs.

PipelineError

Error from Pipeline::run: assembly or startup failure.

Traits

RecFamily

Maps a payload-buffer lifetime to the deserialized record type.

SinkBundle

A sink ready for assembly: the writer, the shard/replica topology, pool tuning, and optional metadata (metric labels, readiness probe).

Functions

chain

Start a chain from a deserializer producing family F.

chain_owned

Start a chain from a deserializer producing owned records T.