Module ops 

Operator chain: statically composed push stages behind one type-erasure boundary per batch.

Stages compose via Collector (monomorphized — a whole chain compiles to one loop); the only virtual call on the data path is RunnableChain::push_batch, once per poll batch. Records are born (deserialized) and die (encoded into shard frames, filtered, or skipped) inside a single push_batch call, so borrowed payloads never cross or outlive the boundary. See docs/DESIGN.md (§ Frozen v1 contracts).

Structs

ChainBuilder

Fluent builder for one pipeline’s operator chain. DF is the deserializer’s record family; CurF the family at the current end of the chain (changed by map_rec and flat_map, and by map for owned payloads).

ChainFactory

Stamps out identical chains — one per pipeline thread. Send + Sync when the deserializer, stage closures, encoder, and router are.

ChunkConfig

Tuning for the terminal stage’s per-shard chunking.

Emitter

Stack-borrowed emitter handed to flat_map closures. Parameterized by the output family (a 'static tag), so user closures never name the concrete downstream stack type or the buffer lifetime. Each emit is one virtual call — confined to flat_map stages.

Filter

filter: drop records failing the predicate. A drop releases the record’s ack share — it counts as success for the batch.

FilterPart

Recorded filter stage.

FlatMap

flat_map: one record in, 0..N out via a stack-borrowed Emitter. Carries the output family as a type parameter so the impl is fully constrained (closure argument types are not associated bindings).

FlatMapPart

Recorded flat_map stage.

Inspect

inspect: observe without transforming (no metrics of its own).

InspectPart

Recorded inspect stage.

Map

map: transform the payload.

MapPart

Recorded map/map_rec stage.

Root

The empty stage list.

SinkHandoff

The chain’s terminal stage. Owns one accumulation buffer per shard, seals EncodedChunks at ChunkConfig::target_bytes, and hands them to the sink workers through the bounded ShardQueues — a try_send that never blocks the pipeline thread.

SinkedChain

A fully specified chain, ready to build — or to stamp out one instance per pipeline thread via SinkedChain::build_factory.

TryMap

try_map: fallible transform with a per-stage ErrorPolicy. Skip drops the record (releasing its ack share) and counts it; Fail records a fatal error — the batch aborts and the pipeline stops.

TryMapPart

Recorded try_map/try_map_rec stage.

TypedChain

A concrete chain: deserializer + statically composed operator stack, erased behind RunnableChain. Ops must accept the family’s record type at every buffer lifetime — the HRTB is what makes borrowed records legal behind the erased boundary.

Enums

BlockReason

Why a batch could not complete yet. Both cases are retried with the resume cursor, but only BlockReason::Capacity engages the driver’s backpressure controller — a not-ready wait is an upstream dependency (e.g. a schema fetch), not sink pressure, and pausing the source for it would misreport the pipeline’s state.

PushOutcome

Result of pushing one batch (or a resumed suffix of one) through a chain.

Traits

Assemble

Assembles recorded parts into the concrete collector stack, given the terminal stage. Takes &self so one set of parts can assemble many identical chains — stage closures must be Clone (plain closures and closures over Clone/Arc state are).

Collector

Push-model stage: receives one record, forwards 0..N downstream.

CollectorFor

Family-erased collector: accepts the family’s record type at any buffer lifetime through a lifetime-generic method, which keeps it dyn-compatible. This is what lets flat_map closures hold a plain &mut Emitter<'_, OutF> without naming the downstream stack type.

RunnableChain

THE SEAM — the one erasure boundary between a pipeline thread’s driver loop and a typed chain. The methods are generic over the buffer lifetime only, so Box<dyn RunnableChain> is legal.

StageLifecycle

Per-batch lifecycle cascade implemented by every stage. Combinators handle their own concern and delegate downstream; the terminal stage anchors the recursion.

Functions

chain

Start a chain from a deserializer producing family F.

chain_owned

Start a chain from a deserializer producing owned records T.