Module pipeline 

Pipeline runtime: pinned driver threads, the source controller, and process assembly.

Thread anatomy (see docs/DESIGN.md § Process anatomy):

main (run())          controller (std thread)      driver 0..N (std threads)
  metrics/admin/io  ──  owns Source + Checkpointer ── own lanes + chain
  joins everything      poll_events / commit tick     poll → push_batch → route
                        pause/resume application      backpressure ticks

Communication: the controller sends [ThreadControl] messages to drivers (lane assignment, drain barriers); drivers send [DriverEvent] requests back (pause/resume — only the controller touches the Source — and fatal reports). All channels are unbounded crossbeam channels: control traffic is rare and must never block a poll loop.

Shutdown (also the full-revocation path, per DESIGN.md § Shutdown): SIGTERM → controller stops event polling and sends Shutdown to every driver → each driver flushes its chain, drops its lanes, and arrives at the barrier → main joins driver threads (dropping chains closes the shard queues) → the sink drains under the remaining deadline → the controller runs a final drain + commit + flush_commits → the process reports an ExitReport. A sink that cannot flush by the deadline is abandoned loudly; unacknowledged offsets are never committed, so the data replays after restart (at-least-once).

Re-exports

pub use crate::sink::DrainReport;

pub use crate::sink::SinkDrainFn;

pub use crate::sink::SinkProbeFn;

Structs

ChainCtx

Per-thread wiring handed to the chain factory — everything the terminal .sink(...) stage needs, so assemblies stop threading queues, budget, and the pipeline name by hand.

ExitReport

Outcome of PipelineRuntime::run.

FatalErrorReport

Owned copy of the fatal error carried in the exit report.

Pipeline

The pipeline builder — see the module docs for the full picture.

PipelineRuntime

One pipeline process: source, per-thread chains, and a sink, assembled per docs/DESIGN.md § Process anatomy.

RuntimeOptions

Knobs that are not part of the user-facing YAML (loop granularities, test hooks). The defaults suit production; tests shrink the timings.

ShutdownHandle

Triggers a graceful drain from anywhere (tests, custom signal wiring).

SinkOptions

Sink wiring knobs that live outside connector config.

SinkRuntime

The sink half the runtime drives: the shared shard-queue handle plus a drain hook invoked once at shutdown with the remaining drain budget.

Enums

BuildError

Error assembling a pipeline (cold path, before anything runs).

ExitState

Terminal state of a pipeline run.

PipelineError

Error from Pipeline::run: assembly or startup failure.

StartError

The pipeline could not start.

Functions

metrics_settings

The MetricsSettings a pipeline configuration maps to.