Crate etl_core 

The engine of the etl-rs framework.

etl-core contains the pipeline runtime and every technology-neutral abstraction: records and their checkpoint tokens, the operator chain, the source and sink traits, checkpointing, backpressure, configuration loading, metrics, and the admin server.

Applications should depend on the etl facade crate rather than on etl-core directly.

The architecture and its invariants are documented in docs/DESIGN.md; the metric taxonomy in docs/METRICS.md.

Re-exports

pub use error::DeserError;

pub use error::ErrorClass;

pub use error::ErrorPolicy;

pub use error::FatalError;

pub use error::SinkError;

pub use error::SourceError;

pub use record::Flow;

pub use record::PartitionId;

pub use record::RawPayload;

pub use record::Record;

pub use record::RecordMeta;

pub use bytes;

Modules

admin

Admin HTTP server: /metrics, /healthz, and /readyz.

backpressure

Backpressure: the global in-flight byte budget and the watermark pause/resume controller with hysteresis.

checkpoint

Checkpointing: acknowledgement tracking and watermark commits.

config

Pipeline configuration: typed framework sections plus opaque per-component passthrough, loaded from YAML with ${VAR:-default} environment interpolation.

deser

Deserialization contract: one borrowed payload in, 0..N records out, push-style.

error

Error taxonomy and per-stage error policies.

metrics

Metrics: exporter installation and pre-registered handle structs for every pipeline stage.

ops

Operator chain: statically composed push stages behind one type-erasure boundary per batch.

pipeline

Pipeline runtime: pinned driver threads, the source controller, and process assembly.

record

Core record types: payloads, metadata, and the flow-control signal.

sink

Sink abstraction: pipeline threads encode, shard workers batch and write.

source

Source abstraction: a control plane (Source) and a data plane (SourceLane).

telemetry

Structured logging: tracing initialisation (JSON for Kubernetes) and rate-limited hot-path logging helpers.

Macros

rate_limited_warn

tracing::warn! behind a RateLimit. When events were suppressed since the last allowed one, the emitted line carries a suppressed field with the count.