Reference
Where to look things up. This guide teaches concepts and contracts; the sources of truth it defers to are listed here.
In this section
- Configuration — the full YAML schema: every key, type, and default in the framework sections, plus pointers to each connector's own section.
- Glossary — precise definitions of the terms used throughout the guide (lane, epoch, watermark, sealed batch, ...).
API reference (docs.rs)
Signatures, trait contracts, and per-item docs live in the rustdoc:
| Crate | Role |
|---|---|
| etl | The facade — the only crate applications depend on. Feature-forwards the connectors (kafka, clickhouse, avro, full). |
| etl-core | The engine: record/ack types, source/sink traits, operator chain, checkpointer, backpressure, pipeline builder and runtime, config, metrics, admin server. |
| etl-kafka | Kafka source (single consumer, split partition queues). |
| etl-clickhouse | ClickHouse ShardWriter and RowBinary encoder. |
| etl-avro | Avro deserializer: Confluent wire format, schema-registry client. |
| etl-test | In-memory source/sink mocks with scripting handles, for testing pipelines. |
Applications should import through etl (which re-exports the connector
crates behind features); the per-crate docs matter mostly to connector
authors — see Extending.
Canonical documents
- docs/METRICS.md — the full metric taxonomy: every metric name, type, label set, histogram bucketing, and alerting starting points. The Monitoring page explains the model; this file owns the list.
- docs/DESIGN.md — the architecture and every decision's rationale, including the frozen v1 trait contracts and the decision log. The source of truth when code and intuition disagree.
Related pages elsewhere in the guide
- Connector configuration bodies: Kafka, ClickHouse, Avro, Memory.
- Operational defaults in context: Tuning.