Guides
Task-oriented how-tos. Each guide gets you to one concrete outcome; none of them re-explains the architecture — for that, start with Concepts or docs/DESIGN.md.
| Guide | Outcome |
|---|---|
| Assembling a pipeline | Build and run a pipeline with the Pipeline builder — the primary assembly path. Read this one first. |
| Configuring pipelines | Write a pipeline YAML: typed framework sections, environment interpolation, opaque connector sections. |
| Testing pipelines | Test a whole assembly deterministically with etl-test's in-memory source and capturing sink. |
| Schema validation | Fail fast when your row struct, columns list, and the live ClickHouse table disagree. |
| Graceful shutdown | Drain cleanly on SIGTERM, size drain_timeout against Kubernetes, and interpret exit codes. |
| Manual assembly | Desugar every builder step into the public primitives it composes — the escape hatch. |
Where to go next:
- Connector-specific configuration and guarantees: Connectors.
- Running in production: Deployment.
- Writing your own source, sink, or operator: Extending.
- Every configuration key in one table: Configuration reference.