Benchmarks

Methodology and recorded results. Numbers are machine-specific; every table states its environment, and re-run instructions accompany each harness (see Methodology).

The charts on these pages render from the versioned JSONL that each rig appends to benchmarks/results/ (the record shape lives in benchmarks/src/report.rs). Each metric carries its own unit and its own direction of goodness, so a chart reads "higher is better" or "lower is better" from the data rather than from the plotting code. Until a rig has been re-run under the current schema, its chart shows a "no recorded data yet" placeholder. Every rendered chart also ships an exact data table (its <details> fallback); the committed JSONL under benchmarks/results/ is the durable record.

Operator-chain cost

~9ns/record

Zero per-record allocations · 109M records/s through a realistic chain

Native vs RowBinary: server parse CPU

−91.8%

events schema, parse-isolated (Null engine)

Native vs RowBinary: compressed wire

49–71% smaller

events, lz4 / zstd:3

Split-queue vs per-thread consumers

−2.2%

at realistic per-record work — split queues confirmed

The two rigs, and the honest framing

The avro-fast backend (borrowed, zero-copy) decodes roughly 7.2× faster than apache-avro on the CPU rig. But a full end-to-end pass sustains hundreds of thousands of rows per second, and the ClickHouse sink — not the decoder — sets that rate, so the decode win shrinks to a fraction of a percent end to end (within ±0.5% across n=3): real, but not the headline. Two rigs exist precisely to say this out loud rather than cherry-pick:

• CPU rig (no broker, no server): decode → flat_map → Native encode. Shows the true multiple the zero-copy decoder buys.
• E2E rig (Kafka + ClickHouse): the same work end to end. Shows the bottleneck has moved to the sink.

The Avro-fast pipeline page states both.

Tiers

1. Micro — cargo bench (criterion + divan AllocProfiler) in crates/etl-core/benches and crates/etl-avro/benches (the fast-backend decode variants are feature-gated: cargo bench -p etl-avro --features fast).
2. Framework overhead — benchmarks/pipeline_synthetic (release binary, JSON report): in-memory source → chain → null sink; no broker in the loop.
3. At scale — benchmarks/e2e_kafka_clickhouse + benchmarks/loadgen against local containers or external clusters (env-configured endpoints).

Nightly CI compares criterion baselines with critcmp — including etl-avro's fast-backend decode variants, which run as their own --features fast step so the gated benches are regression-tracked too; divan allocation assertions are hard failures.

The studies

Page	What it answers	Backing data
Avro-fast pipeline	The zero-copy decode A/B, CPU-bound and at scale	avro_pipeline, e2e_kafka_clickhouse
ClickHouse format	Native vs RowBinary go/no-go	ch_native_format
Framework overhead	The framework ceiling and a tuning finding	pipeline_synthetic
Kafka topology	Per-thread vs split-queue consumers	kafka_topology
Zero-copy seam	Borrowed vs owned operator chain	divan micro-bench (prose)

A few figures are hand-recorded where no rig emits them — the zero-copy seam micro-bench table, the ClickHouse insert-transport table on the ClickHouse format page, the Kafka topology spike table, the framework-overhead 24× collapse figure, and the fast-backend criterion table. Each is disclosed inline on its own page.