Avro-fast pipeline

The avro-fast backend (serde_avro_fast, single-pass typed decode, borrowed records) is measured against the apache-avro owned path at two layers. The point of two rigs is honesty: the CPU rig shows the true decode multiple, and the end-to-end rig shows that at scale the bottleneck has already moved to the sink.

CPU rig — decode isolated

In-process, no broker, no server: a synthetic source replays a pre-encoded Avro batch datum (one datum = 50 events) into the chosen decoder. STAGE=decode isolates the decoder, reproducing the etl-avro micro-bench inside the normalized harness.

Avro decode cost per event

Lower is better

Per-event decode time for one 50-event batch datum, by decoder backend.

serde_avro_fast (borrowed) (highlighted)other arms

Data table

Variant	Value	95% CI	n
serde_avro_fast (borrowed)	22.590098168735793 ns	22.50430349569085 – 22.675892841780737	9
serde_avro_fast (owned)	42.6025793475575 ns	42.19149651951602 – 43.01366217559898	9
apache-avro (owned)	163.75838111356936 ns	163.01027746338238 – 164.50648476375633	9

Apple M5 Max · commit e1d508e821 · 2026-07-10

Borrowed fast decode vs apache-avro (owned)

7.2×

STAGE=decode · lower ns/event is better

STAGE=pipeline adds the rest of the CPU work — flat_map fan-out, filter, columnar encode, and shard handoff — so throughput reflects the whole in-process seam, not just the decoder.

CPU-rig pipeline throughput

Higher is better

Decode + flat_map + filter + encode + shard handoff, events per second.

serde_avro_fast (borrowed) (highlighted)other arms

Data table

Variant	Value	95% CI	n
serde_avro_fast (borrowed) · RowBinary	21,955,672.689608965 records/s	21903651.785101444 – 22007693.594116487	9
serde_avro_fast (borrowed) · Native	17,631,355.217508323 records/s	17574511.819038685 – 17688198.61597796	9
serde_avro_fast (owned) · RowBinary	14,271,294.660410881 records/s	14148721.844879635 – 14393867.475942127	9
serde_avro_fast (owned) · Native	12,022,069.90510023 records/s	11893307.378602477 – 12150832.431597983	9
apache-avro (owned) · RowBinary	5,198,127.463792451 records/s	5158224.833822218 – 5238030.093762684	9
apache-avro (owned) · Native	4,926,922.583382715 records/s	4915393.898011487 – 4938451.268753942	9

Apple M5 Max · commit e1d508e821 · 2026-07-10

Borrowed fast pipeline vs apache-avro (owned), RowBinary

4.2×

STAGE=pipeline · whole in-process seam · lower ns/event is better

The two multiples are the point of the CPU rig: the borrowed decoder is ~7.2× faster to decode and ~4.2× faster through the whole in-process pipeline (decode → flat_map → filter → encode → handoff) than the apache-avro owned path, the pipeline pair both on RowBinary. Native costs more CPU to encode than RowBinary, so it trails on this rig; that trade only pays off server-side, and end to end (below) it disappears entirely.

E2E rig — the bottleneck moves to the sink

The same work end to end: an Avro batch payload produced to Kafka, decoded, fanned out, and written to ClickHouse. Here the decode delta is a small share of per-row cost because the sink dominates.

End-to-end rows/s (Kafka → ClickHouse)

Higher is better

Sustained rows landed in ClickHouse per second, by decoder + insert format.

Native (highlighted)other arms

Data table

Variant	Value	95% CI	n
apache-avro (owned) · Native	213,552.84617704048 rows/s	—	3
serde_avro_fast (borrowed) · Native	212,988.24099492145 rows/s	—	3
serde_avro_fast (owned) · Native	212,650.0353983392 rows/s	—	3
serde_avro_fast (borrowed) · RowBinary	212,606.9433286385 rows/s	—	3

Apple M5 Max · commit e1d508e821 · 2026-07-10

The raw deser: none arm is excluded from these charts: it is a byte passthrough that writes one ClickHouse row per Kafka message, while every Avro arm explodes each message into 20 event rows. Its ~55.5k rows/s is 20× fewer rows for the same messages, so it is not comparable on a rows/s axis and would falsely imply Avro decode outruns a raw copy. Each record now carries its explode factor in variant.events (=20 on the Avro arms; the raw arm records variant.payload instead), so the "not comparable to the raw arm" caveat is machine-visible, not just prose.

Sink flush p99

Lower is better

Tail flush latency — a different unit, so its own panel (never a second axis).

Native (highlighted)other arms

Data table

Variant	Value	95% CI	n
apache-avro (owned) · Native	0.024689655172413796 s	—	3
serde_avro_fast (borrowed) · Native	0.0247140625 s	—	3
serde_avro_fast (owned) · Native	0.024737500000000003 s	—	3
serde_avro_fast (borrowed) · RowBinary	0.09919230769230769 s	—	3

Apple M5 Max · commit e1d508e821 · 2026-07-10

Across the four Avro arms the decoder choice barely moves the rate: on Native, borrowed serde_avro_fast lands −0.26% and owned serde_avro_fast −0.42% from owned apache-avro — every arm sits within ±0.5% across n=3 reps (medians ~213k rows/s). No measurable difference, because the ClickHouse sink, not the decoder, is the limiter. The insert format does not move the rate either: fast_borrowed + RowBinary lands within −0.13% of the same decoder on Native (both ~213k rows/s) — the earlier revision's ~−2.8% RowBinary throughput penalty did not survive n=3. What the format does change is the flush tail: RowBinary's sink-flush p99 falls in the 50–100 ms histogram bucket (interpolated ~99 ms) against Native's 10–25 ms bucket (~24.7 ms). Throughput is sink-bound and identical; only the flush-latency tail separates the two formats.

How the E2E rig is driven

The current rig runs saturated: RATE=0 means the producer deliberately outruns the pipeline, so the measured rate is the pipeline's ceiling rather than a throttled target. Each arm is run three times; the charts and quoted figures use per-arm medians (n=3). Every run is a 30 s consumption window, EVENTS=20 (20 event rows exploded per Kafka message, recorded in each record's variant.events), 4 partitions, 2 pipeline threads, against local Kafka and ClickHouse containers. Each run uses a fresh per-pid topic and consumer group, so the un-drained saturation tail is abandoned when the window closes — it is not replayed on a later run. Under saturation the Native arms log only a handful of backpressure_pauses (0–3 across reps), so they are flush-bound, not backpressure-bound.

etl_e2e_latency_seconds is wired: it is observed on the sink ack path (crates/etl-core/src/sink/worker.rs, via SinkShardMetrics::e2e_observed), and every arm now reports e2e p50/p99. Read those tail figures as coarse, though: the e2e latency histogram uses second-wide buckets, so the reported p50/p99 are linear-interpolation positions inside the (0.5, 1.0] bucket (edges 0.5 s and 1.0 s) — 0.750 s / 0.995 s on the reps that stay inside it — the interpolation math, not a fine measurement of the true tail; they should not be read to three decimals. The sink-flush p99 above is bucket-interpolated the same way (its ~24.7 ms and ~99 ms are the 10–25 ms and 50–100 ms bucket bands), so it is not a finer-grained signal than the e2e histogram — both are coarse. A related harness lesson, now encoded in the binaries: metric handles created before metrics::install bind to the no-op recorder and silently render nothing — the harnesses install the recorder first, and the examples should be audited for the same pattern.