01 / Real-time infrastructure
Market data.
Without the bottleneck.
A market-data pipeline redesigned around throughput, predictable latency, and data integrity.
01 / Decision record
The reasoning
underneath the result.
- 01The constraint
- The existing Python pipeline dropped events at peak load. Analytics needed to keep pace with multiple exchanges, at sub-millisecond latency.
- 02The decision
- Move the hot path to Rust. Use Kafka for reliable event streaming, custom lock-free data structures for computation, and tiered storage for real-time and historical access.
- 03The trade-off
- Performance-critical code belongs where its complexity earns its keep. Kafka and PostgreSQL retain clear responsibilities outside the hot path.
- 04The outcome
- The resulting system processes over two million events per second, with p99 latency below 500 microseconds and no data loss reported since deployment.
02 / System boundaries
Every part has
a clear responsibility.
01
Ingest
KafkaReliable event streaming
02
Compute
RustLock-free hot path
03
Serve
Tiered storageLive and historical queries
A high-level reading of the published architecture. Boundaries shown here are conceptual, not a deployment diagram.
03 / Outcomes
What changed.
2M+events per second
<500 μsp99 latency
0events dropped
85%cost reduction
Results reported in this case study. These are separate from the illustrative routing model.