Lunary is an open-source NASDAQ TotalView-ITCH 5.0 parser written in Rust, designed for high-throughput market data processing with minimal overhead.
NASDAQ ITCH distributes order, cancel, execution, and trade events through a binary market data feed. A full trading session can contain hundreds of millions of messages, making parsing throughput, memory efficiency, and predictable latency critical.
Architecture
- Historical feeds are memory-mapped and parsed directly from memory.
- Message processing avoids per-message heap allocations.
- Parallel workloads are divided only at valid message boundaries.
- Worker threads are created before benchmark timing begins.
- Numeric fields are decoded directly from the binary representation with minimal intermediate state.
Benchmark Environment
- Dataset: NASDAQ TotalView-ITCH 5.0, October 30, 2019
- Messages: 293,989,079
- Dataset size: 9,035,016,843 bytes
- Machine: Google Cloud c3-standard-8
- CPU: Intel Xeon Platinum 8481C, 2.70 GHz
- Compute: 4 physical cores, 8 vCPUs
- Memory: 32 GB DDR5
- Measured DRAM latency: about 135 ns
- Operating system: Ubuntu 24.04
- Kernel: Linux 7.0.0-1011-gcp
- Compiler: Rust 1.98.1
- Optimization: -C target-cpu=native
Benchmarks operate entirely on memory-mapped data. Disk and network performance are excluded.
Processing Modes
Framing
Reads each message length and advances directly to the next message without decoding fields.
Full Decode
Decodes every field across all supported message types, including big-endian numeric conversion and checksum accumulation.
Performance
Median results across 10 full-dataset runs.
| Mode | Throughput | Data Rate |
|---|---|---|
| Single Thread, Framing | 213.7 M msg/s | 6.6 GB/s |
| Single Thread, Full Decode | 93.9 M msg/s | 2.89 GB/s |
| 8 vCPUs, Framing | 1,419 M msg/s | 43.6 GB/s |
| 8 vCPUs, Full Decode | 499.7 M msg/s | 15.4 GB/s |
Scaling remains below 8x because the benchmark host provides four physical cores with simultaneous multithreading.
Processing Cost
Average cost measured across batches of 1,000 messages:
- Framing: about 5.2 ns/message
- Full Decode: about 9.9 ns/message
Maximum observed latency was 22 µs on the cloud VM.
Correctness
Full-dataset validation processed all 293,989,079 messages using both Lunary and an independent reference decoder.
Both implementations produced the same checksum:
0x5143fd037119a935
Validation covers all 18 message types present in the dataset.
Benchmark Scope
- Single historical trading session
- Single cloud machine configuration
- No network or NIC processing
- Hardware performance counters unavailable under the VM environment
- Parallel benchmarks measure parser scalability rather than end-to-end feed-handler throughput
- Order book implementation tracks individual orders without maintaining aggregated price levels