Technical Architecture
Lunyn's parsing engine represents a complete rethinking of market data processing architecture. Traditional approaches sacrifice either performance or maintainability. We engineered both.
Core Architecture
Parsing operations occur directly on ingested buffers without intermediate allocations. Message boundaries are identified through pointer arithmetic, and field access happens via calculated offsets.
Performance impact: 3-4x throughput improvement vs. allocation-heavy approaches.
Multi-threaded processing using lock-free queues and wait-free data structures. Each parsing thread operates independently with minimal synchronization. Message distribution uses MPSC (multi-producer, single-consumer) queues for message routing, memory ordering guarantees for safe concurrent access, and per-thread state to eliminate cache contention.
Performance impact: Linear scaling across CPU cores without lock contention.
Critical parsing operations leverage AVX2 instruction sets to process multiple fields simultaneously: Parallel byte-to-integer conversion, vectorized field extraction, and SIMD-optimized timestamp parsing. Specific operations can process 8 message fields in parallel vs. sequential processing.
Performance impact: 2-3x improvement on parsing-intensive message types.
Comprehensive error handling for malformed messages (graceful degradation), out-of-sequence data (sequence validation), protocol violations (detailed error reporting), and edge cases (tested against 500GB+ production data). System maintains parsing state across errors and provides detailed diagnostics for debugging.
System maintains parsing state across errors with detailed diagnostics.
Complete ITCH 5.0 Implementation
| Message Type | Frequency | Parse Latency | Memory |
|---|---|---|---|
| System Event | Rare | 8ns | 16B |
| Stock Directory | Daily | 12ns | 256B |
| Add Order | Very High | 8ns | 64B |
| Add Order (MPID) | High | 9ns | 72B |
| Order Executed | High | 8ns | 48B |
| Order Executed (Price) | Medium | 9ns | 52B |
| Order Cancel | High | 7ns | 40B |
| Order Delete | Medium | 7ns | 32B |
| Order Replace | High | 10ns | 64B |
| Trade (Non-Cross) | High | 9ns | 56B |
| Cross Trade | Low | 11ns | 72B |
| Broken Trade | Rare | 8ns | 40B |
| NOII | High | 13ns | 128B |
| RPII | Medium | 9ns | 48B |
Validated Performance Metrics
Linear scaling characteristics
Measured across all message types
Excluding user buffers
Measured under sustained load
All metrics reproducible via our benchmark suite. Contact us for evaluation access.
View Full Benchmark ReportDeployment Options
Link directly into your application via provided API. Supports real-time streaming interfaces, batch file processing, and custom message routing.
Deploy as microservice with gRPC or REST API, Kafka/Redis output streams, and monitoring and alerting integration.
Command-line tool for batch conversion (ITCH → Parquet/CSV), historical data processing, and data quality validation.
Evaluate the Architecture
Request access to detailed technical documentation, benchmark methodologies, and integration guides for your evaluation.