Databento architecture

Databento's infrastructure is designed with the following considerations:

• Redundancy and high availability. There are at least two redundant servers, connections, and network devices in every hop of the critical path between us and your application.
• Self-hosted. All our servers are self-hosted. We only rely on a cloud provider — AWS — for our public DNS servers, the CDN for our website, and independent monitoring of our service uptime.
• Low latency. Our live data architecture is optimized for low-latency use cases, such as algorithmic trading.
• High data quality. We employ several techniques to achieve lossless capture and highly accurate timestamping.

Live data architecture

Our live data architecture is composed of the following elements:

• Colocation and direct connectivity to venue extranet
• Low latency switches and interconnects
• Lossless capture servers
• Zero-copy messaging
• Real-time data gateways
• SmartNIC-accelerated load balancing, firewall, and routing
• Multiple enterprise IP transit providers
• Inline historical data capture and storage

Users connecting to us over a public network from within the same data center or metro area, using our Raw API and Databento Binary Encoding, can expect to receive data less than 1 millisecond after the venue disseminates the original data from their matching engine.

Colocation and direct connectivity to venue extranet

Our capture servers and real-time data gateways are generally colocated in the same primary data center as the venue's matching engine. We receive the data through a pair of direct connections (also called handoffs) to the venue's private network. This minimizes both the latency of our service and the variance of the network topology between us and the matching engine, yielding highly deterministic timestamps.

The venue will sometimes employ fiber equalization to ensure the same latency is achieved at handoffs across multiple data centers. In these cases, we may colocate our capture servers in any one of the data centers with equalized latency.

The only exception to primary data center colocation is when there are microstructural reasons for our trading users to prefer receiving the data elsewhere. These may include a central location for fragmented venues (like US equities), or proximity to a demarcation point for long-haul connectivity.

Info

You can find the exact locations of our capture servers for each venue from our list of venues.

Low latency switches and interconnects

Depending on the specific venue, the data takes one to three switch hops after the handoff to reach our capture server. One of those switch hops, at most, may be a cross-connect within the same data center. We use a mix of Layer 1 and 3 switches, with 4 nanosecond and sub-300 nanosecond port-to-port latencies, respectively.

ts_in_delta represents the time it takes (in nanoseconds) for a data message to travel from the venue to our capture server's network card.

Info

You can find the schemas that include the ts_in_delta field from our list of schemas.

Lossless capture servers

Most venues disseminate their data over UDP multicast and provide a pair of multicast streams to offer protection against packet loss. Each of our capture servers receives both streams and compares packet sequence numbers to fill data gaps in the first stream. We call this packet arbitration or A/B arbitration.

We perform a second layer of packet arbitration between our capture servers to further reduce the risk of data loss. This is a proprietary part of our system that we call X/Y arbitration. As such, our setup receives at least four redundant streams of the data.

Most software-based capture processes are unable to sustain 100% line rate capture during the 10 to 40 gigabit bursts that we constantly encounter from the data feeds.

To address this, we offload the capture of the packets to an FPGA on each capture server, which reduces packet loss down to the order of one in a billion packets per stream. Combined with our packet arbitration process across four redundant streams, our process significantly mitigates the risk of data loss on the billions of packets that we receive daily on full venue feeds.

Zero-copy messaging

We employ a brokerless messaging pattern with a zero-copy internal messaging protocol to support our low latency, high throughput requirements. Once processed in our capture servers, the data only takes a single sub-300 nanosecond switch hop to our real-time data gateways.

Real-time data gateways

The data is preprocessed in real time to support the various real-time customizations available on our live service. This includes parsing, normalization, and book construction. Depending on the venue, about 5 to 35 microseconds are spent in our capture server including through these preprocessing steps.

Our real-time data gateways are responsible for live data distribution. This includes negotiation of TCP sessions, inline data customization, transcoding, and handling of requests that vary from client to client. Our data spends about 5 microseconds in this layer before it takes another switch hop either to a customer's link (for customers connected over private cross-connect or peering) or to a network gateway that handles load balancing, firewall, and BGP routing (for customers connected over internet).

Info

We also allow users to bypass our firewall with a private cross-connect or peering to achieve sub-41.3 microsecond latencies. Learn more from our Databento locations and connectivity options guide or contact us for details.

The time that the data terminates on the venue's handoff to the time that our processed data leaves any of our real-time data gateways is provided in our data as ts_out - ts_recv.

Info

Receiving ts_out is enabled through a parameter at the authentication stage of the Raw API and in the constructors of the Live clients.

SmartNIC-accelerated load balancing, firewall, and routing

We employ DPDK-based, SmartNIC-accelerated network gateways that process packets up to 100 gigabit line rate with a median latency of 64 microseconds and 99^th percentile latency of 311 microseconds.

Multiple enterprise IP transit providers

To maintain stable routes from public internet, we multiplex connections from multiple IP transit providers and private peering partners on our network gateways.

Inline historical data capture and storage

We capture our historical data on the same servers that normalize and disseminate our live data in real time. This ensures our historical data has the same timestamps as our live data.

Historical data is stored on a self-hosted, highly available storage cluster. To protect data integrity, our storage system has built-in detection and self-healing of silent bit rot.

The stored data is then made available through our historical API servers after a short delay, though licensing restrictions may limit when you can access this data. To ensure high uptime, our historical API servers are deployed as multiple instances in our cluster.

See also

You can read our docs on precision, accuracy, and resolution and timestamps on Databento and how to use them for more details about our data integrity and timestamping accuracy.