Introducing intraday PCAP delivery

Today we're launching a major upgrade to our packet capture (PCAP) service, with the introduction of intraday delivery.

Previously, customers with a daily PCAP subscription needed to wait up to 24 hours for the prior day's PCAP files. With our upgraded infrastructure, most PCAP files are delivered within 20 minutes of the first packet in the file.

Intraday PCAP files are delivered via AWS S3. Setup typically takes 1–2 business days after signing an order form.

PCAPs contain raw market data in its original wire format and sequence.

The key benefit of market data in PCAP format is its losslessness. PCAPs enable precise market replay, incorporating effects like out-of-order packet arrival and microstructural details that may even be absent in normalized MBO (L3) data. Many of our enterprise customers also rely on Databento's PCAP files alongside our normalized data to audit data integrity.

Databento's PCAPs also include our PTP receive timestamps, allowing users to calculate timestamp deltas that are necessary for accurate backtesting in sub-millisecond environments like TCA and smart order routing.

Since the PCAPs preserve the native feed protocol, latency-sensitive users can use PCAPs to test their parser and feed handler implementations and benchmark their trading systems through packet replay.

This solution was developed for many of our users with time-sensitive pipelines for intraday or post-trade analytics.

While some data vendors provide PCAPs, none offer delivery guarantees better than end-of-day (T+1) either due to licensing or system limitations.

PCAPs files can be several GBs each after compression, making it slow slow to rotate and transfer out of colocated capture servers. It's possible to stream the tail of PCAP files for faster delivery, but this effectively resembles the raw multicast feed and is inconvenient for a variety of data pipeline and ETL workflows that are designed around file or S3 access. It's also possible to rotating the files more frequently to mitigate bandwidth bottlenecks, but this can be cumbersome on your ops team and filesystems with huge amounts of file metadata.

Leveraging Databento's real-time distribution licenses and extensive WAN bandwidth, we were able to develop a fast path for intraday PCAP delivery.

Intraday PCAP file deliveries are also cost-effective for customers who want to avoid real-time license fees with delayed access.

Since launch, we've been able to deliver the median PCAP file within 20 minutes of the first packet in the file, and 99% of files are delivered within 43 minutes. We segment our PCAP files in 10-minute intervals, i.e. meaning 99th percentile delivery occurs within within 33 minutes of the last packet in the file.

We capture the data on servers which are at most 1 or 2 switch hops (Arista 7130 or Arista 7060X) away from the exchange handoff. FPGA-based hardware timestamps are included alongside original timestamps embedded in the payload. The hardware timestamps are synced via PTP to a pair of GPS grandmasters with PTP class 6, clock accuracy within 25 ns of UTC.

Our existing users include major proprietary trading firms, quant hedge funds, ISVs, market data redistributors, and developers of low-latency execution and feed handling solutions.

Databento's PCAP service offers several unique features compared to most other PCAP providers, including:

• Lossless, full OPRA A/B tapes at up to 100 gigabit line rate.
• Same data used and tested in our normalized data solutions, ensuring application-level checks (like book building) for data integrity on every ticker.
• Precise truncation of packets at the file timestamp boundaries.
• No missing or out-of-order files during intraday delivery.
• Authorized distributor of ICE and NYSE market data and pcaps.
• Full order book and depth feeds from smaller US equity exchanges.
• Accurate annotation of network topology and RX deltas in capture system.
• FPGA-based hardware timestamps.