Introducing Intraday PCAP Delivery

Today we're launching a major upgrade to our packet capture (PCAP) service , with the introduction of intraday delivery.

Previously, customers with a daily PCAP subscription needed to wait up to 24 hours to receive the previous day's PCAP files. With our upgraded infrastructure, customers can receive PCAP files within 20 minutes of the original market data message in most cases.

Delivery is done through a standard AWS S3 bucket and is usually set up within 1 or 2 business days of signing an order form.

Packet captures contain raw market data, in their original wire format and sequence.

The most valuable benefit of getting market data in PCAP format is its losslessness. PCAPs allow a user to replay the market precisely, incorporating effects like out-of-order packet arrival and microstructural features of the original market data messages that may not be found even in normalized MBO (L3) data. Many of our enterprise customers also rely on our PCAP files and normalized data simultaneously to audit our data integrity.

Databento's PCAPs also include our PTP receive timestamps, which allow users to measure timestamp deltas that are necessary for accurate backtesting in sub-millisecond environments like TCA and smart order routing.

Since the PCAPs preserve the native feed protocol, latency-sensitive users can use PCAPs to test their parser and feed handler implementations and benchmark their trading systems through packet replay.

This solution was developed for several of our users who had time-sensitive pipelines for intraday or post-trade analytics.

While a few market data vendors provide PCAPs, none of them were able to provide deterministic guarantees on delivery time that were better than end-of-day (T+1) either due to licensing or technological limitations.

PCAPs files can be several GBs each after compression and can be slow to rotate and transfer out of capture servers at the colocation site. It's possible to deliver PCAPs arbitrarily fast, but going to the extreme, streaming the tail of PCAP files will simply resemble the raw multicast feed and is inconvenient for a variety of data pipeline and ETL workflows that are designed around file or S3 access. Rotating the files very frequently can ensure file transfers are not bottlenecked by bandwidth, but can be cumbersome on your ops team and filesystems with huge amounts of file metadata.

With Databento's extensive real-time distribution licenses and WAN bandwidth, we were able to create a fast path for these users to receive PCAP files intraday.

Intraday PCAP file deliveries are also ideal for customers who want to avoid real-time access fees, as delayed data license fees are usually much more affordable.

Since going to production, we've been able to deliver the median PCAP file within 20 minutes of the first packet in the file. 99th percentile delivery time is within 43 minutes of the first packet in the file. We segment our PCAP files in 10-minute intervals, i.e. the file is delivered within 33 minutes of the last packet in the file at the 99th percentile.

Our existing users include major proprietary trading firms, quant hedge funds, leading ISVs, market data redistributors, and developers of low latency execution and feed handling solutions.

Our team started Databento after using every commercial PCAP service at our previous roles in tier 1 trading firms. Some of us even architected the existing capture setups that are in use at other data vendors today. So we've built our PCAP service to improve on the many limitations of legacy market data providers. If you're looking for a PCAP provider, just ask: can your vendor do these?

• Lossless, full OPRA A/B tapes up to 100 gigabit line rate.
• Same data used and tested in our normalized data solutions, ensuring application-level checks (like book building) for data integrity on every ticker.
• Precise truncation of packets at the file timestamp boundaries.
• No out-of-order or missing file uploads unless there's a data gap.
• Authorized distributor of ICE and NYSE market data and pcaps.
• Full order book and depth of market feeds from smaller US equity exchanges.
• Accurate annotation of network topology and RX deltas in capture system.
• FPGA-based hardware timestamps.