Raw market data with packet capture (PCAP) files

We require a minimum order of $3,000. Each PCAP delivery usually requires some manual coordination, so this helps ensure that we meet a minimum batch size that covers the engineering support costs.

Contact us for more samples.

We can usually provide up to 1 week of sample data on a recent, randomized week. If you have special delivery requirements, such as file naming conventions, we can also discuss this and see if we can produce a representative sample of your final delivery.

We only allow public download of a small slice of the available sample data since the files are large and take up bandwidth if hosted publicly.

The capture servers are usually 1 or 2 switch hops away from the trading venue's handoff, with very deterministic latency, and we can provide necessary latency offsets if you need them for simulation purposes.

The A/B feeds terminate on separate boundary switches and capture them on separate ports.

The capture cards are FPGA-based, usually a Napatech or Endace card, and are designed to achieve lossless capture up to line rates of 100G and hardware timestamping with higher accuracy than typical NICs like the Xilinx Solarflare cards.

Our timestamps are synchronized against a PTP source, which at most venues will be a pair of Endrun grandmaster clocks of PTP class 6 and clock accuracy of <25 ns against UTC (0x20/32d). The time source of the grandmaster clocks is GPS (0x20). We only slew our time adjustments so the timestamps will not step backwards.

On most trading venues, we capture all of the available channels. Some trading venues have more complex channel configurations, redundant channels for recovery and instrument reference data, or license-restricted access to certain channels, and we may not capture some of the channels that are unnecessary to get complete state and history.

For example, CME has weekly sessions and snapshot channels which may be useful for replaying data in the middle of the week, otherwise you'll need to replay the pcaps starting from Sunday each week, which can be expensive. On the other hand, capturing the entire snapshot loop nonstop is impractical and wasteful, so we limit our capture of snapshots.

Since we provide PCAPs to a wide range of customers, including the leading proprietary trading firms and quant hedge funds, we usually adhere to best practices when it comes to deciding how to deal with the redundant channels.

Each capture file spans 10 minutes, i.e. we rotate our capture files in 10 minute intervals. It's easy to merge or slice the files with standard PCAP utilities if you have other requirements.

Yes. We can provide this at a monthly rate with an annual contract. The exact cost will depend on your delivery requirements and the complexity of the project, but will usually be close to or the same as a one-time delivery of the same amount of data.

We're licensed as a real-time distributor on most trading venues listed and can deliver the data to you in roughly the time it takes to compress and transfer a few gigabytes of data twice (once to our central backup and then once to your desired destination) over WAN.

Depending on whether you have the appropriate real-time or delayed data subscriber licenses, we can either deliver the data intraday or embargo it until it meets historical data definitions so that you don't need to pay additional licensing fees.

For most of the venues we've listed, you are free to use it for internal and distribution purposes.

Most trading venues do not have restrictions on the use and distribution of their data once it is considered historical, which is usually defined by end of the trading session or some fixed number of hours after the end of the trading session or original publication time, or some fixed time like the end of the calendar day.

Some venues require us to obtain permission before we can distribute the data to you, even on historical basis.

Since we licensed as a real-time distributor on most trading venues listed, we have flexibility on how soon we can deliver the data to you based on your existing license arrangements. Other PCAP vendors are usually only licensed or set up to distribute the data after it becomes historical.

Normalization involves converting data from different source schemas into a single, unified set of schemas. Normalized market data is presented in a standardized manner, sparing users from the need to spend reconcile and align data from multiple sources on their own.

Raw market data refers to original, unprocessed data, directly from the source. Raw data retains all the specific details, formats, and structure of its source. For this reason, the most sophisticated trading firms generally prefer to ingest raw market data and normalize it to their own proprietary format. Databento's PCAP offering provides raw market data in its native wire protocol.

Users may prefer raw market data if they:

• Need user-directed flexibility in data handling. Users who want to apply their own normalization or have very specific analysis needs may prefer the raw, unaltered data to work with.

• Want to minimize lossiness. Raw market data includes all original data fields, providing maximum fidelity to the source data. Normalization can involve discarding or combining fields, as well as reducing precision in less-critical fields. For a more in-depth look at what happens during the normalization process, you can read our engineering post comparing normalized vs. raw data. Users may prefer normalized data if they:

• Desire simplified data handling. Normalized data provides a single schema across different data sources, with standardized formatting - for example, consistent timestamp representation across all data.

• Want standardized analysis, backtesting and live trading. Normalized data is ideal for scenarios where data from multiple sources needs to be compared or used together without constant translation or reconciliation. It also improves consistency between historical data used in backtesting and real-time data used in live trading environments.

• Want to cut down on storage requirements. Normalized data, as it's already been processed and often simplified, typically takes up less space and can be easier to manage.