Download and merge PCAPs with Databento for market replay

This tutorial was written by Roman Bansal, a Databento user and founder of Nanoconda. Nanoconda is a low-latency DMA platform that supports high-frequency trading strategies.

In this guide, Roman walks through how to download, decompress, and merge Databento PCAP files from the previous trading day’s market open. The result is a single, ready-to-use file that can be fed into the Nanoconda Trade Simulator to replay historical trading sessions at real-time speed.

Make sure the following tools are installed:

• awscli to download PCAP files from your S3 bucket.
• zstd to decompress .zst files.
• mergecap from Wireshark suite to merge multiple PCAPs into one.

Install any missing packages using your system’s package manager.

• Rocky Linux: sudo dnf install awscli zstd wireshark-cli
• Ubuntu: sudo apt install awscli zstd tshark
• Debian: sudo apt install awscli zstd wireshark-common
• Arch Linux: sudo pacman -S aws-cli zstd wireshark-cli

This pull_yesterday_pcap.sh shell script automates downloading, decompressing, and merging CME PCAP files between 14:00 and 17:00 UTC on the previous trading day.

#!/bin/bash

download_dir="/home/user/databento/$yesterday"
base_dir="$download_dir/startday"
recording_dir="/home/user/recordings"

mkdir -p "$base_dir"
aws s3 cp s3://yourbucket/databento/$yesterday/ "$download_dir" --recursive

cd "$base_dir"

cp ../dc3-glbx-ab-${yesterday}T14*.pcap.zst .
cp ../dc3-glbx-ab-${yesterday}T15*.pcap.zst .
cp ../dc3-glbx-ab-${yesterday}T16*.pcap.zst .

for f in *.pcap.zst; do
    zstd -d "$f" -o "${f%.zst}"
done

mergecap -w merged_output.pcap $(ls -1v *.pcap)

mv merged_output.pcap "$recording_dir/${yesterday}-databento.pcap"

ln -sf "$recording_dir/${yesterday}-databento.pcap" "$recording_dir/lastday.pcap"

rm *.pcap

It performs the following steps to prepare the files for simulation:

• Set variables: Defines the local staging directory ($base_dir) and the output directory for the merged PCAP file ($recording_dir), based on yesterday’s date.
• Download PCAPs from S3: Uses aws s3 cp --recursive to download yesterday’s PCAP files into the parent directory of $download_dir for processing. Databento’s deduplicated CME PCAP files typically follow the naming convention dc3-glbx-ab-*.pcap.zst. You may need to adjust this depending on your feed and whether you receive A/B-side files separately.
• Copy relevant files: Filters for a subset of PCAP files from 14:00 to 17:00 UTC and copies them into $base_dir to capture the U.S. market open.
• Decompress files: Converts all .pcap.zst files in $base_dir to standard .pcap format using zstd.
• Merge PCAPs: Uses mergecap to combine the decompressed files into a single merged_output.pcap, ordered by filename using ls -1v.
• Move and link for simulator: Moves the final merged file to $recording_dir and updates the lastday.pcap symlink, as specified in the Nanoconda XML config.
• Cleanup: Deletes temporary .pcap files from $base_dir to keep the staging directory clean.

After the script runs, /home/user/recordings/lastday.pcap points to the latest processed PCAP file from the previous trading day.

You can also add the entry below to your cron schedule to run the script on weekdays:

02 09 * * 2-6 /home/user/databento/pull_yesterday_pcap.sh

Note: Although CME futures trade nearly 24 hours a day from Sunday evening through Friday evening, this time window captures the surge in activity—particularly for equity index futures—that tends to coincide with the start of U.S. equities trading at 9:30 AM ET (13:30 UTC during daylight time or 14:30 UTC during standard time). You may wish to include additional PCAPs in the script, such as T13, to more precisely account for the observation of daylight saving time.

The script defaults to using yesterday’s date and assumes it was a trading day. Databento usually provides PCAPs on weekends, partial trading days, and exchange holidays, as long as administrative or heartbeat messages are present. If run on a Monday or after a market holiday, PCAPs for the previous day may occasionally be unavailable.

To simulate a historical trading session, point the Nanoconda feed handler to the prepared PCAP file using the pcap-file option in your <Ingester> XML configuration.

Here's an example configuration file:

<Ingester cpu="1" pcap-file="/home/user/recordings/lastday.pcap" pcap-speed="realtime">
    <Highway cpu="2" intf-a="lo" intf-b="lo">
        <Lane mc-a="224.0.31.1:14310" mc-b="224.0.32.1:15310" type="inc" name="l1i" group="310"/>
        <Lane mc-a="224.0.31.5:14314" mc-b="224.0.32.5:15314" type="inc" name="l4i" group="314"/>
        <Lane mc-a="224.0.31.9:14318" mc-b="224.0.32.9:15318" type="inc" name="l8i" group="318"/>
    </Highway>
</Ingester>

• pcap-file points to /home/user/recordings/lastday.pcap
• pcap-speed="realtime" replays the session at original market speed

Once configured, launching the feed handler will simulate the selected session as if it were live. Launch it with:

nanoconda-cli -a cmemdp3 -f config/CME-mdp3-prod-pcap-310-314-318-inc-only.xml

For a full guide to running simulations with the Nanoconda feed handler, see docs.nanoconda.com/quickstart.

Visit our PCAPs page to learn more, view pricing, or contact our sales team. You can also check out our blog on the advantages of raw PCAP files.

Nanoconda provides low-latency direct market access (DMA) solutions, including execution gateways, pre-trade risk management, trading GUIs, and managed colocation - all at a fraction of the traditional cost. Our platform is designed for speed, flexibility, and human control, allowing firms to manage and launch trading strategies directly through a built-in GUI. Our mission is to make high-frequency trading simple, fast, and accessible to firms and traders of all sizes.