Zstandard (zstd)

Zstandard is a fast, lossless compression algorithm that offers high compression ratios, making it suitable for storing and transmitting large amounts of data.

This compression algorithm is recommended by Databento for all historical streaming and batch downloads, as it minimizes the amount of data transmitted over the network and stored on file systems. You can select Zstandard compression by using the zstd option through the client libraries or the HTTP API.

Once you have obtained Zstandard compressed data from Databento, you have a number of options for decompressing:

• Databento client libraries. Decompression from Zstandard is handled internally. Refer to the API reference for usage details
• dbn-cli. The Databento CLI tool for working with DBN data also includes support for decompressing .zst files
• 7-Zip. To decompress using 7-Zip for Windows, you'll need to install the 7-Zip Zstandard edition. Then right-click on the .zst file, navigate to the 7-Zip menu, and select 'Extract here' or 'Extract to <folder>'
• zstd CLI. Used directly from the command line (described below), or via your own bash scripts

Installing dbn-cli

To install the dbn-cli (dbn) library, ensure you have Cargo (the Rust package manager) installed. Then, run the following command:

cargo install dbn-cli

For more details, visit dbn-cli on crates.io.

Installing Zstandard

The Zstandard (zstd) library can be installed on most operating systems, as detailed below:

macOS

brew install zstd

Linux

Debian/Ubuntu:

sudo apt-get update
sudo apt-get install zstd

CentOS/RHEL:

sudo dnf install zstd

Arch:

sudo pacman -S zstd

Windows

Using Chocolatey:

choco install zstd

Alternatively, you can also download the Zstandard binaries from the official releases page on GitHub and add the directory to your PATH. You can also install from source by following the instructions in the Zstandard GitHub repository.

Verify your installation by checking the version of Zstandard:

zstd --version

Decompressing

Once Zstandard is installed on your machine, you can utilize its command line interface (CLI) for various operations, including decompression and recompression. Below, we'll explore commands specifically for decompressing Zstandard-compressed files in different scenarios.

To view all available options in the Zstandard CLI, you can run the help command:

zstd -h

Decompress to a file

To decompress a file that has been compressed with Zstandard, use the -d (or --decompress) option followed by the filename. For example, if you have a file named data.zst, you can decompress it with the following command:

zstd -d data.zst

This will create a decompressed file named data in the same directory.

Tip

You can decompress a Zstandard-compressed Databento batched data file by running the following command:

zstd -d glbx-mdp3-20231201.trades.csv.zst

This will result in the decompressed file, glbx-mdp3-20231201.trades.csv, being created in the current directory.

If you want to decompress the file to a specific output file, you can use the -o option:

zstd -d data.zst -o decompressed_data.txt

This command will decompress data.zst into a file named decompressed_data.txt.

By default, zstd will not overwrite existing files. If you need to decompress and overwrite any existing files, you can use the -f (force) option:

zstd -d -f data.zst

This will decompress data.zst and overwrite any existing file with the same name as the output.

Decompressing multiple files

You can also decompress multiple files at once by specifying multiple filenames:

zstd -d file1.zst file2.zst file3.zst

Using wildcards for batch decompression

In a directory with multiple .zst files, you can decompress all of them using a wildcard (*). This is particularly useful for batch processing:

zstd -d *.zst

This command will decompress all files in the current directory with the .zst extension.

Tip

Running this command in a batch download directory will decompress all .zst files at once.

Decompressing to standard output

If you want to decompress a file and output the contents directly to the terminal (standard output), use the --stdout option:

zstd -d --stdout data.zst

This could be useful for piping the decompressed output to other programs in bash scripts.