Source: https://databento.com/docs/conventions/working-with-zstandard


# Zstandard (zstd)

Zstandard is a fast, lossless compression algorithm that offers high compression ratios,
making it suitable for storing and transmitting large amounts of data.

This compression algorithm is recommended by Databento for all historical streaming and batch downloads, as it
minimizes the amount of data transmitted over the network and stored on file systems.
You can select Zstandard compression by using the `zstd` option through the client libraries or the HTTP API.

Once you have obtained Zstandard compressed data from Databento, you have a number of options for
decompressing:
- **Databento client libraries.** Decompression from Zstandard is handled internally. Refer to the [API reference](https://databento.com/docs/api-historical.md) for usage details
- **dbn-cli.** The Databento CLI tool for working with DBN data also includes support for decompressing `.zst` files. See [Installing dbn-cli](https://databento.com/docs/conventions/databento-binary-encoding.md#installing-dbn-cli) for setup instructions
- **7-Zip.** To decompress using [7-Zip](https://www.7-zip.org/) for Windows, you'll need to install the [7-Zip Zstandard edition](https://github.com/mcmilk/7-Zip-zstd). Then right-click on the `.zst` file, navigate to the 7-Zip menu, and select 'Extract here' or 'Extract to \<folder\>'
- **zstd CLI.** Used directly from the command line (described below), or via your own bash scripts

### Installing Zstandard

The Zstandard (`zstd`) library can be installed on most operating systems, as detailed below:

**macOS**

    brew install zstd

**Linux**

Debian/Ubuntu:

    sudo apt-get update
    sudo apt-get install zstd

CentOS/RHEL:

    sudo dnf install zstd

Arch:

    sudo pacman -S zstd

**Windows**

Using Chocolatey:

    choco install zstd

Alternatively, you can also download the Zstandard binaries from the [official releases page on GitHub](https://github.com/facebook/zstd/releases) and add the directory to your `PATH`.
You can also install from source by following the instructions in the [Zstandard GitHub repository](https://github.com/facebook/zstd).

Verify your installation by checking the version of Zstandard:

    zstd --version

### Decompressing

Once Zstandard is installed on your machine, you can utilize its command line interface (CLI) for
various operations, including decompression and recompression. Below, we'll explore commands
specifically for decompressing Zstandard-compressed files in different scenarios.

To view all available options in the Zstandard CLI, you can run the help command:

    zstd -h

**Decompress to a file**

To decompress a file that has been compressed with Zstandard, use the `-d` (or `--decompress`)
option followed by the filename. For example, if you have a file named `data.zst`, you can
decompress it with the following command:

    zstd -d data.zst

This will create a decompressed file named data in the same directory.

> **Tip**
>
> You can decompress a Zstandard-compressed Databento batched data file by running the following command:
>
> `zstd -d glbx-mdp3-20231201.trades.csv.zst`
>
> This will result in the decompressed file, `glbx-mdp3-20231201.trades.csv`, being created in the current directory.

If you want to decompress the file to a specific output file, you can use the `-o` option:

    zstd -d data.zst -o decompressed_data.txt

This command will decompress `data.zst` into a file named `decompressed_data.txt`.

By default, `zstd` will *not* overwrite existing files. If you need to decompress and overwrite any
existing files, you can use the `-f` (force) option:

    zstd -d -f data.zst

This will decompress data.zst and overwrite any existing file with the same name as the output.

**Decompressing multiple files**

You can also decompress multiple files at once by specifying multiple filenames:

    zstd -d file1.zst file2.zst file3.zst

**Using wildcards for batch decompression**

In a directory with multiple `.zst` files, you can decompress all of them using a wildcard (`*`).
This is particularly useful for batch processing:

    zstd -d *.zst

This command will decompress all files in the current directory with the `.zst` extension.

> **Tip**
>
> Running this command in a batch download directory will decompress all .zst files at once.

**Decompressing to standard output**

If you want to decompress a file and output the contents directly to the terminal (standard output),
use the `--stdout` option:

    zstd -d --stdout data.zst

This could be useful for piping the decompressed output to other programs in bash scripts.