Batch download

Overview

After you've selected a product from a dataset, you can customize and submit your download request.

Info

Within one request, you can download multiple products from the same dataset — with no limit to the number of products. However, you can't download various products from different datasets within one request.

Data selection

• Schema. Specify the schema you want to receive your data in — MBO, MBP, TBBO, etc. You can read more about our supported schemas here. After you select a schema, we specify the usage rate ($/GB).
• Time range. You can specify your data time range. After you select a time range, we specify the data usage amount.

Basic customization

Choose your preferred encoding, compression, and delivery method for no additional charge.

• Encoding. DBN has the fastest transfer speed and smallest size. CSV works best for spreadsheets and JSON for web applications. Learn more about how to read DBN files here.
• Compression. Choose between zstd or no compression option. Learn more about how to decompress zstd files here.
• Delivery method. Direct download (HTTP or FTP) is currently the only option available, with S3 and disk coming soon.

Advanced customization

Split your data into different files based on the following:

• Duration. Split data into one file per day, week, or month.
• Size limit. Split files above a size limit of 2, 4, or 10GB.
• Instrument. Split files by instrument.

The following are additional customizations available for CSV and JSON encodings:

• Price format. Scale prices to decimal strings.
• Timestamp format. Format timestamps as ISO 8601 strings.
• Symbol field. Add a symbol field to the data.

After you've customized your download request details, submit the request by clicking the Submit request button on the right side of the page.

We will queue and process your request, and the files will be ready for use on your Databento portal's Download center page. You can read more about accessing and managing downloads in our Download center documentation.

Support files

Each completed batch job package includes the following four supporting JSON files:

• condition.json
• manifest.json
• metadata.json
• symbology.json

For CSV encoding, an additional symbology.csv is also included. Some of these files facilitate the processing of the batch job and are provided for your convenience.

• condition.json: This file outlines the dataset condition per date and aids in identifying the appropriate data for processing, including any missing or pending days. The same information can be obtained by requesting from the metadata.get_dataset_condition endpoint.

• manifest.json: This file lists all the batched data and support files contained in the package, including file names, sizes and SHA-256 hashes. You can use this information to confirm that your data is complete and free of tampering or corruption. The same information can be obtained by requesting from the batch.list_files endpoint.

• metadata.json: This file describes the original request for the batch download, including any advanced customization options. The same information can be obtained by requesting from the batch.list_jobs endpoint.

• symbology.json: This file provides symbology mappings for the batched data. This allows you to interpret the integer instrument IDs and map these to their corresponding raw symbols across date intervals. The same information can be obtained by requesting from the symbology.resolve endpoint.

• symbology.csv: This additional file is provided as an alternative representation of the data contained in the symbology.json when CSV encoding is specified. It presents the data in a more accessible format for use with spreadsheets, by flattening into a row per symbol and date within each symbology mapping interval.