Source: https://databento.com/docs/api-historical/batch ## Batch downloads Batch downloads allow you to download flat files directly from within your portal. For more information, see [Streaming vs. batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md). ### Historical.batch.submit\_job Make a batch download job request for flat files. Once a request is submitted, our system processes the request and prepares the batch files in the background. The status of your request and the files can be accessed from the [Download center]($PLATFORM_HOSTNAME/portal/download-center) from your user portal. This method takes longer than a streaming request, but is advantageous for larger requests as it supports delivery mechanisms that allow multiple accesses of the data without additional cost for each subsequent download after the first. Related: [batch.list\_jobs](https://databento.com/docs/api-historical/batch/batch-list-jobs.md), [batch.get\_job\_details](https://databento.com/docs/api-historical/batch/batch-get-job-details.md). #### API method ```python Historical.batch.submit_job( dataset: Dataset | str, symbols: Iterable[str | int] | str | int, schema: Schema | str, start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, encoding: Encoding | str = "dbn", compression: Compression | str = "zstd", pretty_px: bool = False, pretty_ts: bool = False, map_symbols: bool | None = None, split_symbols: bool = False, split_duration: Duration | str = "day", split_size: int | None = None, delivery: Delivery | str = "download", stype_in: SType | str = "raw_symbol", stype_out: SType | str = "instrument_id", limit: int | None = None, ) -> dict[str, Any] ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `symbols` | Iterable[str \ | int] or str or int | required | The product symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` then will select **all** symbols. | | `schema` | Schema or str | required | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | | `encoding` | Encoding or str | optional | The data [encoding](https://databento.com/docs/api-historical/basics/encodings.md). Must be one of 'dbn', 'csv', 'json'. For fastest transfer speed, 'dbn' is recommended. | | `compression` | Compression or str | optional | The data [compression](https://databento.com/docs/api-historical/basics/compression.md) mode. Must be either 'zstd', 'none', or None. For fastest transfer speed, 'zstd' is recommended. | | `pretty_px` | bool, default False | optional | If prices should be formatted to the correct scale (using the fixed-precision scalar 1e-9). Only applicable for 'csv' or 'json' encodings. | | `pretty_ts` | bool, default False | optional | If timestamps should be formatted as ISO 8601 strings. Only applicable for 'csv' or 'json' encodings. | | `map_symbols` | bool | optional | If a symbol field should be included with each text-encoded record. If `None`, will default to `True` for `csv` and `json` encodings and `False` for `dbn`. | | `split_symbols` | bool, default False | optional | If files should be split by raw symbol. Cannot be requested with `'ALL_SYMBOLS'`. Cannot be used with `limit`. | | `split_duration` | Duration or str, default 'day' | optional | The maximum time duration before batched data is split into multiple files. Must be one of 'day', 'week', 'month', 'year', or 'none'. A week starts on Sunday UTC. | | `split_size` | int | optional | The maximum size (in bytes) of each batched data file before being split. Must be an integer between 1e9 and 10e9 inclusive (1GB - 10GB). Defaults to no split size. | | `delivery` | Delivery or str, default 'download' | optional | The delivery mechanism for the batched data files once processed. Only 'download' is supported at this time. | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. | | `stype_out` | SType or str, default 'instrument\_id' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of output symbols. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. Must be a valid symbology combination with `stype_in`. See [symbology combinations](https://databento.com/docs/conventions/symbology.md#supported-symbology-combinations). | | `limit` | int | optional | The maximum number of records to return. If `None` then no limit. Cannot be used with `split_symbols`. | #### Returns `dict[str, Any]` The description of the submitted batch job. | Field | Type | Description | | `id` | str | The unique job ID for the request. | | `user_id` | str | The user ID of the user who made the request. | | `api_key` | str or None | The API key name for the request (if Basic Auth was used). | | `cost_usd` | float or None | The cost of the job in US dollars (`None` until the job is done processing). | | `dataset` | str | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). | | `symbols` | str | The list of symbols specified in the request. | | `stype_in` | str | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. | | `stype_out` | str | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of output symbols. | | `schema` | str | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). | | `start` | str | The ISO 8601 timestamp start of request time range (inclusive). | | `end` | str | The ISO 8601 timestamp end of request time range (exclusive). | | `limit` | int or None | The maximum number of records to return. | | `encoding` | str | The data [encoding](https://databento.com/docs/api-historical/basics/encodings.md). | | `compression` | str | The data [compression](https://databento.com/docs/api-historical/basics/compression.md) mode. | | `pretty_px` | bool | If prices are formatted to the correct scale (using the fixed-precision scalar 1e-9). | | `pretty_ts` | bool | If timestamps are formatted as ISO 8601 strings. | | `map_symbols` | bool | If a symbol field is included with each text-encoded record. | | `split_symbols` | bool | If files are split by raw symbol. | | `split_duration` | str | The maximum time interval for an individual file before splitting into multiple files. | | `split_size` | int or None | The maximum size for an individual file before splitting into multiple files. | | `packaging` | str or None | The packaging method of the batch data, one of 'none', 'zip', or 'tar' (deprecated). | | `delivery` | str | The delivery mechanism of the batch data. Only 'download' is supported at this time. | | `record_count` | int or None | The number of data records (`None` until the job is processed). | | `billed_size` | int or None | The size of the raw binary data used to process the batch job (used for billing purposes). | | `actual_size` | int or None | The total size of the result of the batch job after splitting and compression. | | `package_size` | int or None | The total size of the result of the batch job after any packaging (including metadata). | | `state` | str | The current status of the batch job. One of 'queued', 'processing', 'done', or 'expired'. | | `ts_received` | str | The ISO 8601 timestamp when Databento received the batch job. | | `ts_queued` | str or None | The ISO 8601 timestamp when the batch job was queued. | | `ts_process_start` | str or None | The ISO 8601 timestamp when the batch job began processing (if it's begun). | | `ts_process_done` | str or None | The ISO 8601 timestamp when the batch job finished processing (if it's finished). | | `ts_expiration` | str or None | The ISO 8601 timestamp when the batch job will expire from the [Download center](https://databento.com/docs/portal/download-center.md). | | `progress` | int or None | The progress percentage (0-100). Always `None` in submit responses. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") details = client.batch.submit_job( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", encoding="dbn", start="2022-06-06T12:00:00+00:00", end="2022-06-10T00:00:00+00:00", ) print(details) ``` #### Example response ```plaintext { "id": "GLBX-20221217-MN5S5S4WAS", "user_id": "NBPDLF33", "api_key": "prod-001", "cost_usd": None, "dataset": "GLBX.MDP3", "symbols": "ESM2", "stype_in": "raw_symbol", "stype_out": "instrument_id", "schema": "trades", "start": "2022-06-06T12:00:00.000000000Z", "end": "2022-06-10T00:00:00.000000000Z", "limit": None, "encoding": "dbn", "compression": "zstd", "pretty_px": False, "pretty_ts": False, "map_symbols": False, "split_symbols": False, "split_duration": "day", "split_size": None, "packaging": None, "delivery": "download", "record_count": None, "billed_size": None, "actual_size": None, "package_size": None, "state": "queued", "ts_received": "2022-12-17T00:36:37.844913000Z", "ts_queued": None, "ts_process_start": None, "ts_process_done": None, "ts_expiration": None } ``` ### Historical.batch.list\_jobs > **Info** > > The response format of this endpoint was changed on 2026-09-29. > Users can obtain the deprecated response format by specifying `short=False` until 2026-10-27. > For information about this change, see this [announcement](https://databento.com/blog/batch-api-changes-2026-08). List batch job details for the user account. The job details will be sorted in order of `ts_received`. To obtain the full details for a job, use [batch.get\_job\_details](https://databento.com/docs/api-historical/batch/batch-get-job-details.md), which returns the full set of fields for a single job. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.list_jobs( states: Iterable[JobState | str] | JobState | str | None = "queued,processing,done", since: pd.Timestamp | datetime | date | str | int | None = None, short: bool | None = None, ) -> list[dict[str, Any]] ``` #### Parameters | Parameter | Type | Required | Description | | `states` | Iterable[JobState \ | str] or JobState or str | optional | The filter for job states as a list of comma separated values. Can include 'queued', 'processing', 'done', and 'expired'. Defaults to all except 'expired'. | | `since` | pd.Timestamp, datetime, date, str, or int | optional | The filter for timestamp submitted (will not include jobs prior to this). Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `short` | bool | optional | If False, return the deprecated long form response for each job, which includes all job fields reported by [batch.get\_job\_details](https://databento.com/docs/api-historical/batch/batch-get-job-details.md). Defaults to the short form. | #### Returns `list[dict[str, Any]]` A list of batch job details. | Field | Type | Description | | `id` | str | The unique job ID for the request. | | `state` | str | The current status of the batch job. One of 'queued', 'processing', 'done', or 'expired'. | | `ts_received` | str | The ISO 8601 timestamp when Databento received the batch job. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") jobs = client.batch.list_jobs( states=["queued", "processing", "done"], since="2022-06-01", ) print(jobs) ``` #### Example response ```plaintext [ { "id": "GLBX-20221126-DBVXWPJJQN", "state": "done", "ts_received": "2022-11-26T09:23:17.519708000Z" }, ... ``` ### Historical.batch.list\_files List files for a batch job. This will include all data files and [support files](https://databento.com/docs/portal/download-center.md#support-files). Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.list_files( job_id: str, ) -> list[dict[str, Any]] ``` #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job identifier. | #### Returns `list[dict[str, Any]]` The file details for the batch job. | Field | Type | Description | | `filename` | str | The file name. | | `size` | int | The size of the file in bytes. | | `hash` | str | The SHA256 hash of the file. | | `urls` | dict | A map of download protocol to URL. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") files = client.batch.list_files(job_id="GLBX-20220610-5DEFXVTMSM") print(files) ``` #### Example response ```plaintext [ { "filename": "metadata.json", "size": 1102, "hash": "sha256:0168d53e1705b69b1d6407f10bb3ab48aac492fa0f68f863cc9b092931cc67a7", "urls": { "https": "https://api.databento.com/v0/batch/download/46PCMCVF/GLBX-20230203-WF9WJYSCDU/metadata.json", "ftp": "ftp://ftp.databento.com/46PCMCVF/GLBX-20230203-WF9WJYSCDU/metadata.json", } }, { "filename": "glbx-mdp3-20220610.mbo.csv.zst", "size": 21832, "hash": "sha256:1218930af153b4953632216044ef87607afa467fc7ab7fbb1f031fceacf9d52a", "urls": { "https": "https://api.databento.com/v0/batch/download/46PCMCVF/GLBX-20230203-WF9WJYSCDU/glbx-mdp3-20220610.mbo.csv.zst", "ftp": "ftp://ftp.databento.com/46PCMCVF/GLBX-20230203-WF9WJYSCDU/glbx-mdp3-20220610.mbo.csv.zst", } } ] ``` ### Historical.batch.get\_job\_details Get the full job details of a batch job. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.get_job_details( job_id: str, ) -> dict[str, Any] ``` #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job ID. This can be found in the [download center](https://databento.com/portal/download-center). | #### Returns `dict[str, Any]` The batch job details. See [batch.submit\_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md#returns) for a detailed list of returned values. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") job_details = client.batch.get_job_details( job_id="GLBX-20221126-DBVXWPJJQN", ) print(job_details) ``` #### Example response ```plaintext { "id": "GLBX-20221126-DBVXWPJJQN", "user_id": "NBPDLF33", "api_key": "prod-001", "cost_usd": 23.6454, "dataset": "GLBX.MDP3", "symbols": "ZC.FUT,ES.FUT", "stype_in": "parent", "stype_out": "instrument_id", "schema": "mbo", "start": "2022-10-24T00:00:00.000000000Z", "end": "2022-11-24T00:00:00.000000000Z", "limit": None, "encoding": "csv", "compression": "zstd", "pretty_px": False, "pretty_ts": False, "map_symbols": False, "split_symbols": False, "split_duration": "day", "split_size": None, "packaging": None, "delivery": "download", "record_count": 412160224, "billed_size": 23080972544, "actual_size": 8144595219, "package_size": 8144628684, "state": "done", "ts_received": "2022-11-26T09:23:17.519708000Z", "ts_queued": "2022-12-03T14:34:57.897790000Z", "ts_process_start": "2022-12-03T14:35:00.495167000Z", "ts_process_done": "2022-12-03T14:48:15.710116000Z", "ts_expiration": "2023-01-02T14:48:15.710116000Z", "progress": 100 } ``` ### Historical.batch.download Download a batch job or a specific file to `{output_dir}/{job_id}/`. Will automatically generate any necessary directories if they do not already exist. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.download( job_id: str, output_dir: PathLike[str] | str | None = None, filename_to_download: str | None = None, keep_zip: bool = False, ) -> list[Path] ``` #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job identifier. | | `output_dir` | PathLike[str] or str | optional | The directory to download the file(s) to. If `None`, defaults to the current working directory. | | `filename_to_download` | str | optional | The specific file to download. If `None` then will download all files for the batch job. | | `keep_zip` | bool, default False | optional | If `True`, and `filename_to_download` is `None`, all job files will be saved as a .zip archive in the `output_dir`. | #### Returns `list[Path]` A list of paths to the downloaded files. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") # Download all files for the batch job client.batch.download( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", ) # Alternatively, you can download a specific file client.batch.download( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", filename_to_download="metadata.json", ) ``` ### Historical.batch.download\_async Asynchronously download a batch job or a specific file to `{output_dir}/{job_id}/`. Will automatically generate any necessary directories if they do not already exist. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.download_async( job_id: str, output_dir: PathLike[str] | str | None = None, filename_to_download: str | None = None, keep_zip: bool = False, ) -> Awaitable[list[Path]] ``` > **Info** > > This method is a [coroutine](https://docs.python.org/3/library/asyncio-task.html#coroutines) and must be used with an [await](https://docs.python.org/3/reference/expressions.html#await) expression. #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job identifier. | | `output_dir` | PathLike[str] or str | optional | The directory to download the file(s) to. If `None`, defaults to the current working directory. | | `filename_to_download` | str | optional | The specific file to download. If `None` then will download all files for the batch job. | | `keep_zip` | bool, default False | optional | If `True`, and `filename_to_download` is `None`, all job files will be saved as a .zip archive in the `output_dir`. | #### Returns `list[Path]` A list of paths to the downloaded files. #### Example usage ```python import asyncio import databento as db client = db.Historical("$YOUR_API_KEY") # Download all files for the batch job coro = client.batch.download_async( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", ) asyncio.run(coro) # Alternatively, you can download a specific file coro = client.batch.download_async( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", filename_to_download="metadata.json", ) asyncio.run(coro) ```