Historical.batch.submit_job

Make a batch download job request for flat files.

Once a request is submitted, our system processes the request and prepares the batch files in the background. The status of your request and the files can be accessed from the Download center from your user portal.

This method takes longer than a streaming request, but is advantageous for larger requests as it supports delivery mechanisms that allow multiple accesses of the data without additional cost for each subsequent download after the first.

Parameters

Parameter	Type	Required	Description
`dataset`	Dataset or str	required	The dataset code (string identifier). Must be one of the values from list_datasets.
`symbols`	Iterable[str \| int] or str or int	required	The product symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` or `None` then will select all symbols.
`schema`	Schema or str	required	The data record schema. Must be one of the values from list_schemas.
`start`	pd.Timestamp, datetime, date, str, or int	required	The inclusive start of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes pd.Timestamp, Python datetime, Python date, ISO 8601 string, or UNIX timestamp in nanoseconds. Assumes UTC as timezone unless otherwise specified.
`end`	pd.Timestamp, datetime, date, str, or int	optional	The exclusive end of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes pd.Timestamp, Python datetime, Python date, ISO 8601 string, or UNIX timestamp in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the forward filled value of `start` based on the resolution provided.
`encoding`	Encoding or str	optional	The data encoding. Must be one of 'dbn', 'csv', 'json'. For fastest transfer speed, 'dbn' is recommended.
`compression`	Compression or str	optional	The data compression mode. Must be either 'zstd', 'none', or None. For fastest transfer speed, 'zstd' is recommended.
`pretty_px`	bool, default False	optional	If prices should be formatted to the correct scale (using the fixed-precision scalar 1e-9). Only applicable for 'csv' or 'json' encodings.
`pretty_ts`	bool, default False	optional	If timestamps should be formatted as ISO 8601 strings. Only applicable for 'csv' or 'json' encodings.
`map_symbols`	bool	optional	If a symbol field should be included with each text-encoded record. If `None`, will default to `True` for `csv` and `json` encodings and `False` for `dbn`.
`split_symbols`	bool, default False	optional	If files should be split by raw symbol. Cannot be requested with `'ALL_SYMBOLS'`. Cannot be used with `limit`.
`split_duration`	Duration or str, default 'day'	optional	The maximum time duration before batched data is split into multiple files. Must be one of 'day', 'week', 'month', 'year', or 'none'. A week starts on Sunday UTC.
`split_size`	int	optional	The maximum size (in bytes) of each batched data file before being split. Must be an integer between 1e9 and 10e9 inclusive (1GB - 10GB). Defaults to no split size.
`delivery`	Delivery or str, default 'download'	optional	The delivery mechanism for the batched data files once processed. Only 'download' is supported at this time.
`stype_in`	SType or str, default 'raw_symbol'	optional	The symbology type of input `symbols`. Must be one of 'raw_symbol', 'instrument_id', 'parent', or 'continuous'.
`stype_out`	SType or str, default 'instrument_id'	optional	The symbology type of output symbols. Must be one of 'raw_symbol', 'instrument_id', 'parent', or 'continuous'. Must be a valid symbology combination with `stype_in`. See symbology combinations.
`limit`	int	optional	The maximum number of records to return. If `None` then no limit. Cannot be used with `split_symbols`.

Returns

dict[str, Any]

The description of the submitted batch job.

Field	Type	Description
`id`	str	The unique job ID for the request.
`user_id`	str	The user ID of the user who made the request.
`api_key`	str or None	The API key name for the request (if Basic Auth was used).
`cost_usd`	float or None	The cost of the job in US dollars (`None` until the job is done processing).
`dataset`	str	The dataset code (string identifier).
`symbols`	str	The list of symbols specified in the request.
`stype_in`	str	The symbology type of input `symbols`.
`stype_out`	str	The symbology type of output symbols.
`schema`	str	The data record schema.
`start`	str	The ISO 8601 timestamp start of request time range (inclusive).
`end`	str	The ISO 8601 timestamp end of request time range (exclusive).
`limit`	int or None	The maximum number of records to return.
`encoding`	str	The data encoding.
`compression`	str	The data compression mode.
`pretty_px`	bool	If prices are formatted to the correct scale (using the fixed-precision scalar 1e-9).
`pretty_ts`	bool	If timestamps are formatted as ISO 8601 strings.
`map_symbols`	bool	If a symbol field is included with each text-encoded record.
`split_symbols`	bool	If files are split by raw symbol.
`split_duration`	str	The maximum time interval for an individual file before splitting into multiple files.
`split_size`	int or None	The maximum size for an individual file before splitting into multiple files.
`packaging`	str or None	The packaging method of the batch data, one of 'none', 'zip', or 'tar' (deprecated).
`delivery`	str	The delivery mechanism of the batch data. Only 'download' is supported at this time.
`record_count`	int or None	The number of data records (`None` until the job is processed).
`billed_size`	int or None	The size of the raw binary data used to process the batch job (used for billing purposes).
`actual_size`	int or None	The total size of the result of the batch job after splitting and compression.
`package_size`	int or None	The total size of the result of the batch job after any packaging (including metadata).
`state`	str	The current status of the batch job. One of 'received', 'queued', 'processing', 'done', or 'expired'.
`ts_received`	str	The ISO 8601 timestamp when Databento received the batch job.
`ts_queued`	str or None	The ISO 8601 timestamp when the batch job was queued.
`ts_process_start`	str or None	The ISO 8601 timestamp when the batch job began processing (if it's begun).
`ts_process_done`	str or None	The ISO 8601 timestamp when the batch job finished processing (if it's finished).
`ts_expiration`	str or None	The ISO 8601 timestamp when the batch job will expire from the Download center.
`progress`	int or None	The progress percentage (0-100). Always `None` in submit responses.

API method

Historical.batch.submit_job(
    dataset: Dataset | str,
    symbols: Iterable[str | int] | str | int,
    schema: Schema | str,
    start: pd.Timestamp | datetime | date | str | int,
    end: pd.Timestamp | datetime | date | str | int | None = None,
    encoding: Encoding | str = "dbn",
    compression: Compression | str = "zstd",
    pretty_px: bool = False,
    pretty_ts: bool = False,
    map_symbols: bool | None = None,
    split_symbols: bool = False,
    split_duration: Duration | str = "day",
    split_size: int | None = None,
    delivery: Delivery | str = "download",
    stype_in: SType | str = "raw_symbol",
    stype_out: SType | str = "instrument_id",
    limit: int | None = None,
) -> dict[str, Any]

Example usage

import databento as db

client = db.Historical("$YOUR_API_KEY")

details = client.batch.submit_job(
    dataset="GLBX.MDP3",
    symbols=["ESM2"],
    schema="trades",
    encoding="dbn",
    start="2022-06-06T12:00:00+00:00",
    end="2022-06-10T00:00:00+00:00",
)
print(details)

Example response

{
    "id": "GLBX-20221217-MN5S5S4WAS",
    "user_id": "NBPDLF33",
    "api_key": "prod-001",
    "cost_usd": None,
    "dataset": "GLBX.MDP3",
    "symbols": "ESM2",
    "stype_in": "raw_symbol",
    "stype_out": "instrument_id",
    "schema": "trades",
    "start": "2022-06-06T12:00:00.000000000Z",
    "end": "2022-06-10T00:00:00.000000000Z",
    "limit": None,
    "encoding": "dbn",
    "compression": "zstd",
    "pretty_px": False,
    "pretty_ts": False,
    "map_symbols": False,
    "split_symbols": False,
    "split_duration": "day",
    "split_size": None,
    "packaging": None,
    "delivery": "download",
    "record_count": None,
    "billed_size": None,
    "actual_size": None,
    "package_size": None,
    "state": "queued",
    "ts_received": "2022-12-17T00:36:37.844913000Z",
    "ts_queued": None,
    "ts_process_start": None,
    "ts_process_done": None,
    "ts_expiration": None
}