Language: HTTP Available languages: [Python](https://databento.com/docs/api-historical/basics.python.md) · [C++](https://databento.com/docs/api-historical/basics.cpp.md) · [Rust](https://databento.com/docs/api-historical/basics.rust.md) · [HTTP](https://databento.com/docs/api-historical/basics.http.md) Source: https://databento.com/docs/api-historical/basics?lang=http ## Basics ### Overview Our historical API has the following structure: - [Metadata](https://databento.com/docs/api-historical/metadata.http.md) provides information about the datasets themselves. - [Time series](https://databento.com/docs/api-historical/timeseries.http.md) provides all types of time series data. This includes subsampled data (second, minute, hour, daily aggregates), trades, top-of-book, order book deltas, order book snapshots, summary statistics, static data and macro indicators. We also provide properties of products such as expirations, tick sizes and symbols as time series data. - [Symbology](https://databento.com/docs/api-historical/symbology.http.md) provides methods that help find and resolve symbols across different symbology systems. - [Batch](https://databento.com/docs/api-historical/batch.http.md) provides a means of submitting and querying for details of batch download requests. ### Authentication Databento uses API keys to authenticate requests. You can view and manage your keys on the [API keys]($PLATFORM_HOSTNAME/portal/keys) page of our web platform. Each API key is a 32-character string starting with `db-`. > **Info** > > Our HTTP API authenticates requests using the HTTP basic auth scheme, which > takes a username and password. Use your API key as the username and leave > the password blank. #### Example usage ```sh curl -G 'https://hist.databento.com/v0/metadata.list_datasets' \ -u $YOUR_API_KEY: ``` Appending a colon (:) after `$YOUR_API_KEY` prevents `curl` from asking for a password. Related: [Securing your API keys](https://databento.com/docs/portal/api-keys.http.md#securing-your-api-keys). ### Schemas and conventions A schema is a data record format represented as a collection of different data fields. Our datasets support multiple schemas, such as order book, tick data, bar aggregates, and so on. You can get a dictionary describing the fields of each schema from our [List of market data schemas](https://databento.com/docs/schemas/whats-a-schema.http.md). You can get a list of all supported schemas for any given dataset using the [list_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.http.md) method. The same information can also be found on each dataset's detail page found through the [Explore]($PLATFORM_HOSTNAME) feature. The following table provides details about the data types and conventions used for various fields that you will commonly encounter in the data. | Name | Field | Description | | Dataset | `dataset` | A unique string name assigned to each dataset by Databento. Full list of datasets can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.http.md). | | Publisher ID | `publisher_id` | A unique 16-bit unsigned integer assigned to each publisher by Databento. Full list of publisher IDs can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.http.md). | | Instrument ID | `instrument_id` | A unique 32-bit unsigned integer assigned to each instrument by the venue. Information about instrument IDs for any given dataset can be found in the [symbology](https://databento.com/docs/api-historical/basics/symbology.http.md). | | Order ID | `order_id` | A unique 64-bit unsigned integer assigned to each order by the venue. | | Timestamp (event) | `ts_event` | The matching-engine-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp (receive) | `ts_recv` | The capture-server-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp delta (in) | `ts_in_delta` | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [timestamping](https://databento.com/docs/architecture/timestamping-guide.http.md) guide. | | Timestamp out | `ts_out` | The Databento gateway-sending timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). See [timestamping](https://databento.com/docs/architecture/timestamping-guide.http.md) guide. | | Price | `price` | The price expressed as signed integer where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | Book side | `side` | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified by the original source. | | Size | `size` | The order quantity. | | Flag | `flag` | A bit field indicating event end, message characteristics, and data quality. | | Action | `action` | The event type or order book operation. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, **T**rade, **F**ill, or **N**one. | | Sequence number | `sequence` | The original message sequence number from the venue. | ### Datasets Databento provides time series datasets for a variety of markets, sourced from different publishers. Our available datasets can be browsed through the [search]($PLATFORM_HOSTNAME) feature on our site. Each dataset is assigned a unique string identifier (dataset ID) in the form `PUBLISHER.DATASET`, such as `GLBX.MDP3`. For publishers that are also markets, we use standard four-character ISO 10383 [Market Identifier Codes (MIC)](https://www.iso20022.org/market-identifier-codes). Otherwise, Databento arbitrarily assigns a four-character identifier for the publisher. These dataset IDs are also found on the [Data catalog](https://databento.com/docs/portal/data-catalog.http.md) and [Download request](https://databento.com/docs/portal/batch-download.http.md) features of the Databento user portal. When a publisher provides multiple data products with different levels of granularity, Databento subscribes to the most-granular product. We then provide this dataset with [alternate schemas](https://databento.com/docs/schemas/whats-a-schema.http.md) to make it easy to work with the level of detail most appropriate for your application. More information about different types of venues and publishers is available in our [FAQs](https://databento.com/docs/faqs/venues-and-publishers.http.md). ### Symbology Databento's historical API supports several ways to select an instrument in a [dataset](https://databento.com/docs/api-historical/basics/datasets.http.md). An instrument is specified using a **symbol** and a **symbology type**, also referred to as an **stype**. The supported symbology types are: - **[Raw symbology](https://databento.com/docs/conventions/symbology.http.md#raw-symbol)** (`raw_symbol`) original string symbols used by the publisher in the source data. - **[Instrument ID symbology](https://databento.com/docs/conventions/symbology.http.md#instrument-id)** (`instrument_id`) unique numeric ID assigned to each instrument by the publisher. - **[Parent symbology](https://databento.com/docs/conventions/symbology.http.md#parent)** (`parent`) groups instruments related to the market for the same underlying. - **[Continuous contract symbology](https://databento.com/docs/conventions/symbology.http.md#continuous)** (`continuous`) proprietary symbology that specifies instruments based on certain systematic rules. When requesting data from our [timeseries.get_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.http.md) or [batch.submit_job](https://databento.com/docs/api-historical/batch/batch-submit-job.http.md) endpoints, an input and output symbology type can be specified. By default, our client libraries will use raw symbology for the input type and instrument ID symbology for the output type. Not all symbology types are supported for every dataset. The process of converting between one symbology type to another is called **symbology resolution**. This conversion can be done, for no cost, with the [symbology.resolve](https://databento.com/docs/api-historical/symbology/symbology-resolve.http.md) endpoint. For more about symbology at Databento, see our [Knowledge base](https://databento.com/docs/conventions/symbology.http.md). ### Encodings **DBN** Databento Binary Encoding ([DBN](https://databento.com/docs/conventions/databento-binary-encoding.http.md)) is an extremely fast message encoding and highly-compressible storage format for normalized market data. It includes a self-describing metadata header and adopts a binary format with zero-copy serialization. We recommend using our [Python](https://databento.com/docs/api-historical/helpers/dbn-store.http.md?lang=python), [C++](https://databento.com/docs/api-historical/helpers/dbn-store.http.md?lang=cpp), or [Rust client libraries](https://databento.com/docs/api-historical/helpers/dbn-store.http.md?lang=rust) to read DBN files locally. A [CLI tool](https://crates.io/crates/dbn-cli) is also available for converting DBN files to CSV, JSON, Apache Parquet, or Apache Arrow. **CSV** Comma-separated values (CSV) is a simple text file format for tabular data. CSVs can be easily opened with Excel, loaded into [pandas](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html) data frames, or parsed in C++. Our CSVs have one header line, followed by one record per line. Lines use UNIX-style `\n` separators. **JSON** [JavaScript Object Notation](https://www.json.org/json-en.html) (JSON) is a flexible text file format with broad language support and wide adoption across web apps. Our JSON files follow the [JSON lines specification](https://jsonlines.org/), where each line of the file is a JSON record. Lines use UNIX-style `\n` separators. ### Compression Databento provides options for compressing files from our API. Available compression formats depend on the [encoding](https://databento.com/docs/api-historical/basics/encodings.http.md) you select. **zstd** The `zstd` compression option uses the [Zstandard](http://facebook.github.io/zstd/) format. This option is available for all encodings, and is recommended for faster transfer speeds and smaller files. Read more about [working with Zstandard-compressed files](https://databento.com/docs/conventions/working-with-zstandard.http.md). **none** The `none` compression option disables compression entirely, resulting in significantly larger files. However, this can be useful for loading small CSV files directly into Excel. ### Dates and times Our API exposes several parameters that expect a timestamp. Databento supports several formats based on [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html). Timestamps can be specified as [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string or UNIX nanoseconds. - `yyyy-mm-dd`, e.g. `2022-02-28` (midnight UTC) - `yyyy-mm-ddTHH:MM`, e.g. `2022-02-28T23:50` - `yyyy-mm-ddTHH:MM:SS`, e.g. `2022-02-28T23:50:59` - `yyyy-mm-ddTHH:MM:SS.NNNNNNNNN`, e.g. `2022-02-28T23:50:59.123456789` - [UNIX nanoseconds](https://en.wikipedia.org/wiki/Unix_time), e.g. `1646110259123456789` Timezone specification is also supported. - `yyyy-mm-ddTHH:MMZ` - `yyyy-mm-ddTHH:MM±hh` - `yyyy-mm-ddTHH:MM±hhmm` - `yyyy-mm-ddTHH:MM±hh:mm` In our market data, we always use [UNIX nanoseconds](https://en.wikipedia.org/wiki/Unix_time) to encode timestamps as a single number. Our client libraries can help convert this into a human-readable format. **Bare dates** Some parameters require a bare date, without a time. In these cases, the date must be specified in `yyyy-mm-dd` format, e.g. `2022-02-28`. ### Errors Our historical API uses HTTP response codes to indicate the success or failure of an API request. On error, the HTTP body will also contain a JSON object with a human-readable error message in the `detail` field. - `2xx` indicates success. - `4xx` indicates an error with the client's request. - `5xx` indicates an error with Databento's servers. The full list of the response codes and associated causes is as follows: | Code | Message | Cause | | 200 | OK | Successful request. | | 206 | Partial Content | Successful request, with partially resolved symbols. | | 400 | Bad Request | Invalid request. Usually due to a missing, malformed or unsupported parameter. | | 401 | Unauthorized | Invalid username or API key. | | 402 | Payment Required | Issue with your account payment information. | | 403 | Forbidden | The API key has insufficient permissions to perform the request. | | 404 | Not Found | A resource is not found, or a requested symbol does not exist. | | 409 | Conflict | A resource already exists. | | 422 | Unprocessable Entity | The request is well formed, but we cannot or will not process the contained instructions. | | 429 | Too Many Requests | API rate limit exceeded. | | 500 | Internal Server Error | Unexpected condition encountered in our system. | | 503 | Service Unavailable | Data gateway is offline or overloaded. | | 504 | Gateway Timeout | Data gateway is available but other parts of our system are offline or overloaded. | #### Example usage ```sh curl -i 'https://hist.databento.com/v0/metadata.list_datasets' ``` #### Example response ```plaintext HTTP/2 401 date: Wed, 30 Nov 2022 17:59:08 GMT server: uvicorn www-authenticate: Basic content-length: 30 content-type: application/json {"detail":"Not authenticated"} ``` ### Warnings Our historical API uses the `X-Warning` response header to attach non-fatal warnings to a successful request. A warning means that your request was processed, but some aspect of the request or its result deserves attention. For example, one or more symbols may not have resolved, or the request may rely on behavior that is scheduled for removal. A warning does not change the HTTP status code of the response. The header is only sent with `2xx` responses, and is omitted entirely when there are no warnings. Failed requests are reported as [errors](https://databento.com/docs/api-historical/basics/errors.http.md) instead. #### Format The value of the `X-Warning` header is a JSON array of strings. Each string is a single warning in the form `: `, where `` is one of the following: | Category | Cause | | `Warning` | Something about the request or its result should be reviewed. | | `DeprecationWarning` | The request uses an endpoint, parameter, or behavior that will be removed in the future. | A single response may contain more than one warning. HTTP header names are case-insensitive, so HTTP/2 clients will typically report the header in lowercase as `x-warning`. #### Reading warnings To read warnings, check each response for the `X-Warning` header, parse its value as JSON, and split each element on the first `: ` to separate the category from the message. Messages are intended to be human-readable and may change between releases, so your application should act on the category rather than matching on the text of a message. We recommend logging every warning, and treating a `DeprecationWarning` as a prompt to update your integration before the date given in the message. #### Common warnings Warnings are currently returned in cases such as: - A [timeseries.get_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.http.md) request where one or more symbols did not resolve. These responses also use the `206 Partial Content` status code. - A request with a time range that includes one or more days whose [dataset condition](https://databento.com/docs/api-historical/metadata/metadata-get-dataset-condition.http.md) is not `available`. - A request that returns no data, or whose time range falls entirely inside a weekend. - A streaming request larger than 5 GB, where [batch download](https://databento.com/docs/api-historical/batch.http.md) is recommended instead. - An `mbo` or `definition` request whose time range does not begin at UTC midnight, so the snapshot records for that day are not included. - A request that uses a deprecated endpoint, parameter, or response format. This list is not exhaustive, and new warnings may be added at any time. Related: [Errors](https://databento.com/docs/api-historical/basics/errors.http.md). #### Example usage ```sh curl -i -G 'https://hist.databento.com/v0/metadata.get_record_count' \ -u $YOUR_API_KEY: \ -d dataset=GLBX.MDP3 \ -d symbols=ESM2 \ -d schema=mbo \ -d start='2022-06-06T12:00' \ -d end='2022-06-06T12:10' ``` #### Example response ```plaintext HTTP/2 200 date: Thu, 01 Oct 2026 14:32:11 GMT server: uvicorn content-length: 6 content-type: application/json x-warning: ["Warning: The request time range does not start at UTC midnight, which contains a synthetic snapshot of the full order book state. See: https://databento.com/docs/schemas-and-data-formats/mbo#snapshots"] 184621 ``` ### Rate limits Our historical API allows each team up to: - 100 concurrent connections. - 100 [time series](https://databento.com/docs/api-historical/timeseries.http.md) requests per second. - 100 [symbology](https://databento.com/docs/api-historical/symbology.http.md) requests per second. - 20 [metadata](https://databento.com/docs/api-historical/metadata.http.md) requests per second. - 20 [batch list jobs](https://databento.com/docs/api-historical/batch/batch-list-jobs.http.md) requests per second. - 20 [batch submit job](https://databento.com/docs/api-historical/batch/batch-submit-job.http.md) requests per minute. When a request exceeds a rate limit, a `BentoClientError` exception is raised with a 429 error code. **Retry-After** The Retry-After response header indicates how long the user should wait before retrying. If you find that your application has been rate-limited, you can retry after waiting for the time specified in the Retry-After header. If you are using our HTTP API through cURL, you may want to use the `sleep 1` command to wait 1 second before your next request. This code snippet works best for our current APIs with their rate limits. Future APIs may have different rate limits, and might require a different default time delay. ### Size limits There is no size limit for either stream or batch download requests. Batch download is more manageable for large datasets, so we recommend using batch download for requests over 5 GB. You can also manage the size of your request by splitting it into multiple, smaller requests. The historical API allows you to make stream and batch download requests with time ranges specified up to nanosecond resolution. You can also use the `limit` parameter in any request to limit the number of data records returned from the service. [Batch download](https://databento.com/docs/api-historical/batch.http.md) supports different delivery methods which can be specified using the `delivery` parameter. #### Example usage ```sh curl -X POST -G 'https://hist.databento.com/v0/batch.submit_job' \ -u $YOUR_API_KEY: \ -d dataset=GLBX.MDP3 \ -d schema=trades \ -d start='2022-06-06T00:00' \ -d end='2022-06-10T00:10' \ -d limit=10000 ``` ### Metered pricing Databento only charges for the data that you use. You can find rates (per GB) for the various datasets and estimate pricing on our [Data catalog]($PLATFORM_HOSTNAME/portal/browse). We meter the data by its uncompressed size in [binary encoding](https://databento.com/docs/conventions/databento-binary-encoding.http.md). When you stream the data, you are billed incrementally for each outbound byte of data sent from our historical gateway. If your connection is interrupted while streaming our data and our historical gateway detects connection timeout over 5 seconds, it will immediately stop sending data and you will not be billed for the remainder of your request. Duplicate streaming requests will incur repeated charges. If you intend to access the same data multiple times, we recommend using our batch download feature. When you make a batch download request, you are only billed once for the request and, subsequently, you can download the data from the [Download center]($PLATFORM_HOSTNAME/portal/download-center) multiple times over 30 days for no additional charge. You will only be billed for usage of [time series](https://databento.com/docs/api-historical/timeseries.http.md) data. Access to metadata, symbology, and account management is free. The [metadata.get_cost](https://databento.com/docs/api-historical/metadata/metadata-get-cost.http.md) endpoint can be used to determine cost before you request any data. Related: [Billing management](https://databento.com/docs/portal/billing.http.md). ### Versioning Our historical API and its client libraries adopt `MAJOR.MINOR.PATCH` format for version numbers. These version numbers conform to [semantic versioning](https://semver.org). We are using major version `0` for initial development where our API is not considered stable. Once we release major version `1`, our public API will be stable. This means that you will be able to upgrade minor or patch versions to pick up new functionality, without breaking your integration. Starting with major versions after `1`, we will provide support for previous versions for one year after the date of the subsequent major release. For example, if version `2.0.0` is released on January 1, 2024, then all versions `1.x.y` of the API and client libraries will be deprecated. However, they will remain supported until January 1, 2025. We may introduce backwards-compatible changes between minor versions in the form of: - New data [encodings](https://databento.com/docs/api-historical/basics/encodings.http.md) - Additional fields to existing data [schemas](https://databento.com/docs/api-historical/basics/schemas-and-conventions.http.md) - Additional batch download [customizations](https://databento.com/docs/api-historical/batch.http.md) Our [Release notes](https://databento.com/docs/release-notes/release-notes-http.http.md) will contain information about both breaking and backwards-compatible changes in each release. Related: [Release notes](https://databento.com/docs/release-notes/release-notes-http.http.md).