Source: https://databento.com/docs/api-historical/basics ## Basics ### Overview Our historical API has the following structure: - [Metadata](https://databento.com/docs/api-historical/metadata.md) provides information about the datasets themselves. - [Time series](https://databento.com/docs/api-historical/timeseries.md) provides all types of time series data. This includes subsampled data (second, minute, hour, daily aggregates), trades, top-of-book, order book deltas, order book snapshots, summary statistics, static data and macro indicators. We also provide properties of products such as expirations, tick sizes and symbols as time series data. - [Symbology](https://databento.com/docs/api-historical/symbology.md) provides methods that help find and resolve symbols across different symbology systems. - [Batch](https://databento.com/docs/api-historical/batch.md) provides a means of submitting and querying for details of batch download requests. ### Authentication Databento uses API keys to authenticate requests. You can view and manage your keys on the [API keys]($PLATFORM_HOSTNAME/portal/keys) page of your portal. Each API key is a 32-character string starting with `db-`. By default, our library uses the environment variable `DATABENTO_API_KEY` as your API key. However, if you pass an API key to the `Historical` constructor through the `key` parameter, then this value will be used instead. Related: [Securing your API keys](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). #### Example usage ```python import databento as db # Establish connection and authenticate client = db.Historical("$YOUR_API_KEY") # Authenticated request print(client.metadata.list_datasets()) ``` ### Schemas and conventions A schema is a data record format represented as a collection of different data fields. Our datasets support multiple schemas, such as order book, trades, bar aggregates, and so on. You can get a dictionary describing the fields of each schema from our [List of market data schemas](https://databento.com/docs/schemas/whats-a-schema.md). You can get a list of all supported schemas for any given dataset using the Historical client's [list_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md) method. The same information can also be found on the dataset details pages on the [user portal]($PLATFORM_HOSTNAME). The following table provides details about the data types and conventions used for various fields that you will commonly encounter in the data. | Name | Field | Description | | Dataset | `dataset` | A unique string name assigned to each dataset by Databento. Full list of datasets can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | Publisher ID | `publisher_id` | A unique 16-bit unsigned integer assigned to each publisher by Databento. Full list of publisher IDs can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md). | | Instrument ID | `instrument_id` | A unique 32-bit unsigned integer assigned to each instrument by the venue. Information about instrument IDs for any given dataset can be found in the [symbology](https://databento.com/docs/api-historical/symbology.md). | | Order ID | `order_id` | A unique 64-bit unsigned integer assigned to each order by the venue. | | Timestamp (event) | `ts_event` | The matching-engine-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp (receive) | `ts_recv` | The capture-server-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp delta (in) | `ts_in_delta` | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [timestamping](https://databento.com/docs/architecture/timestamping-guide.md) guide. | | Timestamp out | `ts_out` | The Databento gateway-sending timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). See [timestamping](https://databento.com/docs/architecture/timestamping-guide.md) guide. | | Price | `price` | The price expressed as signed integer where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | Book side | `side` | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified by the original source. | | Size | `size` | The order quantity. | | Flag | `flag` | A bit field indicating event end, message characteristics, and data quality. | | Action | `action` | The event type or order book operation. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, **T**rade, **F**ill, or **N**one. | | Sequence number | `sequence` | The original message sequence number from the venue. | ### Datasets Databento provides time series datasets for a variety of markets, sourced from different publishers. Our available datasets can be browsed through the [search]($PLATFORM_HOSTNAME) feature on our site. Each dataset is assigned a unique string identifier (dataset ID) in the form `PUBLISHER.DATASET`, such as `GLBX.MDP3`. For publishers that are also markets, we use standard four-character ISO 10383 [Market Identifier Codes (MIC)](https://www.iso20022.org/market-identifier-codes). Otherwise, Databento arbitrarily assigns a four-character identifier for the publisher. These dataset IDs are also found on the [Data catalog](https://databento.com/docs/portal/data-catalog.md) and [Download request](https://databento.com/docs/portal/batch-download.md) features of the Databento user portal. When a publisher provides multiple data products with different levels of granularity, Databento subscribes to the most-granular product. We then provide this dataset with [alternate schemas](https://databento.com/docs/schemas/whats-a-schema.md) to make it easy to work with the level of detail most appropriate for your application. More information about different types of venues and publishers is available in our [FAQs](https://databento.com/docs/faqs/venues-and-publishers.md). ### Symbology Databento's historical API supports several ways to select an instrument in a [dataset](https://databento.com/docs/api-historical/basics/datasets.md). An instrument is specified using a **symbol** and a **symbology type**, also referred to as an **stype**. The supported symbology types are: - **[Raw symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol)** (`raw_symbol`) original string symbols used by the publisher in the source data. - **[Instrument ID symbology](https://databento.com/docs/conventions/symbology.md#instrument-id)** (`instrument_id`) unique numeric ID assigned to each instrument by the publisher. - **[Parent symbology](https://databento.com/docs/conventions/symbology.md#parent)** (`parent`) groups instruments related to the market for the same underlying. - **[Continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous)** (`continuous`) proprietary symbology that specifies instruments based on certain systematic rules. When requesting data from our [timeseries.get_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md) or [batch.submit_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) endpoints, an input and output symbology type can be specified. By default, our client libraries will use raw symbology for the input type and instrument ID symbology for the output type. Not all symbology types are supported for every dataset. The process of converting between one symbology type to another is called **symbology resolution**. This conversion can be done, for no cost, with the [symbology.resolve](https://databento.com/docs/api-historical/symbology/symbology-resolve.md) endpoint. For more about symbology at Databento, see our [Standards and conventions](https://databento.com/docs/conventions/symbology.md). ### Encodings **DBN** Databento Binary Encoding ([DBN](https://databento.com/docs/conventions/databento-binary-encoding.md)) is an extremely fast message encoding and highly-compressible storage format for normalized market data. It includes a self-describing metadata header and adopts a binary format with zero-copy serialization. We recommend using our [Python](https://databento.com/docs/api-historical/helpers/dbn-store.md?lang=python), [C++](https://databento.com/docs/api-historical/helpers/dbn-store.md?lang=cpp), or [Rust client libraries](https://databento.com/docs/api-historical/helpers/dbn-store.md?lang=rust) to read DBN files locally. A [CLI tool](https://crates.io/crates/dbn-cli) is also available for converting DBN files to CSV or JSON. **CSV** Comma-separated values (CSV) is a simple text file format for tabular data, CSVs can be easily opened with Excel, loaded into [pandas](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html) data frames, or parsed in C++. Our CSVs have one header line, followed by one record per line. Lines use UNIX-style `\n` separators. **JSON** [JavaScript Object Notation](https://www.json.org/json-en.html) (JSON) is a flexible text file format with broad language support and wide adoption across web apps. Our JSON files follow the [JSON lines specification](https://jsonlines.org/), where each line of the file is a JSON record. Lines use UNIX-style `\n` separators. ### Compression Databento provides options for compressing files from our API. Available compression formats depend on the [encoding](https://databento.com/docs/api-historical/basics/encodings.md) you select. **Zstd** The `Zstd` compression option uses the [Zstandard](http://facebook.github.io/zstd/) format. This option is available for all encodings, and is recommended for faster transfer speeds and smaller files. You can read Zstandard files in Python using the [zstandard package](https://pypi.org/project/zstandard/). Read more about [working with Zstandard-compressed files](https://databento.com/docs/conventions/working-with-zstandard.md). **None** The `None` compression option disables compression entirely, resulting in significantly larger files. However, this can be useful for loading small CSV files directly into Excel. ### Dates and times Our Python client library has several functions with timestamp arguments. These arguments will have type `pandas.Timestamp | datetime.date | str | int` and support a variety of formats. It's recommended to use [pandas.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), which fully supports timezones and nanosecond-precision. If a `datetime.date` is used, the time is set to midnight UTC. If an `int` is provided, the value is interpreted as [UNIX nanoseconds](https://en.wikipedia.org/wiki/Unix_time). The client library also handles several string-based timestamp formats based on [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html). - `yyyy-mm-dd`, e.g. `"2022-02-28"` (midnight UTC) - `yyyy-mm-ddTHH:MM`, e.g. `"2022-02-28T23:50"` - `yyyy-mm-ddTHH:MM:SS`, e.g. `"2022-02-28T23:50:59"` - `yyyy-mm-ddTHH:MM:SS.NNNNNNNNN`, e.g. `"2022-02-28T23:50:59.123456789"` Timezone specification is also supported. - `yyyy-mm-ddTHH:MMZ` - `yyyy-mm-ddTHH:MM±hh` - `yyyy-mm-ddTHH:MM±hhmm` - `yyyy-mm-ddTHH:MM±hh:mm` **Bare dates** Some parameters require a bare date, without a time. These arguments have type `datetime.date | str` and must either be a `datetime.date` object, or a string in `yyyy-mm-dd` format, e.g. `"2022-02-28"`. ### Errors Our historical API uses HTTP response codes to indicate the success or failure of an API request. The client library provides exceptions that wrap these response codes. - `2xx` indicates success. - `4xx` indicates an error on the client side. Represented as a `BentoClientError`. - `5xx` indicates an error with Databento's servers. Represented as a `BentoServerError`. The full list of the response codes and associated causes is as follows: | Code | Message | Cause | | 200 | OK | Successful request. | | 206 | Partial Content | Successful request, with partially resolved symbols. | | 400 | Bad Request | Invalid request. Usually due to a missing, malformed or unsupported parameter. | | 401 | Unauthorized | Invalid username or API key. | | 402 | Payment Required | Issue with your account payment information. | | 403 | Forbidden | The API key has insufficient permissions to perform the request. | | 404 | Not Found | A resource is not found, or a requested symbol does not exist. | | 409 | Conflict | A resource already exists. | | 422 | Unprocessable Entity | The request is well formed, but we cannot or will not process the contained instructions. | | 429 | Too Many Requests | API rate limit exceeded. | | 500 | Internal Server Error | Unexpected condition encountered in our system. | | 503 | Service Unavailable | Data gateway is offline or overloaded. | | 504 | Gateway Timeout | Data gateway is available but other parts of our system are offline or overloaded. | #### API method ```python class databento.BentoError(Exception) class databento.BentoHttpError(databento.BentoError) class databento.BentoClientError(databento.BentoHttpError) class databento.BentoServerError(databento.BentoHttpError) ``` #### Example usage ```python import databento as db client = db.Historical("INVALID_API_KEY") try: print(client.metadata.list_datasets()) except db.BentoClientError as e: print(e) ``` #### Example response ```plaintext 400 auth_invalid_username_in_basic_auth Invalid username in Basic auth ('INVALID_API_KEY'). documentation: https://databento.com/docs ``` ### Rate limits Our historical API allows each IP address up to: - 100 concurrent connections. - 100 [time series](https://databento.com/docs/api-historical/timeseries.md) requests per second. - 100 [symbology](https://databento.com/docs/api-historical/symbology.md) requests per second. - 20 [metadata](https://databento.com/docs/api-historical/metadata.md) requests per second. - 20 [batch list jobs](https://databento.com/docs/api-historical/batch/batch-list-jobs.md) requests per second. - 20 [batch submit job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) requests per minute. When a request exceeds a rate limit, a `BentoClientError` exception is raised with a 429 error code. **Retry-After** The Retry-After response header indicates how long the user should wait before retrying. If you find that your application has been rate-limited, you can retry after waiting for the time specified in the Retry-After header. If you are using Python, you may use the time.sleep function as seen below to wait for the time specified in the Retry-After header. e.g. `time.sleep(int(response.headers("Retry-After", 1)))` This code snippet works best for our current APIs with their rate limits. Future APIs may have different rate limits, and might require a different default time delay. ### Size limits There is no size limit for either stream or batch download requests. Batch download is more manageable for large datasets, so we recommend using batch download for requests over 5 GB. You can also manage the size of your request by splitting it into multiple, smaller requests. The historical API allows you to make stream and batch download requests with time ranges specified up to nanosecond resolution. You can also use the `limit` parameter in any request to limit the number of data records returned from the service. [Batch download](https://databento.com/docs/api-historical/batch.md) supports different delivery methods which can be specified using the `delivery` parameter. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") job = client.batch.submit_job( dataset="GLBX.MDP3", symbols="CLZ7", schema="trades", start="2022-06-06T00:00:00", end="2022-06-10T00:10:00", limit=10000, ) ``` ### Metered pricing Databento only charges for the data that you use. You can find rates (per GB) for the various datasets and estimate pricing on our [Data catalog]($PLATFORM_HOSTNAME/portal/browse). We meter the data by its uncompressed size in [binary encoding](https://databento.com/docs/conventions/databento-binary-encoding.md). When you stream the data, you are billed incrementally for each outbound byte of data sent from our historical gateway. If your connection is interrupted while streaming our data and our historical gateway detects connection timeout over 5 seconds, it will immediately stop sending data and you will not be billed for the remainder of your request. Duplicate streaming requests will incur repeated charges. If you intend to access the same data multiple times, we recommend using our batch download feature. When you make a batch download request, you are only billed once for the request and, subsequently, you can download the data from the [Download center]($PLATFORM_HOSTNAME/portal/download-center) multiple times over 30 days for no additional charge. You will only be billed for usage of [time series](https://databento.com/docs/api-historical/timeseries.md) data. Access to metadata, symbology, and account management is free. The [Historical.metadata.get\_cost](https://databento.com/docs/api-historical/metadata/metadata-get-cost.md) method can be used to determine cost before you request any data. Related: [Billing management](https://databento.com/docs/portal/billing.md). ### Versioning Our historical and live APIs and their client libraries adopt `MAJOR.MINOR.PATCH` format for version numbers. These version numbers conform to [semantic versioning](https://semver.org). We are using major version `0` for initial development, where our API is not considered stable. Once we release major version `1`, our public API will be stable. This means that you will be able to upgrade minor or patch versions to pick up new functionality, without breaking your integration. Starting with major versions after `1`, we will provide support for previous versions for one year after the date of the subsequent major release. For example, if version `2.0.0` is released on January 1, 2024, then all versions `1.x.y` of the API and client libraries will be deprecated. However, they will remain supported until January 1, 2025. We may introduce backwards-compatible changes between minor versions in the form of: - New data [encodings](https://databento.com/docs/api-historical/basics/encodings.md) - Additional fields to existing data [schemas](https://databento.com/docs/api-historical/basics/schemas-and-conventions.md) - Additional batch download [customizations](https://databento.com/docs/api-historical/batch.md) Our [Release notes](https://databento.com/docs/release-notes/release-notes-python.md) will contain information about both breaking and backwards-compatible changes in each release. Our API and official client libraries are kept in sync with same-day releases for major versions. For instance, `1.x.y` of the C++ client library will use the same functionality found in any `1.x.y` version of the Python client. Related: [Release notes](https://databento.com/docs/release-notes/release-notes-python.md).