Basics
Overview
Our historical API has the following structure:
- Metadata provides information about the datasets themselves.
- Time series provides all types of time series data. This includes subsampled data (second, minute, hour, daily aggregates), trades, top-of-book, order book deltas, order book snapshots, summary statistics, static data and macro indicators. We also provide properties of products such as expirations, tick sizes and symbols as time series data.
- Symbology provides methods that help find and resolve symbols across different symbology systems.
- Batch provides a means of submitting and querying for details of batch download requests.
Authentication
Databento uses API keys to authenticate requests. You can view and manage your keys on the API keys page of your portal.
Each API key is a 32-character string starting with db-. By default, our library uses the environment variable DATABENTO_API_KEY as your API key. However, if you pass an API key to the Historical constructor through the key parameter, then this value will be used instead.
Related: Securing your API keys.
1 2 3 4 5 6 7
import databento as db
# Establish connection and authenticate
client = db.Historical("$YOUR_API_KEY")
# Authenticated request
print(client.metadata.list_datasets()) Schemas and conventions
A schema is a data record format represented as a collection of different data fields. Our datasets support multiple schemas, such as order book, trades, bar aggregates, and so on. You can get a dictionary describing the fields of each schema from our List of market data schemas.
You can get a list of all supported schemas for any given dataset using the Historical client's list_schemas method. The same information can also be found on the dataset details pages on the user portal.
The following table provides details about the data types and conventions used for various fields that you will commonly encounter in the data.
| Name | Field | Description |
|---|---|---|
| Dataset | dataset | A unique string name assigned to each dataset by Databento. Full list of datasets can be found from the metadata. |
| Publisher ID | publisher_id | A unique 16-bit unsigned integer assigned to each publisher by Databento. Full list of publisher IDs can be found from the metadata. |
| Instrument ID | instrument_id | A unique 32-bit unsigned integer assigned to each instrument by the venue. Information about instrument IDs for any given dataset can be found in the symbology. |
| Order ID | order_id | A unique 64-bit unsigned integer assigned to each order by the venue. |
| Timestamp (event) | ts_event | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. |
| Timestamp (receive) | ts_recv | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. |
| Timestamp delta (in) | ts_in_delta | The matching-engine-sending timestamp expressed as the number of nanoseconds before ts_recv. See timestamping guide. |
| Timestamp out | ts_out | The Databento gateway-sending timestamp expressed as the number of nanoseconds since the UNIX epoch. See timestamping guide. |
| Price | price | The price expressed as signed integer where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. |
| Book side | side | The side that initiates the event. Can be Ask for a sell order (or sell aggressor in a trade), Bid for a buy order (or buy aggressor in a trade), or None where no side is specified by the original source. |
| Size | size | The order quantity. |
| Flag | flag | A bit field indicating event end, message characteristics, and data quality. |
| Action | action | The event type or order book operation. Can be Add, Cancel, Modify, cleaR book, Trade, Fill, or None. |
| Sequence number | sequence | The original message sequence number from the venue. |
Datasets
Databento provides time series datasets for a variety of markets, sourced from different publishers. Our available datasets can be browsed through the search feature on our site.
Each dataset is assigned a unique string identifier (dataset ID) in the form PUBLISHER.DATASET, such as GLBX.MDP3. For publishers that are also markets, we use standard four-character ISO 10383 Market Identifier Codes (MIC). Otherwise, Databento arbitrarily assigns a four-character identifier for the publisher.
These dataset IDs are also found on the Data catalog and Download request features of the Databento user portal.
When a publisher provides multiple data products with different levels of granularity, Databento subscribes to the most-granular product. We then provide this dataset with alternate schemas to make it easy to work with the level of detail most appropriate for your application.
More information about different types of venues and publishers is available in our FAQs.
Symbology
Databento's historical API supports several ways to select an instrument in a dataset. An instrument is specified using a symbol and a symbology type, also referred to as an stype. The supported symbology types are:
- Raw symbology (
raw_symbol) original string symbols used by the publisher in the source data. - Instrument ID symbology (
instrument_id) unique numeric ID assigned to each instrument by the publisher. - Parent symbology (
parent) groups instruments related to the market for the same underlying. - Continuous contract symbology (
continuous) proprietary symbology that specifies instruments based on certain systematic rules.
When requesting data from our timeseries.get_range or batch.submit_job endpoints, an input and output symbology type can be specified. By default, our client libraries will use raw symbology for the input type and instrument ID symbology for the output type. Not all symbology types are supported for every dataset.
The process of converting between one symbology type to another is called symbology resolution. This conversion can be done, for no cost, with the symbology.resolve endpoint.
For more about symbology at Databento, see our Standards and conventions.
Encodings
DBN
Databento Binary Encoding (DBN) is an extremely fast message encoding and highly-compressible storage format for normalized market data. It includes a self-describing metadata header and adopts a binary format with zero-copy serialization.
We recommend using our Python, C++, or Rust client libraries to read DBN files locally. A CLI tool is also available for converting DBN files to CSV or JSON.
CSV
Comma-separated values (CSV) is a simple text file format for tabular data, CSVs can be easily opened with Excel, loaded into pandas data frames, or parsed in C++.
Our CSVs have one header line, followed by one record per line. Lines use UNIX-style \n separators.
JSON
JavaScript Object Notation (JSON) is a flexible text file format with broad language support and wide adoption across web apps.
Our JSON files follow the JSON lines specification, where each line of the file is a JSON record. Lines use UNIX-style \n separators.
Compression
Databento provides options for compressing files from our API. Available compression formats depend on the encoding you select.
Zstd
The Zstd compression option uses the Zstandard format.
This option is available for all encodings, and is recommended for faster transfer speeds and smaller files.
You can read Zstandard files in Python using the zstandard package.
Read more about working with Zstandard-compressed files.
None
The None compression option disables compression entirely, resulting in significantly larger files. However, this can be useful for loading small CSV files directly into Excel.
Dates and times
Our Python client library has several functions with timestamp arguments. These arguments will have type pandas.Timestamp | datetime.date | str | int and support a variety of formats.
It's recommended to use pandas.Timestamp, which fully supports timezones and nanosecond-precision. If a datetime.date is used, the time is set to midnight UTC. If an int is provided, the value is interpreted as UNIX nanoseconds.
The client library also handles several string-based timestamp formats based on ISO 8601.
yyyy-mm-dd, e.g."2022-02-28"(midnight UTC)yyyy-mm-ddTHH:MM, e.g."2022-02-28T23:50"yyyy-mm-ddTHH:MM:SS, e.g."2022-02-28T23:50:59"yyyy-mm-ddTHH:MM:SS.NNNNNNNNN, e.g."2022-02-28T23:50:59.123456789"
Timezone specification is also supported.
yyyy-mm-ddTHH:MMZyyyy-mm-ddTHH:MM±hhyyyy-mm-ddTHH:MM±hhmmyyyy-mm-ddTHH:MM±hh:mm
Bare dates
Some parameters require a bare date, without a time. These arguments have type datetime.date | str and must either be a datetime.date object, or a string in yyyy-mm-dd format, e.g. "2022-02-28".
Errors
Our historical API uses HTTP response codes to indicate the success or failure of an API request. The client library provides exceptions that wrap these response codes.
2xxindicates success.4xxindicates an error on the client side. Represented as aBentoClientError.5xxindicates an error with Databento's servers. Represented as aBentoServerError.
The full list of the response codes and associated causes is as follows:
| Code | Message | Cause |
|---|---|---|
| 200 | OK | Successful request. |
| 206 | Partial Content | Successful request, with partially resolved symbols. |
| 400 | Bad Request | Invalid request. Usually due to a missing, malformed or unsupported parameter. |
| 401 | Unauthorized | Invalid username or API key. |
| 402 | Payment Required | Issue with your account payment information. |
| 403 | Forbidden | The API key has insufficient permissions to perform the request. |
| 404 | Not Found | A resource is not found, or a requested symbol does not exist. |
| 409 | Conflict | A resource already exists. |
| 422 | Unprocessable Entity | The request is well formed, but we cannot or will not process the contained instructions. |
| 429 | Too Many Requests | API rate limit exceeded. |
| 500 | Internal Server Error | Unexpected condition encountered in our system. |
| 503 | Service Unavailable | Data gateway is offline or overloaded. |
| 504 | Gateway Timeout | Data gateway is available but other parts of our system are offline or overloaded. |
1 2 3 4
class databento.BentoError(Exception)
class databento.BentoHttpError(databento.BentoError)
class databento.BentoClientError(databento.BentoHttpError)
class databento.BentoServerError(databento.BentoHttpError) 1 2 3 4 5 6 7 8
import databento as db
client = db.Historical("INVALID_API_KEY")
try:
print(client.metadata.list_datasets())
except db.BentoClientError as e:
print(e) 1 2 3
400 auth_invalid_username_in_basic_auth
Invalid username in Basic auth ('INVALID_API_KEY').
documentation: https://databento.com/docs Rate limits
Our historical API allows each IP address up to:
- 100 concurrent connections.
- 100 time series requests per second.
- 100 symbology requests per second.
- 20 metadata requests per second.
- 20 batch list jobs requests per second.
- 20 batch submit job requests per minute.
When a request exceeds a rate limit, a BentoClientError exception is raised with a 429 error code.
Retry-After
The Retry-After response header indicates how long the user should wait before retrying.
If you find that your application has been rate-limited, you can retry after waiting for the time specified in the Retry-After header.
If you are using Python, you may use the time.sleep function as seen below to wait for the time specified in the Retry-After header. e.g. time.sleep(int(response.headers("Retry-After", 1)))
This code snippet works best for our current APIs with their rate limits. Future APIs may have different rate limits, and might require a different default time delay.
Size limits
There is no size limit for either stream or batch download requests. Batch download is more manageable for large datasets, so we recommend using batch download for requests over 5 GB.
You can also manage the size of your request by splitting it into multiple, smaller requests. The historical API allows you to make stream and batch download requests with time ranges specified up to nanosecond resolution. You can also use the limit parameter in any request to limit the number of data records returned from the service.
Batch download supports different delivery methods which can be specified using the delivery parameter.
1 2 3 4 5 6 7 8 9 10 11 12
import databento as db
client = db.Historical("$YOUR_API_KEY")
job = client.batch.submit_job(
dataset="GLBX.MDP3",
symbols="CLZ7",
schema="trades",
start="2022-06-06T00:00:00",
end="2022-06-10T00:10:00",
limit=10000,
) Metered pricing
Databento only charges for the data that you use. You can find rates (per GB) for the various datasets and estimate pricing on our Data catalog. We meter the data by its uncompressed size in binary encoding.
When you stream the data, you are billed incrementally for each outbound byte of data sent from our historical gateway. If your connection is interrupted while streaming our data and our historical gateway detects connection timeout over 5 seconds, it will immediately stop sending data and you will not be billed for the remainder of your request.
Duplicate streaming requests will incur repeated charges. If you intend to access the same data multiple times, we recommend using our batch download feature. When you make a batch download request, you are only billed once for the request and, subsequently, you can download the data from the Download center multiple times over 30 days for no additional charge.
You will only be billed for usage of time series data. Access to metadata, symbology, and account management is free. The Historical.metadata.get_cost method can be used to determine cost before you request any data.
Related: Billing management.
Versioning
Our historical and live APIs and their client libraries adopt MAJOR.MINOR.PATCH format for version numbers. These version numbers conform to semantic versioning. We are using major version 0 for initial development, where our API is not considered stable.
Once we release major version 1, our public API will be stable. This means that you will be able to upgrade minor or patch versions to pick up new functionality, without breaking your integration.
Starting with major versions after 1, we will provide support for previous versions for one year after the date of the subsequent major release. For example, if version 2.0.0 is released on January 1, 2024, then all versions 1.x.y of the API and client libraries will be deprecated. However, they will remain supported until January 1, 2025.
We may introduce backwards-compatible changes between minor versions in the form of:
- New data encodings
- Additional fields to existing data schemas
- Additional batch download customizations
Our Release notes will contain information about both breaking and backwards-compatible changes in each release.
Our API and official client libraries are kept in sync with same-day releases for major versions. For instance, 1.x.y of the C++ client library will use the same functionality found in any 1.x.y version of the Python client.
Related: Release notes.