# Databento Docs > Databento provides live and historical market data via a unified API covering equities, futures, options, and other asset classes. --- Source: https://databento.com/docs/api-historical?lang=python # API reference - Historical Databento's historical data service can be accessed programmatically over its HTTP API. To make it easier to integrate the API, we also provide official client libraries that simplify the code you need to write. Our HTTP API is designed as a collection of RPC-style methods, which can be called using URLs in the form `https://hist.databento.com/v0/METHOD_FAMILY.METHOD`. Our client libraries wrap these HTTP RPC-style methods with more idiomatic interfaces in their respective languages. You can use our API to [stream](https://databento.com/docs/api-historical/timeseries.md) or load data directly into your application. You can also use our API to make [batch download](https://databento.com/docs/api-historical/batch.md) requests, which instruct our service to prepare the data as flat files that can be downloaded from the [Download center]($PLATFORM_HOSTNAME/portal/download-center). --- Source: https://databento.com/docs/api-historical/basics?lang=python ## Basics --- Source: https://databento.com/docs/api-historical/basics/overview?lang=python ### Overview Our historical API has the following structure: - [Metadata](https://databento.com/docs/api-historical/metadata.md) provides information about the datasets themselves. - [Time series](https://databento.com/docs/api-historical/timeseries.md) provides all types of time series data. This includes subsampled data (second, minute, hour, daily aggregates), trades, top-of-book, order book deltas, order book snapshots, summary statistics, static data and macro indicators. We also provide properties of products such as expirations, tick sizes and symbols as time series data. - [Symbology](https://databento.com/docs/api-historical/symbology.md) provides methods that help find and resolve symbols across different symbology systems. - [Batch](https://databento.com/docs/api-historical/batch.md) provides a means of submitting and querying for details of batch download requests. --- Source: https://databento.com/docs/api-historical/basics/authentication?lang=python ### Authentication Databento uses API keys to authenticate requests. You can view and manage your keys on the [API keys]($PLATFORM_HOSTNAME/portal/keys) page of your portal. Each API key is a 32-character string starting with `db-`. By default, our library uses the environment variable `DATABENTO_API_KEY` as your API key. However, if you pass an API key to the `Historical` constructor through the `key` parameter, then this value will be used instead. Related: [Securing your API keys](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). #### Example usage ```python import databento as db # Establish connection and authenticate client = db.Historical("$YOUR_API_KEY") # Authenticated request print(client.metadata.list_datasets()) ``` --- Source: https://databento.com/docs/api-historical/basics/schemas-and-conventions?lang=python ### Schemas and conventions A schema is a data record format represented as a collection of different data fields. Our datasets support multiple schemas, such as order book, trades, bar aggregates, and so on. You can get a dictionary describing the fields of each schema from our [List of market data schemas](https://databento.com/docs/schemas/whats-a-schema.md). You can get a list of all supported schemas for any given dataset using the Historical client's [list_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md) method. The same information can also be found on the dataset details pages on the [user portal]($PLATFORM_HOSTNAME). The following table provides details about the data types and conventions used for various fields that you will commonly encounter in the data. | Name | Field | Description | | Dataset | `dataset` | A unique string name assigned to each dataset by Databento. Full list of datasets can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | Publisher ID | `publisher_id` | A unique 16-bit unsigned integer assigned to each publisher by Databento. Full list of publisher IDs can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md). | | Instrument ID | `instrument_id` | A unique 32-bit unsigned integer assigned to each instrument by the venue. Information about instrument IDs for any given dataset can be found in the [symbology](https://databento.com/docs/api-historical/symbology.md). | | Order ID | `order_id` | A unique 64-bit unsigned integer assigned to each order by the venue. | | Timestamp (event) | `ts_event` | The matching-engine-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp (receive) | `ts_recv` | The capture-server-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp delta (in) | `ts_in_delta` | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [timestamping](https://databento.com/docs/architecture/timestamping-guide.md) guide. | | Timestamp out | `ts_out` | The Databento gateway-sending timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). See [timestamping](https://databento.com/docs/architecture/timestamping-guide.md) guide. | | Price | `price` | The price expressed as signed integer where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | Book side | `side` | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified by the original source. | | Size | `size` | The order quantity. | | Flag | `flag` | A bit field indicating event end, message characteristics, and data quality. | | Action | `action` | The event type or order book operation. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, **T**rade, **F**ill, or **N**one. | | Sequence number | `sequence` | The original message sequence number from the venue. | --- Source: https://databento.com/docs/api-historical/basics/datasets?lang=python ### Datasets Databento provides time series datasets for a variety of markets, sourced from different publishers. Our available datasets can be browsed through the [search]($PLATFORM_HOSTNAME) feature on our site. Each dataset is assigned a unique string identifier (dataset ID) in the form `PUBLISHER.DATASET`, such as `GLBX.MDP3`. For publishers that are also markets, we use standard four-character ISO 10383 [Market Identifier Codes (MIC)](https://www.iso20022.org/market-identifier-codes). Otherwise, Databento arbitrarily assigns a four-character identifier for the publisher. These dataset IDs are also found on the [Data catalog](https://databento.com/docs/portal/data-catalog.md) and [Download request](https://databento.com/docs/portal/batch-download.md) features of the Databento user portal. When a publisher provides multiple data products with different levels of granularity, Databento subscribes to the most-granular product. We then provide this dataset with [alternate schemas](https://databento.com/docs/schemas/whats-a-schema.md) to make it easy to work with the level of detail most appropriate for your application. More information about different types of venues and publishers is available in our [FAQs](https://databento.com/docs/faqs/venues-and-publishers.md). --- Source: https://databento.com/docs/api-historical/basics/symbology?lang=python ### Symbology Databento's historical API supports several ways to select an instrument in a [dataset](https://databento.com/docs/api-historical/basics/datasets.md). An instrument is specified using a **symbol** and a **symbology type**, also referred to as an **stype**. The supported symbology types are: - **[Raw symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol)** (`raw_symbol`) original string symbols used by the publisher in the source data. - **[Instrument ID symbology](https://databento.com/docs/conventions/symbology.md#instrument-id)** (`instrument_id`) unique numeric ID assigned to each instrument by the publisher. - **[Parent symbology](https://databento.com/docs/conventions/symbology.md#parent)** (`parent`) groups instruments related to the market for the same underlying. - **[Continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous)** (`continuous`) proprietary symbology that specifies instruments based on certain systematic rules. When requesting data from our [timeseries.get_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md) or [batch.submit_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) endpoints, an input and output symbology type can be specified. By default, our client libraries will use raw symbology for the input type and instrument ID symbology for the output type. Not all symbology types are supported for every dataset. The process of converting between one symbology type to another is called **symbology resolution**. This conversion can be done, for no cost, with the [symbology.resolve](https://databento.com/docs/api-historical/symbology/symbology-resolve.md) endpoint. For more about symbology at Databento, see our [Standards and conventions](https://databento.com/docs/conventions/symbology.md). --- Source: https://databento.com/docs/api-historical/basics/encodings?lang=python ### Encodings **DBN** Databento Binary Encoding ([DBN](https://databento.com/docs/conventions/databento-binary-encoding.md)) is an extremely fast message encoding and highly-compressible storage format for normalized market data. It includes a self-describing metadata header and adopts a binary format with zero-copy serialization. We recommend using our [Python](https://databento.com/docs/api-historical/helpers/dbn-store.md?lang=python), [C++](https://databento.com/docs/api-historical/helpers/dbn-store.md?lang=cpp), or [Rust client libraries](https://databento.com/docs/api-historical/helpers/dbn-store.md?lang=rust) to read DBN files locally. A [CLI tool](https://crates.io/crates/dbn-cli) is also available for converting DBN files to CSV or JSON. **CSV** Comma-separated values (CSV) is a simple text file format for tabular data, CSVs can be easily opened with Excel, loaded into [pandas](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html) data frames, or parsed in C++. Our CSVs have one header line, followed by one record per line. Lines use UNIX-style `\n` separators. **JSON** [JavaScript Object Notation](https://www.json.org/json-en.html) (JSON) is a flexible text file format with broad language support and wide adoption across web apps. Our JSON files follow the [JSON lines specification](https://jsonlines.org/), where each line of the file is a JSON record. Lines use UNIX-style `\n` separators. --- Source: https://databento.com/docs/api-historical/basics/compression?lang=python ### Compression Databento provides options for compressing files from our API. Available compression formats depend on the [encoding](https://databento.com/docs/api-historical/basics/encodings.md) you select. **Zstd** The `Zstd` compression option uses the [Zstandard](http://facebook.github.io/zstd/) format. This option is available for all encodings, and is recommended for faster transfer speeds and smaller files. You can read Zstandard files in Python using the [zstandard package](https://pypi.org/project/zstandard/). Read more about [working with Zstandard-compressed files](https://databento.com/docs/conventions/working-with-zstandard.md). **None** The `None` compression option disables compression entirely, resulting in significantly larger files. However, this can be useful for loading small CSV files directly into Excel. --- Source: https://databento.com/docs/api-historical/basics/dates-and-times?lang=python ### Dates and times Our Python client library has several functions with timestamp arguments. These arguments will have type `pandas.Timestamp | datetime.date | str | int` and support a variety of formats. It's recommended to use [pandas.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), which fully supports timezones and nanosecond-precision. If a `datetime.date` is used, the time is set to midnight UTC. If an `int` is provided, the value is interpreted as [UNIX nanoseconds](https://en.wikipedia.org/wiki/Unix_time). The client library also handles several string-based timestamp formats based on [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html). - `yyyy-mm-dd`, e.g. `"2022-02-28"` (midnight UTC) - `yyyy-mm-ddTHH:MM`, e.g. `"2022-02-28T23:50"` - `yyyy-mm-ddTHH:MM:SS`, e.g. `"2022-02-28T23:50:59"` - `yyyy-mm-ddTHH:MM:SS.NNNNNNNNN`, e.g. `"2022-02-28T23:50:59.123456789"` Timezone specification is also supported. - `yyyy-mm-ddTHH:MMZ` - `yyyy-mm-ddTHH:MM±hh` - `yyyy-mm-ddTHH:MM±hhmm` - `yyyy-mm-ddTHH:MM±hh:mm` **Bare dates** Some parameters require a bare date, without a time. These arguments have type `datetime.date | str` and must either be a `datetime.date` object, or a string in `yyyy-mm-dd` format, e.g. `"2022-02-28"`. --- Source: https://databento.com/docs/api-historical/basics/errors?lang=python ### Errors Our historical API uses HTTP response codes to indicate the success or failure of an API request. The client library provides exceptions that wrap these response codes. - `2xx` indicates success. - `4xx` indicates an error on the client side. Represented as a `BentoClientError`. - `5xx` indicates an error with Databento's servers. Represented as a `BentoServerError`. The full list of the response codes and associated causes is as follows: | Code | Message | Cause | | 200 | OK | Successful request. | | 206 | Partial Content | Successful request, with partially resolved symbols. | | 400 | Bad Request | Invalid request. Usually due to a missing, malformed or unsupported parameter. | | 401 | Unauthorized | Invalid username or API key. | | 402 | Payment Required | Issue with your account payment information. | | 403 | Forbidden | The API key has insufficient permissions to perform the request. | | 404 | Not Found | A resource is not found, or a requested symbol does not exist. | | 409 | Conflict | A resource already exists. | | 422 | Unprocessable Entity | The request is well formed, but we cannot or will not process the contained instructions. | | 429 | Too Many Requests | API rate limit exceeded. | | 500 | Internal Server Error | Unexpected condition encountered in our system. | | 503 | Service Unavailable | Data gateway is offline or overloaded. | | 504 | Gateway Timeout | Data gateway is available but other parts of our system are offline or overloaded. | #### API method ```python class databento.BentoError(Exception) class databento.BentoHttpError(databento.BentoError) class databento.BentoClientError(databento.BentoHttpError) class databento.BentoServerError(databento.BentoHttpError) ``` #### Example usage ```python import databento as db client = db.Historical("INVALID_API_KEY") try: print(client.metadata.list_datasets()) except db.BentoClientError as e: print(e) ``` #### Example response ```plaintext 400 auth_invalid_username_in_basic_auth Invalid username in Basic auth ('INVALID_API_KEY'). documentation: https://databento.com/docs ``` --- Source: https://databento.com/docs/api-historical/basics/rate-limits?lang=python ### Rate limits Our historical API allows each IP address up to: - 100 concurrent connections. - 100 [time series](https://databento.com/docs/api-historical/timeseries.md) requests per second. - 100 [symbology](https://databento.com/docs/api-historical/symbology.md) requests per second. - 20 [metadata](https://databento.com/docs/api-historical/metadata.md) requests per second. - 20 [batch list jobs](https://databento.com/docs/api-historical/batch/batch-list-jobs.md) requests per second. - 20 [batch submit job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) requests per minute. When a request exceeds a rate limit, a `BentoClientError` exception is raised with a 429 error code. **Retry-After** The Retry-After response header indicates how long the user should wait before retrying. If you find that your application has been rate-limited, you can retry after waiting for the time specified in the Retry-After header. If you are using Python, you may use the time.sleep function as seen below to wait for the time specified in the Retry-After header. e.g. `time.sleep(int(response.headers("Retry-After", 1)))` This code snippet works best for our current APIs with their rate limits. Future APIs may have different rate limits, and might require a different default time delay. --- Source: https://databento.com/docs/api-historical/basics/size-limits?lang=python ### Size limits There is no size limit for either stream or batch download requests. Batch download is more manageable for large datasets, so we recommend using batch download for requests over 5 GB. You can also manage the size of your request by splitting it into multiple, smaller requests. The historical API allows you to make stream and batch download requests with time ranges specified up to nanosecond resolution. You can also use the `limit` parameter in any request to limit the number of data records returned from the service. [Batch download](https://databento.com/docs/api-historical/batch.md) supports different delivery methods which can be specified using the `delivery` parameter. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") job = client.batch.submit_job( dataset="GLBX.MDP3", symbols="CLZ7", schema="trades", start="2022-06-06T00:00:00", end="2022-06-10T00:10:00", limit=10000, ) ``` --- Source: https://databento.com/docs/api-historical/basics/metered-pricing?lang=python ### Metered pricing Databento only charges for the data that you use. You can find rates (per GB) for the various datasets and estimate pricing on our [Data catalog]($PLATFORM_HOSTNAME/portal/browse). We meter the data by its uncompressed size in [binary encoding](https://databento.com/docs/conventions/databento-binary-encoding.md). When you stream the data, you are billed incrementally for each outbound byte of data sent from our historical gateway. If your connection is interrupted while streaming our data and our historical gateway detects connection timeout over 5 seconds, it will immediately stop sending data and you will not be billed for the remainder of your request. Duplicate streaming requests will incur repeated charges. If you intend to access the same data multiple times, we recommend using our batch download feature. When you make a batch download request, you are only billed once for the request and, subsequently, you can download the data from the [Download center]($PLATFORM_HOSTNAME/portal/download-center) multiple times over 30 days for no additional charge. You will only be billed for usage of [time series](https://databento.com/docs/api-historical/timeseries.md) data. Access to metadata, symbology, and account management is free. The [Historical.metadata.get\_cost](https://databento.com/docs/api-historical/metadata/metadata-get-cost.md) method can be used to determine cost before you request any data. Related: [Billing management](https://databento.com/docs/portal/billing.md). --- Source: https://databento.com/docs/api-historical/basics/versioning?lang=python ### Versioning Our historical and live APIs and their client libraries adopt `MAJOR.MINOR.PATCH` format for version numbers. These version numbers conform to [semantic versioning](https://semver.org). We are using major version `0` for initial development, where our API is not considered stable. Once we release major version `1`, our public API will be stable. This means that you will be able to upgrade minor or patch versions to pick up new functionality, without breaking your integration. Starting with major versions after `1`, we will provide support for previous versions for one year after the date of the subsequent major release. For example, if version `2.0.0` is released on January 1, 2024, then all versions `1.x.y` of the API and client libraries will be deprecated. However, they will remain supported until January 1, 2025. We may introduce backwards-compatible changes between minor versions in the form of: - New data [encodings](https://databento.com/docs/api-historical/basics/encodings.md) - Additional fields to existing data [schemas](https://databento.com/docs/api-historical/basics/schemas-and-conventions.md) - Additional batch download [customizations](https://databento.com/docs/api-historical/batch.md) Our [Release notes](https://databento.com/docs/release-notes/release-notes-python.md) will contain information about both breaking and backwards-compatible changes in each release. Our API and official client libraries are kept in sync with same-day releases for major versions. For instance, `1.x.y` of the C++ client library will use the same functionality found in any `1.x.y` version of the Python client. Related: [Release notes](https://databento.com/docs/release-notes/release-notes-python.md). --- Source: https://databento.com/docs/api-historical/client?lang=python ## Client --- Source: https://databento.com/docs/api-historical/client/historical?lang=python ### Historical To access Databento's historical API, first create an instance of the `Historical` client. The entire API is exposed through instance methods of the client. Note that the API key can be passed as a parameter, which is [not recommended for production applications](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). Instead, you can leave out this parameter to pass your API key via the `DATABENTO_API_KEY` environment variable: Currently, only `BO1` is supported for historical data. #### API method ```python class Historical( key: str | None = None, gateway: HistoricalGateway | str = HistoricalGateway.BO1, ) ``` #### Example usage ```python import databento as db # Pass as parameter client = db.Historical("$YOUR_API_KEY") # Or, pass as `DATABENTO_API_KEY` environment variable client = db.Historical() ``` #### Parameters | Parameter | Type | Required | Description | | `key` | str | optional | 32-character API key. Found on your [API keys]($PLATFORM_HOSTNAME/portal/keys) page. If `None` then `DATABENTO_API_KEY` environment variable is used. | | `gateway` | HistoricalGateway or str | optional | Site of historical gateway to connect to. Currently only `BO1` is supported. If `None` then will connect to the default historical gateway. | --- Source: https://databento.com/docs/api-historical/metadata?lang=python ## Metadata --- Source: https://databento.com/docs/api-historical/metadata/metadata-list-publishers?lang=python ### Historical.metadata.list\_publishers List all [publisher ID](https://databento.com/docs/faqs/venues-and-publishers.md) mappings. Use this method to list the details of publishers, including their dataset and venue mappings. #### API method ```python Historical.metadata.list_publishers() -> list[dict[str, int | str]] ``` #### Returns `list[dict[str, int | str]]` A list of publisher details objects. | Field | Type | Description | | `publisher_id` | int | The publisher ID assigned by Databento. | | `dataset` | str | The dataset ID for the publisher. | | `venue` | str | The venue for the publisher. | | `description` | str | The publisher description. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") publishers = client.metadata.list_publishers() print(publishers) ``` #### Example response ```plaintext [{'dataset': 'GLBX.MDP3', 'description': 'CME Globex MDP 3.0', 'publisher_id': 1, 'venue': 'GLBX'}, {'dataset': 'XNAS.ITCH', 'description': 'Nasdaq TotalView-ITCH', 'publisher_id': 2, 'venue': 'XNAS'}, {'dataset': 'XBOS.ITCH', 'description': 'Nasdaq Texas TotalView-ITCH', 'publisher_id': 3, 'venue': 'XBOS'}, {'dataset': 'XPSX.ITCH', 'description': 'Nasdaq PSX TotalView-ITCH', 'publisher_id': 4, 'venue': 'XPSX'}, {'dataset': 'BATS.PITCH', 'description': 'Cboe BZX Depth', 'publisher_id': 5, 'venue': 'BATS'}, {'dataset': 'BATY.PITCH', 'description': 'Cboe BYX Depth', 'publisher_id': 6, 'venue': 'BATY'}, {'dataset': 'EDGA.PITCH', 'description': 'Cboe EDGA Depth', 'publisher_id': 7, 'venue': 'EDGA'}, {'dataset': 'EDGX.PITCH', 'description': 'Cboe EDGX Depth', 'publisher_id': 8, 'venue': 'EDGX'}, {'dataset': 'XNYS.PILLAR', 'description': 'NYSE Integrated', 'publisher_id': 9, 'venue': 'XNYS'}, {'dataset': 'XCIS.PILLAR', 'description': 'NYSE National Integrated', 'publisher_id': 10, 'venue': 'XCIS'}, {'dataset': 'XASE.PILLAR', 'description': 'NYSE American Integrated', 'publisher_id': 11, 'venue': 'XASE'}, {'dataset': 'XCHI.PILLAR', 'description': 'NYSE Texas Integrated', 'publisher_id': 12, 'venue': 'XCHI'}, {'dataset': 'XCIS.BBO', 'description': 'NYSE National BBO', 'publisher_id': 13, 'venue': 'XCIS'}, {'dataset': 'XCIS.TRADES', 'description': 'NYSE National Trades', 'publisher_id': 14, 'venue': 'XCIS'}, {'dataset': 'MEMX.MEMOIR', 'description': 'MEMX MEMOIR Depth', 'publisher_id': 15, 'venue': 'MEMX'}, {'dataset': 'EPRL.DOM', 'description': 'MIAX Pearl Depth', 'publisher_id': 16, 'venue': 'EPRL'}, {'dataset': 'XNAS.NLS', 'description': 'FINRA/Nasdaq TRF Carteret', 'publisher_id': 17, 'venue': 'FINN'}, {'dataset': 'XNAS.NLS', 'description': 'FINRA/Nasdaq TRF Chicago', 'publisher_id': 18, 'venue': 'FINC'}, {'dataset': 'XNYS.TRADES', 'description': 'FINRA/NYSE TRF', 'publisher_id': 19, 'venue': 'FINY'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - NYSE American Options', 'publisher_id': 20, 'venue': 'AMXO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - BOX Options', 'publisher_id': 21, 'venue': 'XBOX'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Cboe Options', 'publisher_id': 22, 'venue': 'XCBO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - MIAX Emerald', 'publisher_id': 23, 'venue': 'EMLD'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Cboe EDGX Options', 'publisher_id': 24, 'venue': 'EDGO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Nasdaq GEMX', 'publisher_id': 25, 'venue': 'GMNI'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Nasdaq ISE', 'publisher_id': 26, 'venue': 'XISX'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Nasdaq MRX', 'publisher_id': 27, 'venue': 'MCRY'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - MIAX Options', 'publisher_id': 28, 'venue': 'XMIO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - NYSE Arca Options', 'publisher_id': 29, 'venue': 'ARCO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Options Price Reporting Authority', 'publisher_id': 30, 'venue': 'OPRA'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - MIAX Pearl', 'publisher_id': 31, 'venue': 'MPRL'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Nasdaq Options', 'publisher_id': 32, 'venue': 'XNDQ'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Nasdaq Texas Options', 'publisher_id': 33, 'venue': 'XBXO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Cboe C2 Options', 'publisher_id': 34, 'venue': 'C2OX'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Nasdaq PHLX', 'publisher_id': 35, 'venue': 'XPHL'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - Cboe BZX Options', 'publisher_id': 36, 'venue': 'BATO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - MEMX Options', 'publisher_id': 37, 'venue': 'MXOP'}, {'dataset': 'IEXG.TOPS', 'description': 'IEX TOPS', 'publisher_id': 38, 'venue': 'IEXG'}, {'dataset': 'DBEQ.BASIC', 'description': 'DBEQ Basic - NYSE Texas', 'publisher_id': 39, 'venue': 'XCHI'}, {'dataset': 'DBEQ.BASIC', 'description': 'DBEQ Basic - NYSE National', 'publisher_id': 40, 'venue': 'XCIS'}, {'dataset': 'DBEQ.BASIC', 'description': 'DBEQ Basic - IEX', 'publisher_id': 41, 'venue': 'IEXG'}, {'dataset': 'DBEQ.BASIC', 'description': 'DBEQ Basic - MIAX Pearl', 'publisher_id': 42, 'venue': 'EPRL'}, {'dataset': 'ARCX.PILLAR', 'description': 'NYSE Arca Integrated', 'publisher_id': 43, 'venue': 'ARCX'}, {'dataset': 'XNYS.BBO', 'description': 'NYSE BBO', 'publisher_id': 44, 'venue': 'XNYS'}, {'dataset': 'XNYS.TRADES', 'description': 'NYSE Trades', 'publisher_id': 45, 'venue': 'XNYS'}, {'dataset': 'XNAS.QBBO', 'description': 'Nasdaq QBBO', 'publisher_id': 46, 'venue': 'XNAS'}, {'dataset': 'XNAS.NLS', 'description': 'Nasdaq Trades', 'publisher_id': 47, 'venue': 'XNAS'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - NYSE Texas', 'publisher_id': 48, 'venue': 'XCHI'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - NYSE National', 'publisher_id': 49, 'venue': 'XCIS'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - IEX', 'publisher_id': 50, 'venue': 'IEXG'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - MIAX Pearl', 'publisher_id': 51, 'venue': 'EPRL'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - Nasdaq', 'publisher_id': 52, 'venue': 'XNAS'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - NYSE', 'publisher_id': 53, 'venue': 'XNYS'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - FINRA/Nasdaq TRF Carteret', 'publisher_id': 54, 'venue': 'FINN'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - FINRA/NYSE TRF', 'publisher_id': 55, 'venue': 'FINY'}, {'dataset': 'EQUS.PLUS', 'description': 'Databento US Equities Plus - FINRA/Nasdaq TRF Chicago', 'publisher_id': 56, 'venue': 'FINC'}, {'dataset': 'IFEU.IMPACT', 'description': 'ICE Europe Commodities', 'publisher_id': 57, 'venue': 'IFEU'}, {'dataset': 'NDEX.IMPACT', 'description': 'ICE Endex', 'publisher_id': 58, 'venue': 'NDEX'}, {'dataset': 'DBEQ.BASIC', 'description': 'Databento US Equities Basic - Consolidated', 'publisher_id': 59, 'venue': 'DBEQ'}, {'dataset': 'EQUS.PLUS', 'description': 'EQUS Plus - Consolidated', 'publisher_id': 60, 'venue': 'EQUS'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - MIAX Sapphire', 'publisher_id': 61, 'venue': 'SPHR'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - NYSE Texas', 'publisher_id': 62, 'venue': 'XCHI'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - NYSE National', 'publisher_id': 63, 'venue': 'XCIS'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - IEX', 'publisher_id': 64, 'venue': 'IEXG'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - MIAX Pearl', 'publisher_id': 65, 'venue': 'EPRL'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Nasdaq', 'publisher_id': 66, 'venue': 'XNAS'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - NYSE', 'publisher_id': 67, 'venue': 'XNYS'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - FINRA/Nasdaq TRF Carteret', 'publisher_id': 68, 'venue': 'FINN'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - FINRA/NYSE TRF', 'publisher_id': 69, 'venue': 'FINY'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - FINRA/Nasdaq TRF Chicago', 'publisher_id': 70, 'venue': 'FINC'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Cboe BZX', 'publisher_id': 71, 'venue': 'BATS'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Cboe BYX', 'publisher_id': 72, 'venue': 'BATY'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Cboe EDGA', 'publisher_id': 73, 'venue': 'EDGA'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Cboe EDGX', 'publisher_id': 74, 'venue': 'EDGX'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Nasdaq Texas', 'publisher_id': 75, 'venue': 'XBOS'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Nasdaq PSX', 'publisher_id': 76, 'venue': 'XPSX'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - MEMX', 'publisher_id': 77, 'venue': 'MEMX'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - NYSE American', 'publisher_id': 78, 'venue': 'XASE'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - NYSE Arca', 'publisher_id': 79, 'venue': 'ARCX'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Long-Term Stock Exchange', 'publisher_id': 80, 'venue': 'LTSE'}, {'dataset': 'XNAS.BASIC', 'description': 'Nasdaq Basic - Nasdaq', 'publisher_id': 81, 'venue': 'XNAS'}, {'dataset': 'XNAS.BASIC', 'description': 'Nasdaq Basic - FINRA/Nasdaq TRF Carteret', 'publisher_id': 82, 'venue': 'FINN'}, {'dataset': 'XNAS.BASIC', 'description': 'Nasdaq Basic - FINRA/Nasdaq TRF Chicago', 'publisher_id': 83, 'venue': 'FINC'}, {'dataset': 'IFEU.IMPACT', 'description': 'ICE Europe - Off-Market Trades', 'publisher_id': 84, 'venue': 'XOFF'}, {'dataset': 'NDEX.IMPACT', 'description': 'ICE Endex - Off-Market Trades', 'publisher_id': 85, 'venue': 'XOFF'}, {'dataset': 'XNAS.NLS', 'description': 'Nasdaq NLS - Nasdaq Texas', 'publisher_id': 86, 'venue': 'XBOS'}, {'dataset': 'XNAS.NLS', 'description': 'Nasdaq NLS - Nasdaq PSX', 'publisher_id': 87, 'venue': 'XPSX'}, {'dataset': 'XNAS.BASIC', 'description': 'Nasdaq Basic - Nasdaq Texas', 'publisher_id': 88, 'venue': 'XBOS'}, {'dataset': 'XNAS.BASIC', 'description': 'Nasdaq Basic - Nasdaq PSX', 'publisher_id': 89, 'venue': 'XPSX'}, {'dataset': 'EQUS.SUMMARY', 'description': 'Databento Equities Summary', 'publisher_id': 90, 'venue': 'EQUS'}, {'dataset': 'XCIS.TRADESBBO', 'description': 'NYSE National Trades and BBO', 'publisher_id': 91, 'venue': 'XCIS'}, {'dataset': 'XNYS.TRADESBBO', 'description': 'NYSE Trades and BBO', 'publisher_id': 92, 'venue': 'XNYS'}, {'dataset': 'XNAS.BASIC', 'description': 'Nasdaq Basic - Consolidated', 'publisher_id': 93, 'venue': 'EQUS'}, {'dataset': 'EQUS.ALL', 'description': 'Databento US Equities (All Feeds) - Consolidated', 'publisher_id': 94, 'venue': 'EQUS'}, {'dataset': 'EQUS.MINI', 'description': 'Databento US Equities Mini', 'publisher_id': 95, 'venue': 'EQUS'}, {'dataset': 'XNYS.TRADES', 'description': 'NYSE Trades - Consolidated', 'publisher_id': 96, 'venue': 'EQUS'}, {'dataset': 'IFUS.IMPACT', 'description': 'ICE Futures US', 'publisher_id': 97, 'venue': 'IFUS'}, {'dataset': 'IFUS.IMPACT', 'description': 'ICE Futures US - Off-Market Trades', 'publisher_id': 98, 'venue': 'XOFF'}, {'dataset': 'IFLL.IMPACT', 'description': 'ICE Europe Financials', 'publisher_id': 99, 'venue': 'IFLL'}, {'dataset': 'IFLL.IMPACT', 'description': 'ICE Europe Financials - Off-Market Trades', 'publisher_id': 100, 'venue': 'XOFF'}, {'dataset': 'XEUR.EOBI', 'description': 'Eurex EOBI', 'publisher_id': 101, 'venue': 'XEUR'}, {'dataset': 'XEEE.EOBI', 'description': 'European Energy Exchange EOBI', 'publisher_id': 102, 'venue': 'XEEE'}, {'dataset': 'XEUR.EOBI', 'description': 'Eurex EOBI - Off-Market Trades', 'publisher_id': 103, 'venue': 'XOFF'}, {'dataset': 'XEEE.EOBI', 'description': 'European Energy Exchange EOBI - Off-Market Trades', 'publisher_id': 104, 'venue': 'XOFF'}, {'dataset': 'XCBF.PITCH', 'description': 'Cboe Futures Exchange (CFE)', 'publisher_id': 105, 'venue': 'XCBF'}, {'dataset': 'XCBF.PITCH', 'description': 'Cboe Futures Exchange (CFE) - Off-Market Trades', 'publisher_id': 106, 'venue': 'XOFF'}, {'dataset': 'OCEA.MEMOIR', 'description': 'Blue Ocean ATS MEMOIR', 'publisher_id': 107, 'venue': 'OCEA'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - MEMX MX2 Options', 'publisher_id': 108, 'venue': 'MXTO'}, {'dataset': 'OPRA.PILLAR', 'description': 'OPRA - IEX Options LLC', 'publisher_id': 109, 'venue': 'IEXO'}, {'dataset': 'MAIN.CGIF', 'description': 'Cboe Global Indices Feed', 'publisher_id': 110, 'venue': 'MAIN'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - NYSE American', 'publisher_id': 111, 'venue': 'XASE'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Nasdaq Texas', 'publisher_id': 112, 'venue': 'XBOS'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - NYSE National', 'publisher_id': 113, 'venue': 'XCIS'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Texas Stock Exchange', 'publisher_id': 114, 'venue': 'TXSE'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - 24X National Exchange', 'publisher_id': 115, 'venue': '24EQ'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - MIAX Pearl', 'publisher_id': 116, 'venue': 'EPRL'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Nasdaq ISE', 'publisher_id': 117, 'venue': 'XISX'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Cboe EDGA', 'publisher_id': 118, 'venue': 'EDGA'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Cboe EDGX', 'publisher_id': 119, 'venue': 'EDGX'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Long-Term Stock Exchange', 'publisher_id': 120, 'venue': 'LTSE'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - NYSE Texas', 'publisher_id': 121, 'venue': 'XCHI'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - NYSE', 'publisher_id': 122, 'venue': 'XNYS'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - NYSE Arca', 'publisher_id': 123, 'venue': 'ARCX'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Nasdaq', 'publisher_id': 124, 'venue': 'XNAS'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - MEMX', 'publisher_id': 125, 'venue': 'MEMX'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - IEX', 'publisher_id': 126, 'venue': 'IEXG'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Nasdaq PSX', 'publisher_id': 127, 'venue': 'XPSX'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Cboe BYX', 'publisher_id': 128, 'venue': 'BATY'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - Cboe BZX', 'publisher_id': 129, 'venue': 'BATS'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - FINRA/Nasdaq TRF Carteret', 'publisher_id': 130, 'venue': 'FINN'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - FINRA/NYSE TRF', 'publisher_id': 131, 'venue': 'FINY'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - FINRA/Nasdaq TRF Chicago', 'publisher_id': 132, 'venue': 'FINC'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - FINRA Alternative Display Facility', 'publisher_id': 133, 'venue': 'XADF'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - CTA', 'publisher_id': 134, 'venue': 'CTA'}, {'dataset': 'EQUS.SIP', 'description': 'US Equities SIP - UTP', 'publisher_id': 135, 'venue': 'UTP'}, {'dataset': 'MSCI.CGIF', 'description': 'CGIF - Morgan Stanley Capital International', 'publisher_id': 136, 'venue': 'MSCI'}, {'dataset': 'FTSE.CGIF', 'description': 'CGIF - FTSE Russell', 'publisher_id': 137, 'venue': 'FTSE'}, {'dataset': 'INAV.CGIF', 'description': 'CGIF - Intraday Net Asset Values', 'publisher_id': 138, 'venue': 'INAV'}, {'dataset': 'MSTAR.CGIF', 'description': 'CGIF - Morning Star', 'publisher_id': 139, 'venue': 'MSTAR'}, {'dataset': 'CCCY.CGIF', 'description': 'CGIF - Crypto Currency', 'publisher_id': 140, 'venue': 'CCCY'}, {'dataset': 'CGI.CGIF', 'description': 'CGIF - CBOE Global Indices', 'publisher_id': 141, 'venue': 'CGI'}, {'dataset': 'MAIN.CGIF', 'description': 'CGIF - Definitions', 'publisher_id': 142, 'venue': 'CDEF'}, {'dataset': 'XTKS.FLEX', 'description': 'JPX - Tokyo Stock Exchange', 'publisher_id': 143, 'venue': 'XTKS'}, {'dataset': 'XTKT.ITCH', 'description': 'JPX - Tokyo Commodity Exchange', 'publisher_id': 144, 'venue': 'XTKT'}, {'dataset': 'XOSE.ITCH', 'description': 'JPX - Osaka Exchange', 'publisher_id': 145, 'venue': 'XOSE'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - Main Feed', 'publisher_id': 146, 'venue': 'MAIN'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - Morgan Stanley Capital International', 'publisher_id': 147, 'venue': 'MSCI'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - FTSE Russell', 'publisher_id': 148, 'venue': 'FTSE'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - Intraday Net Asset Values', 'publisher_id': 149, 'venue': 'INAV'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - Morning Star', 'publisher_id': 150, 'venue': 'MSTAR'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - Crypto Currency', 'publisher_id': 151, 'venue': 'CCCY'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - CBOE Global Indices', 'publisher_id': 152, 'venue': 'CGI'}, {'dataset': 'DBIX.CORE', 'description': 'CGIF - Definitions', 'publisher_id': 153, 'venue': 'CDEF'} ] ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-list-datasets?lang=python ### Historical.metadata.list\_datasets List all valid [dataset](https://databento.com/docs/api-historical/basics/datasets.md) IDs on Databento. Use this method to list the available dataset IDs (string identifiers), so you can use other methods which take the `dataset` parameter. #### API method ```python Historical.metadata.list_datasets( start_date: date | str | None = None, end_date: date | str | None = None, ) -> list[str] ``` #### Parameters | Parameter | Type | Required | Description | | `start_date` | date or str | optional | The inclusive UTC start date of the request range as a [Python date](https://docs.python.org/3/library/datetime.html#date-objects) or [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. If `None` then first date available. | | `end_date` | date or str | optional | The exclusive UTC end date of the request range as a [Python date](https://docs.python.org/3/library/datetime.html#date-objects) or [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. If `None` then last date available. | #### Returns `list[str]` A list of available dataset IDs. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") datasets = client.metadata.list_datasets() print(datasets) ``` #### Example response ```json [ "ARCX.PILLAR", "BATS.PITCH", "BATY.PITCH", "DBEQ.BASIC", "EDGA.PITCH", "EDGX.PITCH", "EPRL.DOM", "EQUS.MINI", "EQUS.SUMMARY", "GLBX.MDP3", "IEXG.TOPS", "IFEU.IMPACT", "IFLL.IMPACT", "IFUS.IMPACT", "MEMX.MEMOIR", "NDEX.IMPACT", "OCEA.MEMOIR", "OPRA.PILLAR", "XASE.PILLAR", "XBOS.ITCH", "XCBF.PITCH", "XCHI.PILLAR", "XCIS.TRADESBBO", "XEEE.EOBI", "XEUR.EOBI", "XNAS.BASIC", "XNAS.ITCH", "XNYS.PILLAR", "XPSX.ITCH" ] ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-list-schemas?lang=python ### Historical.metadata.list\_schemas List all available schemas for a dataset. #### API method ```python Historical.metadata.list_schemas( dataset: Dataset | str, ) -> list[str] ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | #### Returns `list[str]` A list of available data schemas. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") schemas = client.metadata.list_schemas(dataset="GLBX.MDP3") print(schemas) ``` #### Example response ```json [ "mbo", "mbp-1", "mbp-10", "tbbo", "trades", "bbo-1s", "bbo-1m", "ohlcv-1s", "ohlcv-1m", "ohlcv-1h", "ohlcv-1d", "definition", "statistics", "status" ] ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-list-fields?lang=python ### Historical.metadata.list\_fields List all fields for a particular schema and encoding. #### API method ```python Historical.metadata.list_fields( schema: Schema | str, encoding: Encoding | str, dataset: Dataset | str | None = None, ) -> list[dict[str, str]] ``` #### Parameters | Parameter | Type | Required | Description | | `schema` | Schema or str | required | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `encoding` | Encoding or str | required | The data [encoding](https://databento.com/docs/api-historical/basics/encodings.md). Must be one of 'dbn', 'csv', or 'json'. 'dbn' is recommended. | | `dataset` | Dataset or str | optional | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) used to determine which fields are relevant. If `None` then the fields for the latest DBN encoding version are returned, which may differ from a specific dataset's schema. | #### Returns `list[dict[str, str]]` A list of field details objects. | Field | Type | Description | | `name` | str | The field name. | | `type` | str | The field data type. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") fields = client.metadata.list_fields(schema="trades", encoding="dbn") print(fields) ``` #### Example response ```json [ { "name": "length", "type": "uint8_t" }, { "name": "rtype", "type": "uint8_t" }, { "name": "publisher_id", "type": "uint16_t" }, { "name": "instrument_id", "type": "uint32_t" }, { "name": "ts_event", "type": "uint64_t" }, { "name": "price", "type": "int64_t" }, { "name": "size", "type": "uint32_t" }, { "name": "action", "type": "char" }, { "name": "side", "type": "char" }, { "name": "flags", "type": "uint8_t" }, { "name": "depth", "type": "uint8_t" }, { "name": "ts_recv", "type": "uint64_t" }, { "name": "ts_in_delta", "type": "int32_t" }, { "name": "sequence", "type": "uint32_t" } ] ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-list-unit-prices?lang=python ### Historical.metadata.list\_unit\_prices List unit prices for each feed mode and data schema in US dollars per gigabyte. #### API method ```python Historical.metadata.list_unit_prices( dataset: Dataset | str, ) -> list[dict[str, Any]] ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | #### Returns `list[dict[str, Any]]` A list of maps of feed mode to schema to unit price. | Field | Type | Description | | `mode` | str | The feed mode. Will be one of "historical", "historical-streaming", or "live". | | `unit_prices` | dict[str \ | float] | A mapping of schemas to unit prices in US dollars per gigabyte. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") unit_prices = client.metadata.list_unit_prices(dataset="OPRA.PILLAR") print(unit_prices) ``` #### Example response ```json [ { "mode": "historical", "unit_prices": { "cmbp-1": 0.16, "cbbo-1s": 2.0, "cbbo-1m": 2.0, "tcbbo": 210.0, "trades": 280.0, "ohlcv-1s": 280.0, "ohlcv-1m": 280.0, "ohlcv-1h": 600.0, "ohlcv-1d": 600.0, "statistics": 11.0, "status": 5.0, "definition": 5.0 } }, { "mode": "historical-streaming", "unit_prices": { "cmbp-1": 0.16, "cbbo-1s": 2.0, "cbbo-1m": 2.0, "tcbbo": 210.0, "trades": 280.0, "ohlcv-1s": 280.0, "ohlcv-1m": 280.0, "ohlcv-1h": 600.0, "ohlcv-1d": 600.0, "statistics": 11.0, "status": 5.0, "definition": 5.0 } }, { "mode": "live", "unit_prices": { "cmbp-1": 0.2, "cbbo-1s": 2.4, "cbbo-1m": 2.4, "tcbbo": 252.0, "trades": 336.0, "ohlcv-1s": 336.0, "ohlcv-1m": 336.0, "ohlcv-1h": 720.0, "ohlcv-1d": 720.0, "statistics": 13.2, "status": 6.0, "definition": 6.0 } } ] ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-get-dataset-condition?lang=python ### Historical.metadata.get\_dataset\_condition Get the [dataset](https://databento.com/docs/api-historical/basics/datasets.md) condition from Databento. Use this method to discover data availability and quality. #### API method ```python Historical.metadata.get_dataset_condition( dataset: Dataset | str, start_date: date | str | None = None, end_date: date | str | None = None, ) -> list[dict[str, str | None]] ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `start_date` | date or str | optional | The inclusive UTC start date of the request range as a [Python date](https://docs.python.org/3/library/datetime.html#date-objects) or [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. If `None` then first date available. | | `end_date` | date or str | optional | The inclusive UTC end date of the request range as a [Python date](https://docs.python.org/3/library/datetime.html#date-objects) or [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. If `None` then last date available. | #### Returns `list[dict[str, str | None]]` A list of conditions per date. | Field | Type | Description | | `date` | str | The day of the described data, as an ISO 8601 date string. | | `condition` | str | The condition code describing the quality and availability of the data on the given day. Possible values are listed below. | | `last_modified_date` | str or None | The date when any schema in the dataset on the given day was last generated or modified, as an ISO 8601 date string. Will be `None` when `condition` is 'missing'. | Possible values for condition: - `available`: the data is available with no known issues - `degraded`: the data is available, but there may be missing data or other correctness issues - `pending`: the data is not yet available, but may be available soon - `missing`: the data is not available #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") conditions = client.metadata.get_dataset_condition( dataset="GLBX.MDP3", start_date="2022-06-06", end_date="2022-06-10", ) print(conditions) ``` #### Example response ```json [ { "date": "2022-06-06", "condition": "available", "last_modified_date": "2024-05-18" }, { "date": "2022-06-07", "condition": "available", "last_modified_date": "2024-05-21" }, { "date": "2022-06-08", "condition": "available", "last_modified_date": "2024-05-21" }, { "date": "2022-06-09", "condition": "available", "last_modified_date": "2024-05-21" }, { "date": "2022-06-10", "condition": "available", "last_modified_date": "2024-05-22" } ] ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-get-dataset-range?lang=python ### Historical.metadata.get\_dataset\_range Get the available range for the [dataset](https://databento.com/docs/api-historical/basics/datasets.md) given the user's entitlements. Use this method to discover data availability. The `start` and `end` values in the response can be used with the [timeseries.get\_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md) and [batch.submit\_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) endpoints. This endpoint will return the `start` and `end` timestamps over the entire dataset as well as the per-schema `start` and `end` timestamps under the `schema` key. In some cases, a schema's availability is a subset of the entire dataset availability. #### API method ```python Historical.metadata.get_dataset_range( dataset: Dataset | str, ) -> dict[str, str | dict[str, dict[str, str]]] ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | #### Returns `dict[str, str | dict[str, dict[str, str]]]` The available range for the dataset. | Field | Type | Description | | `start` | str | The inclusive start of the available range as an [ISO 8601 timestamp](https://www.iso.org/iso-8601-date-and-time-format.html). | | `end` | str | The exclusive end of the available range as an [ISO 8601 timestamp](https://www.iso.org/iso-8601-date-and-time-format.html). | | `schema` | dict[str, dict[str, str]] | A mapping of schema names to per-schema `start` and `end` timestamps. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") available_range = client.metadata.get_dataset_range(dataset="XNAS.BASIC") print(available_range) ``` #### Example response ```json { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z", "schema": { "mbo": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "mbp-1": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "mbp-10": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "bbo-1s": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "bbo-1m": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "tbbo": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "trades": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "ohlcv-1s": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "ohlcv-1m": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "ohlcv-1h": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "ohlcv-1d": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "definition": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "statistics": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "status": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" }, "imbalance": { "start":"2018-05-01T00:00:00.000000000Z", "end":"2025-01-30T00:00:00.000000000Z" } } } ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-get-record-count?lang=python ### Historical.metadata.get\_record\_count Get the record count of the time series data query. This method may not be accurate for time ranges that are not discrete multiples of 10 minutes, potentially over-reporting the number of records in such cases. The definition schema is only accurate for discrete multiples of 24 hours. #### API method ```python Historical.metadata.get_record_count( dataset: Dataset | str, start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, symbols: Iterable[str | int] | str | int | None = None, schema: Schema | str = "trades", stype_in: SType | str = "raw_symbol", limit: int | None = None, ) -> int ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | | `symbols` | Iterable[str \ | int] or str or int | optional | The product symbols to filter for. Takes up to 2,000 symbols per request. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols. | | `schema` | Schema or str, default 'trades' | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. | | `limit` | int | optional | The maximum number of records to return. If `None` then no limit. | #### Returns `int` The record count. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") count = client.metadata.get_record_count( dataset="GLBX.MDP3", symbols=["ESM2"], schema="mbo", start="2022-01-06", ) print(count) ``` #### Example response ```plaintext 1329107 ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-get-billable-size?lang=python ### Historical.metadata.get\_billable\_size Get the billable uncompressed raw binary size for historical streaming or batched files. This method may not be accurate for time ranges that are not discrete multiples of 10 minutes, potentially over-reporting the size in such cases. The definition schema is only accurate for discrete multiples of 24 hours. > **Info** > > The amount billed will be based on the actual amount of bytes sent; see our [pricing documentation](https://databento.com/docs/faqs/usage-pricing-and-data-credits.md) for more details. #### API method ```python Historical.metadata.get_billable_size( dataset: Dataset | str, start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, symbols: Iterable[str | int] | str | int | None = None, schema: Schema | str = "trades", stype_in: Stype | str = "raw_symbol", limit: int | None = None, ) -> int ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | | `symbols` | Iterable[str \ | int] or str or int | optional | The product symbols to filter for. Takes up to 2,000 symbols per request. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols. | | `schema` | Schema or str, default 'trades' | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `stype_in` | SType or str, default 'raw_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw_symbol', 'instrument_id', 'parent', or 'continuous'. | | `limit` | int | optional | The maximum number of records to return. If `None` then no limit. | #### Returns `int` The size in number of bytes used for billing. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") size = client.metadata.get_billable_size( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06T00:00:00", end="2022-06-10T12:10:00", ) print(size) ``` #### Example response ```plaintext 99219648 ``` --- Source: https://databento.com/docs/api-historical/metadata/metadata-get-cost?lang=python ### Historical.metadata.get\_cost Get the cost in US dollars for a historical streaming or batch download request. This cost respects any discounts provided by flat rate plans. This method may not be accurate for time ranges that are not discrete multiples of 10 minutes, potentially over-reporting the cost in such cases. The definition schema is only accurate for discrete multiples of 24 hours. > **Info** > > The amount billed will be based on the actual amount of bytes sent; see our [pricing documentation](https://databento.com/docs/faqs/usage-pricing-and-data-credits.md) for more details. #### API method ```python Historical.metadata.get_cost( dataset: Dataset | str, start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, symbols: Iterable[str | int] | str | int | None = None, schema: Schema | str = "trades", stype_in: SType | str = "raw_symbol", limit: int | None = None, ) -> float ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | | `symbols` | Iterable[str \ | int] or str or int | optional | The product symbols to filter for. Takes up to 2,000 symbols per request. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols. | | `schema` | Schema or str, default 'trades' | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `stype_in` | SType or str, default 'raw_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. | | `limit` | int | optional | The maximum number of records to return. If `None` then no limit. | #### Returns `float` The cost in US dollars. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") cost = client.metadata.get_cost( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06T00:00:00", end="2022-06-10T12:10:00", ) print(cost) ``` #### Example response ```plaintext 2.587353944778 ``` --- Source: https://databento.com/docs/api-historical/timeseries?lang=python ## Time series --- Source: https://databento.com/docs/api-historical/timeseries/timeseries-get-range?lang=python ### Historical.timeseries.get\_range Makes a streaming request for time series data from Databento. This is the primary method for getting historical market data, instrument definitions, and status data directly into your application. This method only returns after all of the data has been downloaded, which can take a long time. For large requests, consider using [batch.submit\_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) instead. #### API method ```python Historical.timeseries.get_range( dataset: Dataset | str, start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, symbols: Iterable[str | int] | str | int | None = None, schema: Schema | str = "trades", stype_in: SType | str = "raw_symbol", stype_out: SType | str = "instrument_id", limit: int | None = None, path: PathLike[str] | str | None = None, ) -> DBNStore ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | | `symbols` | Iterable[str \ | int] or str or int | optional | The product symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols. | | `schema` | Schema or str, default 'trades' | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. | | `stype_out` | SType or str, default 'instrument\_id' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of output symbols. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. Must be a valid symbology combination with `stype_in`. See [symbology combinations](https://databento.com/docs/conventions/symbology.md#supported-symbology-combinations). | | `limit` | int | optional | The maximum number of records to return. If `None` then no limit. | | `path` | PathLike[str] or str | optional | The file path to stream the data to. It is recommended to use the ".dbn.zst" suffix. | #### Returns A [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. A full list of fields for each schema is available through [Historical.metadata.list\_fields](https://databento.com/docs/api-historical/metadata/metadata-list-fields.md). #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06T00:00:00", end="2022-06-10T00:10:00", limit=1, ) df = data.to_df() print(df.iloc[0].to_json(indent=4)) ``` #### Example response ```plaintext { "ts_event":1654473600070, "rtype":0, "publisher_id":1, "instrument_id":3403, "action":"T", "side":"A", "depth":0, "price":4108.5, "size":1, "flags":0, "ts_in_delta":18681, "sequence":157862, "symbol":"ESM2" } ``` --- Source: https://databento.com/docs/api-historical/timeseries/timeseries-get-range-async?lang=python ### Historical.timeseries.get\_range\_async Asynchronously request a historical time series data stream from Databento. Primary method for getting historical intraday market data, daily data, instrument definitions and market status data directly into your application. This method only returns after all of the data has been downloaded, which can take a long time. For large requests, consider using [batch.submit\_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) instead. #### API method ```python Historical.timeseries.get_range_async( dataset: Dataset | str, start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, symbols: Iterable[str | int] | str | int | None = None, schema: Schema | str = "trades", stype_in: SType | str = "raw_symbol", stype_out: SType | str = "instrument_id", limit: int | None = None, path: PathLike[str] | str | None = None, ) -> Awaitable[DBNStore] ``` > **Info** > > This method is a [coroutine](https://docs.python.org/3/library/asyncio-task.html#coroutines) and must be used with an [await](https://docs.python.org/3/reference/expressions.html#await) expression. #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | | `symbols` | Iterable[str \ | int] or str or int | optional | The product symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols. | | `schema` | Schema or str, default 'trades' | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. | | `stype_out` | SType or str, default 'instrument\_id' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of output symbols. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. Must be a valid symbology combination with `stype_in`. See [symbology combinations](https://databento.com/docs/conventions/symbology.md#supported-symbology-combinations). | | `limit` | int | optional | The maximum number of records to return. If `None` then no limit. | | `path` | PathLike[str] or str | optional | The file path to stream the data to. It is recommended to use the ".dbn.zst" suffix. | #### Returns A [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. A full list of fields for each schema is available through [Historical.metadata.list\_fields](https://databento.com/docs/api-historical/metadata/metadata-list-fields.md). #### Example usage ```python import asyncio import databento as db client = db.Historical("$YOUR_API_KEY") coro = client.timeseries.get_range_async( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06T00:00:00", end="2022-06-10T00:10:00", limit=1, ) data = asyncio.run(coro) df = data.to_df() print(df.iloc[0].to_json(indent=4)) ``` #### Example response ```plaintext { "ts_event":1654473600070, "rtype":0, "publisher_id":1, "instrument_id":3403, "action":"T", "side":"A", "depth":0, "price":4108.5, "size":1, "flags":0, "ts_in_delta":18681, "sequence":157862, "symbol":"ESM2" } ``` --- Source: https://databento.com/docs/api-historical/symbology?lang=python ## Symbology --- Source: https://databento.com/docs/api-historical/symbology/symbology-resolve?lang=python ### Historical.symbology.resolve Resolve a list of symbols from an input [symbology type](https://databento.com/docs/api-historical/basics/symbology.md), to an output symbology type. Take, for example, a raw symbol to an instrument ID: `ESM2` → `3403`. #### API method ```python Historical.symbology.resolve( dataset: Dataset | str, symbols: Iterable[str | int] | str | int, stype_in: SType | str, stype_out: SType | str, start_date: date | str, end_date: date | str | None = None, ) -> dict[str, Any] ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `symbols` | Iterable[str \ | int] or str or int | required | The symbols to resolve. Takes up to 2,000 symbols per request. Use `'ALL_SYMBOLS'` to request **all** symbols (not available for every dataset). | | `stype_in` | SType or str | required | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. | | `stype_out` | SType or str | required | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of output symbols. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. Must be a valid symbology combination with `stype_in`. See [symbology combinations](https://databento.com/docs/conventions/symbology.md#supported-symbology-combinations). | | `start_date` | date or str | required | The inclusive UTC start date of the request range as a [Python date](https://docs.python.org/3/library/datetime.html#date-objects) or [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. | | `end_date` | date or str | optional | The exclusive UTC end date of the request range as a [Python date](https://docs.python.org/3/library/datetime.html#date-objects) or [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | #### Returns `dict[str, Any]` The results for the symbology resolution. > **See also** > > For more information on symbology resolution, visit our [symbology documentation](https://databento.com/docs/conventions/symbology.md#symbology-resolve-endpoint). | Field | Type | Description | | `result` | dict[str, list[dict[str, str]]] | The symbology mapping result. For each requested symbol, a list of symbology mappings is provided. | | `symbols` | list[str] | The requested symbols. | | `stype_in` | str | The requested input symbology type. | | `stype_out` | str | The requested output symbology type. | | `start_date` | str | The requested symbology start date as an [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. | | `end_date` | str | The requested symbology end date as an [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) date string. | | `partial` | list[str] | The list of symbols, if any, that partially resolved inside the start date and end date interval. | | `not_found` | list[str] | The list of symbols, if any, that failed to resolve inside the start date and end date interval. | | `message` | str | A short message indicating the overall symbology result. Can be one of: "OK", or "Partially resolved", or "Not found" | | `status` | int | A numerical status field indicating the overall symbology result. Can be one of: 0 (OK), 1 (Partially resolved), or 2 (Not found). | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") result = client.symbology.resolve( dataset="GLBX.MDP3", symbols=["ESM2"], stype_in="raw_symbol", stype_out="instrument_id", start_date="2022-06-01", end_date="2022-06-30", ) print(result) ``` #### Example response ```json { "result": { "ESM2": [ { "d0": "2022-06-01", "d1": "2022-06-26", "s": "3403" } ] }, "symbols": [ "ESM2" ], "stype_in": "raw_symbol", "stype_out": "instrument_id", "start_date": "2022-06-01", "end_date": "2022-06-30", "partial": [], "not_found": [], "message": "OK", "status": 0 } ``` --- Source: https://databento.com/docs/api-historical/batch?lang=python ## Batch downloads Batch downloads allow you to download flat files directly from within your portal. For more information, see [Streaming vs. batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md). --- Source: https://databento.com/docs/api-historical/batch/batch-submit-job?lang=python ### Historical.batch.submit\_job Make a batch download job request for flat files. Once a request is submitted, our system processes the request and prepares the batch files in the background. The status of your request and the files can be accessed from the [Download center]($PLATFORM_HOSTNAME/portal/download-center) from your user portal. This method takes longer than a streaming request, but is advantageous for larger requests as it supports delivery mechanisms that allow multiple accesses of the data without additional cost for each subsequent download after the first. Related: [batch.list\_jobs](https://databento.com/docs/api-historical/batch/batch-list-jobs.md), [batch.get\_job\_details](https://databento.com/docs/api-historical/batch/batch-get-job-details.md). #### API method ```python Historical.batch.submit_job( dataset: Dataset | str, symbols: Iterable[str | int] | str | int, schema: Schema | str, start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, encoding: Encoding | str = "dbn", compression: Compression | str = "zstd", pretty_px: bool = False, pretty_ts: bool = False, map_symbols: bool | None = None, split_symbols: bool = False, split_duration: Duration | str = "day", split_size: int | None = None, delivery: Delivery | str = "download", stype_in: SType | str = "raw_symbol", stype_out: SType | str = "instrument_id", limit: int | None = None, ) -> dict[str, Any] ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `symbols` | Iterable[str \ | int] or str or int | required | The product symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` then will select **all** symbols. | | `schema` | Schema or str | required | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Defaults to the [forward filled](https://databento.com/docs/conventions/common-fields-enums-types.md#forward-filling-end-parameters) value of `start` based on the resolution provided. | | `encoding` | Encoding or str | optional | The data [encoding](https://databento.com/docs/api-historical/basics/encodings.md). Must be one of 'dbn', 'csv', 'json'. For fastest transfer speed, 'dbn' is recommended. | | `compression` | Compression or str | optional | The data [compression](https://databento.com/docs/api-historical/basics/compression.md) mode. Must be either 'zstd', 'none', or None. For fastest transfer speed, 'zstd' is recommended. | | `pretty_px` | bool, default False | optional | If prices should be formatted to the correct scale (using the fixed-precision scalar 1e-9). Only applicable for 'csv' or 'json' encodings. | | `pretty_ts` | bool, default False | optional | If timestamps should be formatted as ISO 8601 strings. Only applicable for 'csv' or 'json' encodings. | | `map_symbols` | bool | optional | If a symbol field should be included with each text-encoded record. If `None`, will default to `True` for `csv` and `json` encodings and `False` for `dbn`. | | `split_symbols` | bool, default False | optional | If files should be split by raw symbol. Cannot be requested with `'ALL_SYMBOLS'`. Cannot be used with `limit`. | | `split_duration` | Duration or str, default 'day' | optional | The maximum time duration before batched data is split into multiple files. Must be one of 'day', 'week', 'month', 'year', or 'none'. A week starts on Sunday UTC. | | `split_size` | int | optional | The maximum size (in bytes) of each batched data file before being split. Must be an integer between 1e9 and 10e9 inclusive (1GB - 10GB). Defaults to no split size. | | `delivery` | Delivery or str, default 'download' | optional | The delivery mechanism for the batched data files once processed. Only 'download' is supported at this time. | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. | | `stype_out` | SType or str, default 'instrument\_id' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of output symbols. Must be one of 'raw\_symbol', 'instrument\_id', 'parent', or 'continuous'. Must be a valid symbology combination with `stype_in`. See [symbology combinations](https://databento.com/docs/conventions/symbology.md#supported-symbology-combinations). | | `limit` | int | optional | The maximum number of records to return. If `None` then no limit. Cannot be used with `split_symbols`. | #### Returns `dict[str, Any]` The description of the submitted batch job. | Field | Type | Description | | `id` | str | The unique job ID for the request. | | `user_id` | str | The user ID of the user who made the request. | | `api_key` | str or None | The API key name for the request (if Basic Auth was used). | | `cost_usd` | float or None | The cost of the job in US dollars (`None` until the job is done processing). | | `dataset` | str | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). | | `symbols` | str | The list of symbols specified in the request. | | `stype_in` | str | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. | | `stype_out` | str | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of output symbols. | | `schema` | str | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). | | `start` | str | The ISO 8601 timestamp start of request time range (inclusive). | | `end` | str | The ISO 8601 timestamp end of request time range (exclusive). | | `limit` | int or None | The maximum number of records to return. | | `encoding` | str | The data [encoding](https://databento.com/docs/api-historical/basics/encodings.md). | | `compression` | str | The data [compression](https://databento.com/docs/api-historical/basics/compression.md) mode. | | `pretty_px` | bool | If prices are formatted to the correct scale (using the fixed-precision scalar 1e-9). | | `pretty_ts` | bool | If timestamps are formatted as ISO 8601 strings. | | `map_symbols` | bool | If a symbol field is included with each text-encoded record. | | `split_symbols` | bool | If files are split by raw symbol. | | `split_duration` | str | The maximum time interval for an individual file before splitting into multiple files. | | `split_size` | int or None | The maximum size for an individual file before splitting into multiple files. | | `packaging` | str or None | The packaging method of the batch data, one of 'none', 'zip', or 'tar' (deprecated). | | `delivery` | str | The delivery mechanism of the batch data. Only 'download' is supported at this time. | | `record_count` | int or None | The number of data records (`None` until the job is processed). | | `billed_size` | int or None | The size of the raw binary data used to process the batch job (used for billing purposes). | | `actual_size` | int or None | The total size of the result of the batch job after splitting and compression. | | `package_size` | int or None | The total size of the result of the batch job after any packaging (including metadata). | | `state` | str | The current status of the batch job. One of 'queued', 'processing', 'done', or 'expired'. | | `ts_received` | str | The ISO 8601 timestamp when Databento received the batch job. | | `ts_queued` | str or None | The ISO 8601 timestamp when the batch job was queued. | | `ts_process_start` | str or None | The ISO 8601 timestamp when the batch job began processing (if it's begun). | | `ts_process_done` | str or None | The ISO 8601 timestamp when the batch job finished processing (if it's finished). | | `ts_expiration` | str or None | The ISO 8601 timestamp when the batch job will expire from the [Download center](https://databento.com/docs/portal/download-center.md). | | `progress` | int or None | The progress percentage (0-100). Always `None` in submit responses. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") details = client.batch.submit_job( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", encoding="dbn", start="2022-06-06T12:00:00+00:00", end="2022-06-10T00:00:00+00:00", ) print(details) ``` #### Example response ```plaintext { "id": "GLBX-20221217-MN5S5S4WAS", "user_id": "NBPDLF33", "api_key": "prod-001", "cost_usd": None, "dataset": "GLBX.MDP3", "symbols": "ESM2", "stype_in": "raw_symbol", "stype_out": "instrument_id", "schema": "trades", "start": "2022-06-06T12:00:00.000000000Z", "end": "2022-06-10T00:00:00.000000000Z", "limit": None, "encoding": "dbn", "compression": "zstd", "pretty_px": False, "pretty_ts": False, "map_symbols": False, "split_symbols": False, "split_duration": "day", "split_size": None, "packaging": None, "delivery": "download", "record_count": None, "billed_size": None, "actual_size": None, "package_size": None, "state": "queued", "ts_received": "2022-12-17T00:36:37.844913000Z", "ts_queued": None, "ts_process_start": None, "ts_process_done": None, "ts_expiration": None } ``` --- Source: https://databento.com/docs/api-historical/batch/batch-list-jobs?lang=python ### Historical.batch.list\_jobs > **Info** > > The response format of this endpoint was changed on 2026-09-29. > Users can obtain the deprecated response format by specifying `short=False` until 2026-10-27. > For information about this change, see this [announcement](https://databento.com/blog/batch-api-changes-2026-08). List batch job details for the user account. The job details will be sorted in order of `ts_received`. To obtain the full details for a job, use [batch.get\_job\_details](https://databento.com/docs/api-historical/batch/batch-get-job-details.md), which returns the full set of fields for a single job. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.list_jobs( states: Iterable[JobState | str] | JobState | str | None = "queued,processing,done", since: pd.Timestamp | datetime | date | str | int | None = None, short: bool | None = None, ) -> list[dict[str, Any]] ``` #### Parameters | Parameter | Type | Required | Description | | `states` | Iterable[JobState \ | str] or JobState or str | optional | The filter for job states as a list of comma separated values. Can include 'queued', 'processing', 'done', and 'expired'. Defaults to all except 'expired'. | | `since` | pd.Timestamp, datetime, date, str, or int | optional | The filter for timestamp submitted (will not include jobs prior to this). Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `short` | bool | optional | If False, return the deprecated long form response for each job, which includes all job fields reported by [batch.get\_job\_details](https://databento.com/docs/api-historical/batch/batch-get-job-details.md). Defaults to the short form. | #### Returns `list[dict[str, Any]]` A list of batch job details. | Field | Type | Description | | `id` | str | The unique job ID for the request. | | `state` | str | The current status of the batch job. One of 'queued', 'processing', 'done', or 'expired'. | | `ts_received` | str | The ISO 8601 timestamp when Databento received the batch job. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") jobs = client.batch.list_jobs( states=["queued", "processing", "done"], since="2022-06-01", ) print(jobs) ``` #### Example response ```plaintext [ { "id": "GLBX-20221126-DBVXWPJJQN", "state": "done", "ts_received": "2022-11-26T09:23:17.519708000Z" }, ... ``` --- Source: https://databento.com/docs/api-historical/batch/batch-list-files?lang=python ### Historical.batch.list\_files List files for a batch job. This will include all data files and [support files](https://databento.com/docs/portal/download-center.md#support-files). Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.list_files( job_id: str, ) -> list[dict[str, Any]] ``` #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job identifier. | #### Returns `list[dict[str, Any]]` The file details for the batch job. | Field | Type | Description | | `filename` | str | The file name. | | `size` | int | The size of the file in bytes. | | `hash` | str | The SHA256 hash of the file. | | `urls` | dict | A map of download protocol to URL. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") files = client.batch.list_files(job_id="GLBX-20220610-5DEFXVTMSM") print(files) ``` #### Example response ```plaintext [ { "filename": "metadata.json", "size": 1102, "hash": "sha256:0168d53e1705b69b1d6407f10bb3ab48aac492fa0f68f863cc9b092931cc67a7", "urls": { "https": "https://api.databento.com/v0/batch/download/46PCMCVF/GLBX-20230203-WF9WJYSCDU/metadata.json", "ftp": "ftp://ftp.databento.com/46PCMCVF/GLBX-20230203-WF9WJYSCDU/metadata.json", } }, { "filename": "glbx-mdp3-20220610.mbo.csv.zst", "size": 21832, "hash": "sha256:1218930af153b4953632216044ef87607afa467fc7ab7fbb1f031fceacf9d52a", "urls": { "https": "https://api.databento.com/v0/batch/download/46PCMCVF/GLBX-20230203-WF9WJYSCDU/glbx-mdp3-20220610.mbo.csv.zst", "ftp": "ftp://ftp.databento.com/46PCMCVF/GLBX-20230203-WF9WJYSCDU/glbx-mdp3-20220610.mbo.csv.zst", } } ] ``` --- Source: https://databento.com/docs/api-historical/batch/batch-get-job-details?lang=python ### Historical.batch.get\_job\_details Get the full job details of a batch job. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.get_job_details( job_id: str, ) -> dict[str, Any] ``` #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job ID. This can be found in the [download center](https://databento.com/portal/download-center). | #### Returns `dict[str, Any]` The batch job details. See [batch.submit\_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md#returns) for a detailed list of returned values. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") job_details = client.batch.get_job_details( job_id="GLBX-20221126-DBVXWPJJQN", ) print(job_details) ``` #### Example response ```plaintext { "id": "GLBX-20221126-DBVXWPJJQN", "user_id": "NBPDLF33", "api_key": "prod-001", "cost_usd": 23.6454, "dataset": "GLBX.MDP3", "symbols": "ZC.FUT,ES.FUT", "stype_in": "parent", "stype_out": "instrument_id", "schema": "mbo", "start": "2022-10-24T00:00:00.000000000Z", "end": "2022-11-24T00:00:00.000000000Z", "limit": None, "encoding": "csv", "compression": "zstd", "pretty_px": False, "pretty_ts": False, "map_symbols": False, "split_symbols": False, "split_duration": "day", "split_size": None, "packaging": None, "delivery": "download", "record_count": 412160224, "billed_size": 23080972544, "actual_size": 8144595219, "package_size": 8144628684, "state": "done", "ts_received": "2022-11-26T09:23:17.519708000Z", "ts_queued": "2022-12-03T14:34:57.897790000Z", "ts_process_start": "2022-12-03T14:35:00.495167000Z", "ts_process_done": "2022-12-03T14:48:15.710116000Z", "ts_expiration": "2023-01-02T14:48:15.710116000Z", "progress": 100 } ``` --- Source: https://databento.com/docs/api-historical/batch/batch-download?lang=python ### Historical.batch.download Download a batch job or a specific file to `{output_dir}/{job_id}/`. Will automatically generate any necessary directories if they do not already exist. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.download( job_id: str, output_dir: PathLike[str] | str | None = None, filename_to_download: str | None = None, keep_zip: bool = False, ) -> list[Path] ``` #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job identifier. | | `output_dir` | PathLike[str] or str | optional | The directory to download the file(s) to. If `None`, defaults to the current working directory. | | `filename_to_download` | str | optional | The specific file to download. If `None` then will download all files for the batch job. | | `keep_zip` | bool, default False | optional | If `True`, and `filename_to_download` is `None`, all job files will be saved as a .zip archive in the `output_dir`. | #### Returns `list[Path]` A list of paths to the downloaded files. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") # Download all files for the batch job client.batch.download( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", ) # Alternatively, you can download a specific file client.batch.download( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", filename_to_download="metadata.json", ) ``` --- Source: https://databento.com/docs/api-historical/batch/batch-download-async?lang=python ### Historical.batch.download\_async Asynchronously download a batch job or a specific file to `{output_dir}/{job_id}/`. Will automatically generate any necessary directories if they do not already exist. Related: [Download center]($PLATFORM_HOSTNAME/portal/download-center). #### API method ```python Historical.batch.download_async( job_id: str, output_dir: PathLike[str] | str | None = None, filename_to_download: str | None = None, keep_zip: bool = False, ) -> Awaitable[list[Path]] ``` > **Info** > > This method is a [coroutine](https://docs.python.org/3/library/asyncio-task.html#coroutines) and must be used with an [await](https://docs.python.org/3/reference/expressions.html#await) expression. #### Parameters | Parameter | Type | Required | Description | | `job_id` | str | required | The batch job identifier. | | `output_dir` | PathLike[str] or str | optional | The directory to download the file(s) to. If `None`, defaults to the current working directory. | | `filename_to_download` | str | optional | The specific file to download. If `None` then will download all files for the batch job. | | `keep_zip` | bool, default False | optional | If `True`, and `filename_to_download` is `None`, all job files will be saved as a .zip archive in the `output_dir`. | #### Returns `list[Path]` A list of paths to the downloaded files. #### Example usage ```python import asyncio import databento as db client = db.Historical("$YOUR_API_KEY") # Download all files for the batch job coro = client.batch.download_async( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", ) asyncio.run(coro) # Alternatively, you can download a specific file coro = client.batch.download_async( job_id="GLBX-20220610-5DEFXVTMSM", output_dir="my_data/", filename_to_download="metadata.json", ) asyncio.run(coro) ``` --- Source: https://databento.com/docs/api-historical/helpers?lang=python ## Helpers --- Source: https://databento.com/docs/api-historical/helpers/dbn-store?lang=python ### DBNStore The `DBNStore` object is an I/O helper class for working with [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md)-encoded data. Typically, this object is created when performing historical requests. However, it can be created directly using DBN data on disk or in memory using provided factory methods: - [DBNStore.from\_bytes](https://databento.com/docs/api-historical/helpers/dbn-store-from-bytes.md) - [DBNStore.from\_file](https://databento.com/docs/api-historical/helpers/dbn-store-from-file.md) #### Attributes | Parameter | Type | Description | | `nbytes` | int | The size of the data in bytes. | | `raw` | bytes | The raw data from the I/O stream. | | `metadata` | Metadata | The metadata header for the `DBNStore`. | | `dataset` | str | The dataset ID. | | `schema` | Schema or None | The data record schema. If `None`, the `DBNStore` may contain multiple schemas. | | `symbols` | list[str] | The query symbols for the data. | | `stype_in` | SType or None | The query input symbology type for the data. If `None`, the `DBNStore` may contain mixed STypes. | | `stype_out` | SType | The query output symbology type for the data. | | `start` | pd.Timestamp | The query start for the data as a [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html). | | `end` | pd.Timestamp or None | The query end for the data as a [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html). If `None`, the `DBNStore` data was created without a known end time. | | `limit` | int or None | The query limit for the data. | | `encoding` | Encoding | The data encoding. | | `compression` | Compression | Return the data compression format (if any). | | `mappings` | dict[str, list[dict[str, Any]]] | Return the symbology mappings for the data. | | `symbology` | dict[str, Any] | Return the symbology resolution information for the data. | --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-from-bytes?lang=python ### DBNStore.from\_bytes Read data from a DBN byte stream. #### API method ```python DBNStore.from_bytes( data: BytesIO | bytes | IO[bytes], ) -> DBNStore ``` #### Parameters | Parameter | Type | Required | Description | | `data` | BytesIO or bytes or IO[bytes] | required | The bytes to read from. | #### Returns A [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06", ) # Save streamed data to .dbn.zst path = "GLBX-ESM2-20220606.trades.dbn.zst" data.to_file(path) # Open saved data as a byte stream. with open(path, "rb") as saved: stored_data = db.DBNStore.from_bytes(saved) # Convert to dataframe df = stored_data.to_df() print(df.head()) ``` #### Example response ```plaintext ts_event rtype publisher_id instrument_id action ... size flags ts_in_delta sequence symbol ts_recv ... 2022-06-06 00:00:00.070314216+00:00 2022-06-06 00:00:00.070033767+00:00 0 1 3403 T ... 1 0 18681 157862 ESM2 2022-06-06 00:00:00.090544076+00:00 2022-06-06 00:00:00.089830441+00:00 0 1 3403 T ... 1 0 18604 157922 ESM2 2022-06-06 00:00:00.807324169+00:00 2022-06-06 00:00:00.807018955+00:00 0 1 3403 T ... 4 0 18396 158072 ESM2 2022-06-06 00:00:01.317722490+00:00 2022-06-06 00:00:01.317385867+00:00 0 1 3403 T ... 1 0 22043 158111 ESM2 2022-06-06 00:00:01.317736158+00:00 2022-06-06 00:00:01.317385867+00:00 0 1 3403 T ... 7 0 17280 158112 ESM2 [5 rows x 13 columns] ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-from-file?lang=python ### DBNStore.from\_file Read data from a DBN file. > **See also** > > `databento.read_dbn` is an alias for `DBNStore.from_file`. #### API method ```python DBNStore.from_file( path: PathLike[str] | str, ) -> DBNStore ``` #### Parameters | Parameter | Type | Required | Description | | `path` | PathLike[str] or str | required | The file path to read from. | #### Returns A [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06", ) # Save streamed data to .dbn.zst path = "GLBX-ESM2-20220606.trades.dbn.zst" data.to_file(path) # Read saved .dbn.zst stored_data = db.DBNStore.from_file(path) # Convert to dataframe df = stored_data.to_df() print(df.head()) ``` #### Example response ```plaintext ts_event rtype publisher_id instrument_id action ... size flags ts_in_delta sequence symbol ts_recv ... 2022-06-06 00:00:00.070314216+00:00 2022-06-06 00:00:00.070033767+00:00 0 1 3403 T ... 1 0 18681 157862 ESM2 2022-06-06 00:00:00.090544076+00:00 2022-06-06 00:00:00.089830441+00:00 0 1 3403 T ... 1 0 18604 157922 ESM2 2022-06-06 00:00:00.807324169+00:00 2022-06-06 00:00:00.807018955+00:00 0 1 3403 T ... 4 0 18396 158072 ESM2 2022-06-06 00:00:01.317722490+00:00 2022-06-06 00:00:01.317385867+00:00 0 1 3403 T ... 1 0 22043 158111 ESM2 2022-06-06 00:00:01.317736158+00:00 2022-06-06 00:00:01.317385867+00:00 0 1 3403 T ... 7 0 17280 158112 ESM2 [5 rows x 13 columns] ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-reader?lang=python ### DBNStore.reader Return an I/O reader for the data. #### API method ```python DBNStore.reader() -> IO[bytes] ``` #### Returns A raw [IO stream](https://docs.python.org/3/library/io.html#io.RawIOBase) for reading the DBNStore data. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") bento = client.timeseries.get_range( dataset="GLBX.MDP3", symbols="ZWZ3", start="2022-06-06", ) # Create an IO stream reader reader = bento.reader ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-replay?lang=python ### DBNStore.replay Replay data by passing records sequentially to the given callback. Refer to the [List of fields by schema](https://databento.com/docs/schemas/whats-a-schema.md) article for documentation on the fields contained with each record type. #### API method ```python DBNStore.replay( callback: Callable[[DBNRecord], None], ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `callback` | Callable | required | The callback function or method to be dispatched on every event. | #### Returns `None` #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], start="2022-06-06", ) def print_large_trades(trade): size = getattr(trade, "size", 0) if size >= 200: print(trade) data.replay(print_large_trades) ``` #### Example response ```plaintext TradeMsg(ts_recv=1654524078342408839, pretty_ts_recv='2022-06-06T14:01:18.342408839Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=3403, ts_event=1654524078339857609, pretty_ts_event='2022-06-06T14:01:18.339857609Z', action='T', side='B', depth=0, price=4164000000000, pretty_price=4164, size=291, flags=0, ts_in_delta=20352, sequence=3605032) TradeMsg(ts_recv=1654524133737794739, pretty_ts_recv='2022-06-06T14:02:13.737794739Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=3403, ts_event=1654524133736900455, pretty_ts_event='2022-06-06T14:02:13.736900455Z', action='T', side='B', depth=0, price=4160000000000, pretty_price=4160, size=216, flags=0, ts_in_delta=28024, sequence=3659203) TradeMsg(ts_recv=1654538295589900967, pretty_ts_recv='2022-06-06T17:58:15.589900967Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=3403, ts_event=1654538295588752739, pretty_ts_event='2022-06-06T17:58:15.588752739Z', action='T', side='B', depth=0, price=4140000000000, pretty_price=4140, size=200, flags=0, ts_in_delta=21708, sequence=10031624) ``` --- Source: https://databento.com/docs/api-historical/helpers/request-full-definitions?lang=python ### DBNStore.request\_full\_definitions Request for full instrument [Definition](https://databento.com/docs/schemas/instrument-definitions.md)(s) for all symbols based on the metadata properties. This is useful for retrieving the instrument definitions for saved DBN data. A [timeseries.get\_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md) request is made to obtain the definitions data which will incur a cost. #### API method ```python DBNStore.request_full_definitions( client: Historical, path: PathLike[str] | str | None = None, ) -> DBNStore ``` #### Parameters | Parameter | Type | Required | Description | | `client` | Historical | required | The historical client to use for the request (contains the API key). | | `path` | PathLike[str] or str | optional | The path to stream the data to on disk (will then return a `DBNStore`). | #### Returns A [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. A full list of fields for each schema is available through [Historical.metadata.list\_fields](https://databento.com/docs/api-historical/metadata/metadata-list-fields.md). #### Example usage ```python import databento as db client = db.Historical( key="$YOUR_API_KEY", ) trades = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ES.FUT"], stype_in="parent", schema="trades", start="2022-06-06", ) definitions = trades.request_full_definitions(client).to_df() definitions = definitions.sort_values(["expiration", "symbol"]).set_index("expiration") print(definitions[["symbol"]]) ``` #### Example response ```plaintext symbol expiration 2022-06-17 13:30:00+00:00 ESM2 2022-06-17 13:30:00+00:00 ESM2-ESH3 2022-06-17 13:30:00+00:00 ESM2-ESM3 2022-06-17 13:30:00+00:00 ESM2-ESU2 2022-06-17 13:30:00+00:00 ESM2-ESZ2 2022-09-16 13:30:00+00:00 ESU2 2022-09-16 13:30:00+00:00 ESU2-ESH3 2022-09-16 13:30:00+00:00 ESU2-ESM3 2022-09-16 13:30:00+00:00 ESU2-ESU3 2022-09-16 13:30:00+00:00 ESU2-ESZ2 2022-12-16 14:30:00+00:00 ESZ2 2022-12-16 14:30:00+00:00 ESZ2-ESH3 2022-12-16 14:30:00+00:00 ESZ2-ESM3 2022-12-16 14:30:00+00:00 ESZ2-ESU3 2023-03-17 13:30:00+00:00 ESH3 2023-03-17 13:30:00+00:00 ESH3-ESM3 2023-03-17 13:30:00+00:00 ESH3-ESU3 2023-03-17 13:30:00+00:00 ESH3-ESZ3 2023-06-16 13:30:00+00:00 ESM3 2023-06-16 13:30:00+00:00 ESM3-ESU3 2023-06-16 13:30:00+00:00 ESM3-ESZ3 2023-09-15 13:30:00+00:00 ESU3 2023-09-15 13:30:00+00:00 ESU3-ESH4 2023-09-15 13:30:00+00:00 ESU3-ESZ3 2023-12-15 14:30:00+00:00 ESZ3 2023-12-15 14:30:00+00:00 ESZ3-ESH4 2024-03-15 13:30:00+00:00 ESH4 2024-03-15 13:30:00+00:00 ESH4-ESM4 2024-06-21 13:30:00+00:00 ESM4 2024-09-20 13:30:00+00:00 ESU4 2024-12-20 14:30:00+00:00 ESZ4 2025-12-19 14:30:00+00:00 ESZ5 2026-12-18 14:30:00+00:00 ESZ6 ``` --- Source: https://databento.com/docs/api-historical/helpers/request-symbology?lang=python ### DBNStore.request\_symbology Request to resolve [symbology](https://databento.com/docs/api-historical/basics/symbology.md) mappings based on the metadata properties. #### API method ```python DBNStore.request_symbology( client: Historical, ) -> dict[str, Any] ``` #### Parameters | Parameter | Type | Required | Description | | `client` | Historical | required | The historical client to use for the request (contains the API key). | #### Returns `dict[str, Any]` A result including a map of input symbol to output symbol across a date range. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06", ) # Save streamed data to .dbn.zst data.to_file("GLBX-ESM2-20220606.trades.dbn.zst") # Read saved .dbn.zst stored_data = db.DBNStore.from_file("GLBX-ESM2-20220606.trades.dbn.zst") # Request symbology from .dbn.zst metadata symbology = stored_data.request_symbology(client=client) print(symbology) ``` #### Example response ```json { "result": { "ESM2": [ { "d0": "2022-06-06", "d1": "2022-06-07", "s": "3403" } ] }, "symbols": [ "ESM2" ], "stype_in": "raw_symbol", "stype_out": "instrument_id", "start_date": "2022-06-06", "end_date": "2022-06-07", "partial": [], "not_found": [], "message": "OK", "status": 0 } ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-to-csv?lang=python ### DBNStore.to\_csv Write data to a file in CSV format. #### API method ```python DBNStore.to_csv( path: PathLike[str] | str, pretty_ts: bool = True, pretty_px: bool = True, map_symbols: bool = True, compression: Compression | str = Compression.NONE, schema: Schema | str | None = None, mode: Literal["w", "x"] = "w", ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `path` | PathLike[str] or str | required | The file path to write to. | | `pretty_ts` | bool, default True | optional | Whether timestamp columns are converted to tz-aware `pandas.Timestamp` (UTC). | | `pretty_px` | bool, default True | optional | Whether price columns are correctly scaled as display prices. | | `map_symbols` | bool, default True | optional | If symbology mappings from the metadata should be used to create a 'symbol' column, mapping the instrument ID to its native symbol for every record. | | `compression` | Compression or str | optional | The output compression for writing. Must be either 'zstd' or 'none'. | | `schema` | Schema or str | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md) for the output CSV. Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). This is only required when reading a `DBNStore` with mixed record types. | | `mode` | str | optional | The file write mode to use, either "x" or "w". Defaults to "w". | #### Returns `None` #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06", ) # Save streamed data to .csv data.to_csv("GLBX-ESM2-20220606-trades.csv") ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-to-df?lang=python ### DBNStore.to\_df Converts data to a [pandas](https://pandas.pydata.org/) DataFrame. > **Info** > > The DataFrame index will be set to `ts_recv` if it exists in the schema, otherwise it will be set to `ts_event`. > **See also** > > While not optimized for use with live data due to their column-oriented format, > pandas DataFrames can still be used with live data by first streaming DBN data to a file, > then converting to a DataFrame with DBNStore.from_file().to_df(). > See [this example](https://databento.com/docs/examples/basics-live/live-stream-to-file.md) for more information. #### API method ```python DBNStore.to_df( price_type: PriceType | str = "float", pretty_ts: bool = True, map_symbols: bool = True, schema: Schema | str | None = None, tz: datetime.tzinfo | str = zoneinfo.ZoneInfo("UTC"), count: int | None = None, ) -> pd.DataFrame | DataFrameIterator ``` #### Parameters | Parameter | Type | Required | Description | | `price_type` | PriceType or str, default "float" | optional | The price type to use for price fields. If "fixed", prices will have a type of `int` in fixed decimal format; each unit representing 1e-9 or 0.000000001. If "float", prices will have a type of `float`. If "decimal", prices will be instances of `decimal.Decimal`. | | `pretty_ts` | bool, default True | optional | Whether timestamp columns are converted to tz-aware `pandas.Timestamp`. The timezone can be specified using the `tz` parameter. | | `map_symbols` | bool, default True | optional | If symbology mappings from the metadata should be used to create a 'symbol' column, mapping the instrument ID to its raw symbol for every record. | | `schema` | Schema or str | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md) for the output DataFrame. Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). This is only required when reading a `DBNStore` with mixed record types. | | `tz` | datetime.tzinfo or str, default UTC | optional | If `pretty_ts` is `True`, all timestamps will be converted to the specified timezone. | | `count` | int | optional | If set, instead of returning a single `DataFrame` a `DataFrameIterator` instance will be returned. When iterated, this object will yield a `DataFrame` with at most `count` elements until the entire contents of the `DBNStore` are exhausted. | #### Returns A pandas [DataFrame](https://pandas.pydata.org/docs/reference/frame.html) object. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="trades", symbols=["ESM2"], start="2022-03-06", ) df = data.to_df() print(df.head()) ``` #### Example response ```plaintext ts_event rtype publisher_id instrument_id action ... size flags ts_in_delta sequence symbol ts_recv ... 2022-03-06 23:00:00.039463300+00:00 2022-03-06 23:00:00.036436177+00:00 0 1 3403 T ... 1 0 18828 5178 ESM2 2022-03-06 23:00:01.098111252+00:00 2022-03-06 23:00:01.097477845+00:00 0 1 3403 T ... 1 0 19122 6816 ESM2 2022-03-06 23:00:04.612334175+00:00 2022-03-06 23:00:04.611714663+00:00 0 1 3403 T ... 1 0 18687 10038 ESM2 2022-03-06 23:00:04.613776789+00:00 2022-03-06 23:00:04.613240435+00:00 0 1 3403 T ... 1 0 18452 10045 ESM2 2022-03-06 23:00:06.881864467+00:00 2022-03-06 23:00:06.880575603+00:00 0 1 3403 T ... 1 0 18478 11343 ESM2 [5 rows x 13 columns] ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-to-file?lang=python ### DBNStore.to\_file Write data to a DBN file. #### API method ```python DBNStore.to_file( path: PathLike[str] | str, mode: Literal["w", "x"] = "w", compression: Compression | str | None = None, ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `path` | PathLike[str] or str | required | The file path to write to. | | `mode` | str | optional | The file write mode to use, either "x" or "w". Defaults to "w". | | `compression` | Compression or str | optional | The compression format to write. If `None`, uses the same compression as the underlying data. | #### Returns A [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06", ) # Save streamed data to .dbn.zst data.to_file("GLBX-ESM2-20220606.trades.dbn.zst") ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-to-json?lang=python ### DBNStore.to\_json Write data to a file in JSON format. #### API method ```python DBNStore.to_json( path: PathLike[str] | str, pretty_ts: bool = True, pretty_px: bool = True, map_symbols: bool = True, compression: Compression | str = Compression.NONE, schema: Schema | str | None = None, mode: Literal["w", "x"] = "w", ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `path` | PathLike[str] or str | required | The file path to write to. | | `pretty_ts` | bool, default True | optional | Whether timestamp columns are converted to tz-aware `pandas.Timestamp` (UTC). | | `pretty_px` | bool, default True | optional | Whether price columns are correctly scaled as display prices. | | `map_symbols` | bool, default True | optional | If symbology mappings from the metadata should be used to create a 'symbol' column, mapping the instrument ID to its raw symbol for every record. | | `compression` | Compression or str | optional | The output compression for writing. Must be either 'zstd' or 'none'. | | `schema` | Schema or str | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md) for the output JSON. Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). This is only required when reading a `DBNStore` with mixed record types. | | `mode` | str | optional | The file write mode to use, either "x" or "w". Defaults to "w". | #### Returns `None` #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06", ) # Save streamed data to .json data.to_json("GLBX-ESM2-20220606-trades.json") ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-to-ndarray?lang=python ### DBNStore.to\_ndarray Converts data to a [numpy](https://numpy.org/) N-dimensional array. Each element will contain a Python representation of the binary fields as a `Tuple`. #### API method ```python DBNStore.to_ndarray( schema: Schema | str | None = None, count: int | None = None, ) -> np.ndarray | NDArrayIterator ``` #### Parameters | Parameter | Type | Required | Description | | `schema` | Schema or str | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md) for the output array. Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). This is only required when reading a `DBNStore` with mixed record types. | | `count` | int | optional | If set, instead of returning a single `np.ndarray` a `NDArrayIterator` instance will be returned. When iterated, this object will yield a `np.ndarray` with at most `count` elements until the entire contents of the `DBNStore` are exhausted. | #### Returns A [numpy.ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) object. #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], start="2022-06-06", ) array = data.to_ndarray() print(array[0].item()) ``` #### Example response ```plaintext (12, 0, 1, 3403, 1654473600070033767, 4108500000000, 1, b'T', b'A', 0, 0, 1654473600070314216, 18681, 157862) ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-to-parquet?lang=python ### DBNStore.to\_parquet Write data to a file in [Apache parquet](https://parquet.apache.org/) format. #### API method ```python DBNStore.to_parquet( path: PathLike[str] | str, price_type: PriceType | str = "float", pretty_ts: bool = True, map_symbols: bool = True, schema: Schema | str | None = None, mode: Literal["w", "x"] = "w", **kwargs: Any, ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `path` | PathLike[str] or str | required | The file path to write the data to. | | `price_type` | PriceType or str, default "float" | optional | The price type to use for price fields. If "fixed", prices will have a type of `int` in fixed decimal format; each unit representing 1e-9 or 0.000000001. If "float", prices will have a type of `float`. | | `pretty_ts` | bool, default True | optional | Whether timestamp columns are converted to tz-aware `pyarrow.TimestampType` (UTC). | | `map_symbols` | bool, default True | optional | If symbology mappings from the metadata should be used to create a 'symbol' column, mapping the instrument ID to its raw symbol for every record. | | `schema` | Schema or str | optional | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md) for the output parquet file. Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). This is only required when reading a `DBNStore` with mixed record types. | | `mode` | str | optional | The file write mode to use, either "x" or "w". Defaults to "w". | | `**kwargs` | Any | optional | Keyword arguments to pass to [pyarrow.parquet.ParquetWriter](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html). These can be used to override the default behavior of the writer. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], schema="trades", start="2022-06-06", ) # Save streamed data to .parquet data.to_parquet("GLBX-ESM2-20220606-trades.parquet") ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-iter?lang=python ### DBNStore.\_\_iter\_\_ Using [for](https://docs.python.org/3/reference/compound_stmts.html#the-for-statement); records will be iterated one at a time. Iteration will stop when there are no more records in the `DBNStore` instance. Refer to the [List of fields by schema](https://databento.com/docs/schemas/whats-a-schema.md) article for documentation on the fields contained with each record type. #### API method ```python DBNStore.__iter__() -> Iterator[DBNRecord] ``` #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2"], start="2022-06-06", ) for trade in data: size = getattr(trade, "size", 0) if size >= 200: print(trade) ``` #### Example response ```plaintext TradeMsg(ts_recv=1654524078342408839, pretty_ts_recv='2022-06-06T14:01:18.342408839Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=3403, ts_event=1654524078339857609, pretty_ts_event='2022-06-06T14:01:18.339857609Z', action='T', side='B', depth=0, price=4164000000000, pretty_price=4164, size=291, flags=0, ts_in_delta=20352, sequence=3605032) TradeMsg(ts_recv=1654524133737794739, pretty_ts_recv='2022-06-06T14:02:13.737794739Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=3403, ts_event=1654524133736900455, pretty_ts_event='2022-06-06T14:02:13.736900455Z', action='T', side='B', depth=0, price=4160000000000, pretty_price=4160, size=216, flags=0, ts_in_delta=28024, sequence=3659203) TradeMsg(ts_recv=1654538295589900967, pretty_ts_recv='2022-06-06T17:58:15.589900967Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=3403, ts_event=1654538295588752739, pretty_ts_event='2022-06-06T17:58:15.588752739Z', action='T', side='B', depth=0, price=4140000000000, pretty_price=4140, size=200, flags=0, ts_in_delta=21708, sequence=10031624) ``` --- Source: https://databento.com/docs/api-historical/helpers/dbn-store-insert-symbology-json?lang=python ### DBNStore.insert\_symbology\_json Insert JSON symbology data which may be obtained from a symbology request or loaded from a file. #### API method ```python DBNStore.insert_symbology_json( json_data: str | Mapping[str, Any] | TextIO, clear_existing: bool = True, ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `json_data` | str or Mapping[str, Any] or TextIO | required | The JSON data to insert. | | `clear_existing` | bool | optional | If existing symbology data should be cleared from the internal mappings. | #### Example usage ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="XNAS.ITCH", symbols=["ALL_SYMBOLS"], schema="trades", start="2022-06-06", end="2022-06-07", ) # Request symbology for all symbols and then insert this data symbology_json = data.request_symbology(client) data.insert_symbology_json(symbology_json, clear_existing=True) ``` --- Source: https://databento.com/docs/api-historical/helpers/map_symbols_csv?lang=python ### map_symbols_csv Use a `symbology.json` file to map a symbols column onto an existing CSV file. The result is written to `out_file`. #### API method ```python map_symbols_csv( symbology_file: PathLike[str] | str, csv_file: PathLike[str] | str, out_file: PathLike[str] | str | None = None, ) -> Path: ``` #### Parameters | Parameter | Type | Required | Description | | `symbology_file` | PathLike[str] or str | required | Path to a `symbology.json` file to use as a symbology source. | | `csv_file` | PathLike[str] or str | required | Path to a CSV file that contains encoded DBN data; must contain a `ts_recv` or `ts_event` and `instrument_id` column. | | `out_file` | PathLike[str] or str | optional | Path to a file to write results to. If unspecified, `_mapped` will be appended to the `csv_file` name. | #### Returns [Path](https://docs.python.org/3/library/pathlib.html?highlight=path#pathlib.Path) to the written file. #### Example usage ```python import databento as db result = db.map_symbols_csv( "symbology.json", "xnas-itch-20230821-20230825.ohlcv-1d.csv", ) print(result.read_text()) ``` #### Example response ```plaintext ts_event,rtype,publisher_id,instrument_id,open,high,low,close,volume,symbol 1692576000000000000,35,2,523,133550000000,135200000000,132710000000,134360000000,11015261,AMZN 1692576000000000000,35,2,7290,439090000000,472900000000,437260000000,470990000000,11098972,NVDA 1692576000000000000,35,2,10157,217000000000,233500000000,217000000000,233380000000,21336884,TSLA 1692576000000000000,35,2,7130,430170000000,434700000000,0,433000000000,68661,NOC 1692662400000000000,35,2,7132,431610000000,438480000000,0,437740000000,86950,NOC 1692662400000000000,35,2,10161,236710000000,241550000000,229560000000,232870000000,17069349,TSLA 1692662400000000000,35,2,523,135320000000,135900000000,133740000000,134200000000,8133698,AMZN 1692662400000000000,35,2,7293,475120000000,483440000000,453340000000,457910000000,11700447,NVDA 1692748800000000000,35,2,523,135120000000,137860000000,133220000000,137060000000,11430081,AMZN 1692748800000000000,35,2,7296,463190000000,518870000000,452080000000,502150000000,13361964,NVDA 1692748800000000000,35,2,7135,0,440000000000,0,434030000000,68411,NOC 1692748800000000000,35,2,10163,236590000000,243750000000,226500000000,240810000000,13759234,TSLA 1692835200000000000,35,2,7287,508190000000,512600000000,466720000000,468700000000,21611800,NVDA 1692835200000000000,35,2,10154,242000000000,244140000000,228180000000,229980000000,13724054,TSLA 1692835200000000000,35,2,7126,433650000000,437730000000,0,429880000000,54570,NOC 1692835200000000000,35,2,521,137250000000,137820000000,131410000000,131880000000,11990573,AMZN ``` --- Source: https://databento.com/docs/api-historical/helpers/map_symbols_json?lang=python ### map_symbols_json Use a `symbology.json` file to insert a symbols key into records of an existing JSON file. The result is written to `out_file`. #### API method ```python map_symbols_json( symbology_file: PathLike[str] | str, json_file: PathLike[str] | str, out_file: PathLike[str] | str | None = None, ) -> Path: ``` #### Parameters | Parameter | Type | Required | Description | | `symbology_file` | PathLike[str] or str | required | Path to a `symbology.json` file to use as a symbology source. | | `json_file` | PathLike[str] or str | required | Path to a JSON file that contains encoded DBN data. | | `out_file` | PathLike[str] or str | optional | Path to a file to write results to. If unspecified, `_mapped` will be appended to the `csv_file` name. | #### Returns [Path](https://docs.python.org/3/library/pathlib.html?highlight=path#pathlib.Path) to the written file. #### Example usage ```python import databento as db result = db.map_symbols_json( "symbology.json", "xnas-itch-20230821-20230825.ohlcv-1d.json", ) print(result.read_text()) ``` #### Example response ```plaintext {"hd":{"ts_event":"1692576000000000000","rtype":35,"publisher_id":2,"instrument_id":523},"open":"133550000000","high":"135200000000","low":"132710000000","close":"134360000000","volume":"11015261","symbol":"AMZN"} {"hd":{"ts_event":"1692576000000000000","rtype":35,"publisher_id":2,"instrument_id":7290},"open":"439090000000","high":"472900000000","low":"437260000000","close":"470990000000","volume":"11098972","symbol":"NVDA"} {"hd":{"ts_event":"1692576000000000000","rtype":35,"publisher_id":2,"instrument_id":10157},"open":"217000000000","high":"233500000000","low":"217000000000","close":"233380000000","volume":"21336884","symbol":"TSLA"} {"hd":{"ts_event":"1692576000000000000","rtype":35,"publisher_id":2,"instrument_id":7130},"open":"430170000000","high":"434700000000","low":"0","close":"433000000000","volume":"68661","symbol":"NOC"} {"hd":{"ts_event":"1692662400000000000","rtype":35,"publisher_id":2,"instrument_id":10161},"open":"236710000000","high":"241550000000","low":"229560000000","close":"232870000000","volume":"17069349","symbol":"TSLA"} {"hd":{"ts_event":"1692662400000000000","rtype":35,"publisher_id":2,"instrument_id":523},"open":"135320000000","high":"135900000000","low":"133740000000","close":"134200000000","volume":"8133698","symbol":"AMZN"} {"hd":{"ts_event":"1692662400000000000","rtype":35,"publisher_id":2,"instrument_id":7293},"open":"475120000000","high":"483440000000","low":"453340000000","close":"457910000000","volume":"11700447","symbol":"NVDA"} {"hd":{"ts_event":"1692662400000000000","rtype":35,"publisher_id":2,"instrument_id":7132},"open":"431610000000","high":"438480000000","low":"0","close":"437740000000","volume":"86950","symbol":"NOC"} {"hd":{"ts_event":"1692748800000000000","rtype":35,"publisher_id":2,"instrument_id":523},"open":"135120000000","high":"137860000000","low":"133220000000","close":"137060000000","volume":"11430081","symbol":"AMZN"} {"hd":{"ts_event":"1692748800000000000","rtype":35,"publisher_id":2,"instrument_id":7296},"open":"463190000000","high":"518870000000","low":"452080000000","close":"502150000000","volume":"13361964","symbol":"NVDA"} {"hd":{"ts_event":"1692748800000000000","rtype":35,"publisher_id":2,"instrument_id":7135},"open":"0","high":"440000000000","low":"0","close":"434030000000","volume":"68411","symbol":"NOC"} {"hd":{"ts_event":"1692748800000000000","rtype":35,"publisher_id":2,"instrument_id":10163},"open":"236590000000","high":"243750000000","low":"226500000000","close":"240810000000","volume":"13759234","symbol":"TSLA"} {"hd":{"ts_event":"1692835200000000000","rtype":35,"publisher_id":2,"instrument_id":7287},"open":"508190000000","high":"512600000000","low":"466720000000","close":"468700000000","volume":"21611800","symbol":"NVDA"} {"hd":{"ts_event":"1692835200000000000","rtype":35,"publisher_id":2,"instrument_id":10154},"open":"242000000000","high":"244140000000","low":"228180000000","close":"229980000000","volume":"13724054","symbol":"TSLA"} {"hd":{"ts_event":"1692835200000000000","rtype":35,"publisher_id":2,"instrument_id":7126},"open":"433650000000","high":"437730000000","low":"0","close":"429880000000","volume":"54570","symbol":"NOC"} {"hd":{"ts_event":"1692835200000000000","rtype":35,"publisher_id":2,"instrument_id":521},"open":"137250000000","high":"137820000000","low":"131410000000","close":"131880000000","volume":"11990573","symbol":"AMZN"} ``` --- Source: https://databento.com/docs/api-live?lang=python # API reference - Live You can receive live data from Databento via our live APIs, namely the Databento Raw API. The Raw API is a simple socket-based, subscription-style protocol. Clients communicate with our live data gateways through a regular TCP/IP socket. To make it easier to integrate the API, we also provide official client libraries that simplify the code you need to write. You can use our live APIs to subscribe to real-time data in your application. You can also use the APIs to request intraday historical data and instrument definitions for any number of products in a venue. --- Source: https://databento.com/docs/api-live/basics?lang=python ## Basics --- Source: https://databento.com/docs/api-live/basics/overview?lang=python ### Overview Databento's live API offers both real-time subscriptions and [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md) within the last 24 hours to various market data [schemas](https://databento.com/docs/api-live/basics/schemas-and-conventions.md) and symbols. Multiple subscriptions may be combined into a single session, such as trades, subsampled data (second, minute, hour, daily aggregates), and definitions (expirations, settlement, etc.). For an easy transition from backtesting to live trading, the live API supports the same schemas, [datasets](https://databento.com/docs/api-live/basics/datasets.md), and [symbology](https://databento.com/docs/api-live/basics/symbology.md) as the historical API and returns the same [record structures](https://databento.com/docs/schemas/whats-a-schema.md). Our live clients use our binary [DBN encoding](https://databento.com/docs/conventions/databento-binary-encoding.md) for performant zero-copy market data decoding. --- Source: https://databento.com/docs/api-live/basics/authentication?lang=python ### Authentication Databento uses API keys to authenticate requests. You can view and manage your keys on the [API keys]($PLATFORM_HOSTNAME/portal/keys) page of your portal. Each API key is a 32-character string. By default, our library uses the environment variable `DATABENTO_API_KEY` as your API key. However, if you pass an API key to the `Live` constructor through the `key` parameter, then this value will be used instead. Our Live API uses a challenge-response authentication mechanism to ensure that your API key is never sent over the network. Related: [Securing your API keys](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Authentication happens on the first subscribe client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) ``` --- Source: https://databento.com/docs/api-live/basics/sessions?lang=python ### Sessions Each instance of the [Live](https://databento.com/docs/api-live/client/live.md) client manages a single Raw API session. Each session is associated with one [dataset](https://databento.com/docs/api-live/basics/datasets.md). A session will begin streaming when [Live.start](https://databento.com/docs/api-live/client/start.md) is called. A session can be stopped gracefully with [Live.stop](https://databento.com/docs/api-live/client/stop.md) or forcefully with [Live.terminate](https://databento.com/docs/api-live/client/terminate.md). A session can also be stopped by specifying a timeout with [Live.block_for_close](https://databento.com/docs/api-live/client/block-for-close.md) for synchronous applications and [Live.wait_for_close](https://databento.com/docs/api-live/client/wait-for-close.md) for asynchronous applications. > **See also** > > [Connection limits](https://databento.com/docs/api-live/basics/connection-limits.md) for more details. --- Source: https://databento.com/docs/api-live/basics/schemas-and-conventions?lang=python ### Schemas and conventions A schema is a data record format represented as a collection of different data fields. Our datasets support multiple schemas, such as order book, trades, bar aggregates, and so on. You can get a dictionary describing the fields of each schema from our [List of market data schemas](https://databento.com/docs/schemas/whats-a-schema.md). You can get a list of all supported schemas for any given dataset using the Historical client's [list_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md) method. The same information can also be found on the dataset details pages on the [user portal]($PLATFORM_HOSTNAME). The following table provides details about the data types and conventions used for various fields that you will commonly encounter in the data. | Name | Field | Description | | Dataset | `dataset` | A unique string name assigned to each dataset by Databento. Full list of datasets can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | Publisher ID | `publisher_id` | A unique 16-bit unsigned integer assigned to each publisher by Databento. Full list of publisher IDs can be found from the [metadata](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md). | | Instrument ID | `instrument_id` | A unique 32-bit unsigned integer assigned to each instrument by the venue. Information about instrument IDs for any given dataset can be found in the [symbology](https://databento.com/docs/api-historical/symbology.md). | | Order ID | `order_id` | A unique 64-bit unsigned integer assigned to each order by the venue. | | Timestamp (event) | `ts_event` | The matching-engine-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp (receive) | `ts_recv` | The capture-server-received timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | | Timestamp delta (in) | `ts_in_delta` | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [timestamping](https://databento.com/docs/architecture/timestamping-guide.md) guide. | | Timestamp out | `ts_out` | The Databento gateway-sending timestamp expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). See [timestamping](https://databento.com/docs/architecture/timestamping-guide.md) guide. | | Price | `price` | The price expressed as signed integer where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | Book side | `side` | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified by the original source. | | Size | `size` | The order quantity. | | Flag | `flag` | A bit field indicating event end, message characteristics, and data quality. | | Action | `action` | The event type or order book operation. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, **T**rade, **F**ill, or **N**one. | | Sequence number | `sequence` | The original message sequence number from the venue. | --- Source: https://databento.com/docs/api-live/basics/datasets?lang=python ### Datasets Databento provides time series datasets for a variety of markets, sourced from different publishers. Our available datasets can be browsed through the [search]($PLATFORM_HOSTNAME) feature on our site. Each dataset is assigned a unique string identifier (dataset ID) in the form `PUBLISHER.DATASET`, such as `GLBX.MDP3`. For publishers that are also markets, we use standard four-character ISO 10383 [Market Identifier Codes (MIC)](https://www.iso20022.org/market-identifier-codes). Otherwise, Databento arbitrarily assigns a four-character identifier for the publisher. These dataset IDs are also found on the [Data catalog](https://databento.com/docs/portal/data-catalog.md) and [Download request](https://databento.com/docs/portal/batch-download.md) features of the Databento user portal. When a publisher provides multiple data products with different levels of granularity, Databento subscribes to the most-granular product. We then provide this dataset with [alternate schemas](https://databento.com/docs/schemas/whats-a-schema.md) to make it easy to work with the level of detail most appropriate for your application. More information about different types of venues and publishers is available in our [FAQs](https://databento.com/docs/faqs/venues-and-publishers.md). --- Source: https://databento.com/docs/api-live/basics/symbology?lang=python ### Symbology Databento's live API supports several ways to select an instrument in a [dataset](https://databento.com/docs/api-historical/basics/datasets.md). An instrument is specified using a **symbol** and a **symbology type**, also referred to as an **stype**. The supported symbology types are: - **[Raw symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol)** (`raw_symbol`) original string symbols used by the publisher in the source data. - **[Instrument ID symbology](https://databento.com/docs/conventions/symbology.md#instrument-id)** (`instrument_id`) unique numeric ID assigned to each instrument by the publisher. - **[Parent symbology](https://databento.com/docs/conventions/symbology.md#parent)** (`parent`) groups instruments related to the market for the same underlying. - **[Continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous)** (`continuous`) proprietary symbology that specifies instruments based on certain systematic rules. > **Info** > > In the live API, existing subscriptions to continuous contracts will not be remapped to different instruments. > However, submitting a new identical subscription may result in a new mapping. When subscribing to live data, an input symbology type can be specified. By default, our client libraries will use raw symbology for the input type. Not all symbology types are supported for every dataset. If one or more symbols in a subscription fail to resolve, the gateway emits an [`ErrorMsg`](https://databento.com/docs/api-live/basics/errors.md) record with `code=SYMBOL_RESOLUTION_FAILED` for each unresolved symbol and continues streaming data for any symbols that did resolve. If a symbol later resolves (for example, a new listing from the publisher), the gateway publishes a `SymbolMappingMsg` and begins streaming data for that symbol. For live data, symbology mappings are provided through [SymbolMappingMsg](https://databento.com/docs/conventions/symbology.md#symbolmappingmsg) records. These records are sent after the session has started and can be used to map the `instrument_id` from a data record's header to a text symbol. For more about symbology at Databento, see our [Standards and conventions](https://databento.com/docs/conventions/symbology.md). --- Source: https://databento.com/docs/api-live/basics/dates-and-times?lang=python ### Dates and times Our Python client library has several functions with timestamp arguments. These arguments will have type `pandas.Timestamp | datetime.date | str | int` and support a variety of formats. It's recommended to use [pandas.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), which fully supports timezones and nanosecond-precision. If a `datetime.date` is used, the time is set to midnight UTC. If an `int` is provided, the value is interpreted as [UNIX nanoseconds](https://en.wikipedia.org/wiki/Unix_time). The client library also handles several string-based timestamp formats based on [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html). - `yyyy-mm-dd`, e.g. `"2022-02-28"` (midnight UTC) - `yyyy-mm-ddTHH:MM`, e.g. `"2022-02-28T23:50"` - `yyyy-mm-ddTHH:MM:SS`, e.g. `"2022-02-28T23:50:59"` - `yyyy-mm-ddTHH:MM:SS.NNNNNNNNN`, e.g. `"2022-02-28T23:50:59.123456789"` Timezone specification is also supported. - `yyyy-mm-ddTHH:MMZ` - `yyyy-mm-ddTHH:MM±hh` - `yyyy-mm-ddTHH:MM±hhmm` - `yyyy-mm-ddTHH:MM±hh:mm` **Bare dates** Some parameters require a bare date, without a time. These arguments have type `datetime.date | str` and must either be a `datetime.date` object, or a string in `yyyy-mm-dd` format, e.g. `"2022-02-28"`. --- Source: https://databento.com/docs/api-live/basics/intraday-replay?lang=python ### Intraday replay Our live API offers intraday replay within the last 24 hours. On weekends and other periods when data isn't being published, the gateway may report less than 24 hours of replay available. Users can connect to the live service and request data from a particular start time. Data will be filtered on `ts_event` for all schemas except CBBO and BBO, which will be filtered on `ts_recv`. A different start time can be specified for each subscription. There can be multiple subscriptions for the same schema, with each subscription having a different start time. When the session starts, records will be immediately replayed for each schema. A replay completed [SystemMsg](https://databento.com/docs/api-live/basics/system-messages.md) will be sent for each replayed schema once it catches up to real-time data. Once a session has started, newly added subscriptions are not eligible for intraday replay. As a special case for the `GLBX.MDP3` dataset, we also provide replay of the entire weekly session in the MBO and definition schemas outside of the 24-hour window to aid with recovery, as these schemas are highly stateful. Pass `start=0` to easily request the full replay history available for each schema. Our Python client library supports several convenient date formats, such as [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) strings, or [UNIX timestamps](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Refer to the [Dates and times](https://databento.com/docs/api-live/basics/dates-and-times.md) article for more information on how our client library handles timestamps. #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe with a specified start time for intraday replay client.subscribe( dataset="GLBX.MDP3", schema="trades", symbols="ES.FUT", stype_in="parent", start="2023-04-17T09:00:00", ) ``` --- Source: https://databento.com/docs/api-live/basics/live-snapshot?lang=python ### Snapshot Users can request a snapshot for a live subscription to obtain the recent order book state without [replaying](https://databento.com/docs/api-live/basics/intraday-replay.md) the whole trading session. This is only supported for the [MBO](https://databento.com/docs/schemas/mbo.md) schema. More details can be found in this [article](https://databento.com/docs/conventions/mbo-snapshot.md). #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to live snapshot client.subscribe( dataset="GLBX.MDP3", schema="mbo", symbols="ES.FUT", stype_in="parent", snapshot=True, ) ``` --- Source: https://databento.com/docs/api-live/basics/system-messages?lang=python ### System messages Our live API uses a system record (`SystemMsg`) to indicate non-error information to clients. One use is heartbeating, to ensure the TCP connection remains open and to help clients detect a connection issue. A heartbeat will only be sent if no other data record was sent to the client during the heartbeat interval. The interval between heartbeat messages can be configured with the `heartbeat_interval_s` parameter to [Live](https://databento.com/docs/api-live/client/live.md). The `is_heartbeat()` method will return `True` if the record is a heartbeat. The gateway sends an end-of-interval message (`END_OF_INTERVAL`) once it has published every record for one interval of an OHLCV, BBO, or CBBO schema. Its `ts_event` is the exclusive end of that interval. For OHLCV schemas, that is the bar's `ts_event` plus the bar interval, so an `ohlcv-1m` end-of-interval message with a `ts_event` of 14:31:00 follows the bars stamped 14:30:00. For BBO and CBBO schemas, it equals the `ts_recv` of the records in the interval. The gateway sends one end-of-interval message per schema and interval. | Field | Type | Description | | `msg` | str | The message from the gateway. | | `code` | int | Describes the type of system message. See table below. | #### System code variants Corresponds with the `SystemCode` enum. | Variant | `code` | Description | | HEARTBEAT | `0` | A message sent in the absence of other records to indicate the connection remains open. | | SUBSCRIPTION\_ACK | `1` | An acknowledgement of a subscription request. | | SLOW\_READER\_WARNING | `2` | The gateway has detected this session is falling behind real-time data. | | REPLAY\_COMPLETED | `3` | Indicates a replay subscription has caught up with real-time data. This message will be sent per schema. | | END\_OF\_INTERVAL | `4` | Signals that all records for an interval-based schema have been published for the interval ending at `ts_event`. | --- Source: https://databento.com/docs/api-live/basics/errors?lang=python ### Errors Our live API uses an error record (`ErrorMsg`) to indicate failures to clients. Error records are processed like any other record, and as such will be passed on to any callbacks, streams, and iterators for the [Live](https://databento.com/docs/api-live/client/live.md) client. | Field | Type | Description | | `err` | str | The error message. | | `code` | int | Describes the type of error message. See table below. | | `is_last` | bool | `True` if this is the last in a series of error records. | **Error code variants** Corresponds with the `ErrorCode` enum. | Variant | `code` | Fatal | Description | | AUTH\_FAILED | `1` | ✓ | The authentication step failed. | | API\_KEY\_DEACTIVATED | `2` | ✓ | The user account or API key were deactivated. | | CONNECTION\_LIMIT\_EXCEEDED | `3` | ✓ | The user has exceeded their open connection limit. | | SYMBOL\_RESOLUTION\_FAILED | `4` | | A symbol in a subscription failed to resolve. Emitted per unresolved symbol. | | INVALID\_SUBSCRIPTION | `5` | ✓ | There was an issue with a subscription request (other than symbol resolution). | | INTERNAL\_ERROR | `6` | ✓ | An error occurred in the gateway. | | SKIPPED\_RECORDS\_AFTER\_SLOW\_READING | `7` | | A slow client was detected and records were skipped by the gateway to allow catching up. | | REPLAY\_DATA\_AGED\_OUT | `8` | ✓ | Replay data is no longer retained and the subscription's schema is incompatible with skipping records. | Fatal errors cause the gateway to close the session after sending the error record. Non-fatal errors are informational and the session continues. #### Logging Our Python client library is fully compatible with the built-in [logging](https://docs.python.org/3/library/logging.html) module. When using the live client to build an application, it is recommended to enable logging in the `databento` module. This logging is disabled by default. --- Source: https://databento.com/docs/api-live/basics/connection-limits?lang=python ### Connection limits With our live API, there is a limit of 10 simultaneous connections ([sessions](https://databento.com/docs/api-live/basics/sessions.md)) per ([dataset](https://databento.com/docs/api-live/basics/datasets.md)) per team for Standard plans. Plus and Unlimited plans will be limited to 50 simultaneous connections per dataset per team. Creating additional API keys will not affect the maximum number of connections per team. In addition, a single gateway will allow at most five incoming connections per second from the same IP address. If an IP address goes over this limit, incoming connections will be immediately closed by the gateway -- existing connections will not be affected. If this happens, clients should wait one second before retrying. #### Subscription rate limits [Subscription requests](https://databento.com/docs/api-live/client/live-blocking-subscribe.md) are throttled to prevent abuse and accidental performance impact on other users. Subscriptions above the limit of 10 per second will be artificially delayed until the session is back under the rate limit, but not rejected; the gateway will send a [subscription acknowledgement](https://databento.com/docs/api-live/basics/system-messages.md) when it has finished processing a subscription request. --- Source: https://databento.com/docs/api-live/basics/error-detection?lang=python ### Error detection When maintaining a connected [Live client](https://databento.com/docs/api-live/client/live.md), clients should monitor their connection for errors. There are three main ways in which a session can enter an error state: * Hung connection: The client is not receiving any data from the gateway * Disconnect without error: The client is explicitly disconnected by the gateway, without receiving an error message * Disconnect with error: The client is explicitly disconnected by the gateway. Immediately prior to being disconnected, the client will receive an [error record](https://databento.com/docs/api-live/basics/errors.md) #### Hung connection To detect hung connections, clients are instructed to make use of system heartbeats. Clients can configure a heartbeat interval when creating the [Live client](https://databento.com/docs/api-live/client/live.md) by setting the `heartbeat_interval_s` parameter. If the heartbeat interval is not set by the client, it will default to 30 seconds. Once a session is started, if no data is sent by the gateway for the entirety of a heartbeat interval, the gateway will send a [system message](https://databento.com/docs/api-live/basics/system-messages.md) to the client to indicate a heartbeat. If the gateway is regularly sending other messages to the client (for example, `MboMsg`), heartbeats will not be sent. Once a session is started, if a client does not receive any messages from the gateway for the duration of one heartbeat interval plus ten seconds, the session can be considered hung. Clients are instructed to disconnect from the gateway and reconnect upon detecting hung connections. Clients with unstable internet connections may need larger intervals than ten seconds to ensure the connection is truly hung, as opposed to merely delayed. #### Disconnect without error From the point of view of the [Live client](https://databento.com/docs/api-live/client/live.md), a disconnect is detected when the underlying TCP session is closed. When using [Live.block_for_close](https://databento.com/docs/api-live/client/block-for-close.md) or [Live.wait_for_close](https://databento.com/docs/api-live/client/wait-for-close.md), an exception will be raised if the TCP session was closed without an error. Upon being disconnected, clients are instructed to wait one second and initiate a new connection. Waiting too short an interval to reconnect may trigger the gateway's rate limiter. > **See also** > > [Connection limits](https://databento.com/docs/api-live/basics/connection-limits.md) for more details. #### Disconnect with error From the point of view of the [Live client](https://databento.com/docs/api-live/client/live.md), a disconnect with error is detected when the underlying TCP session is closed after an `ErrorMsg` or a API error response is received. Any `ErrorMsg` records received by the client can be consumed by the application, such as when [iterating](https://databento.com/docs/api-live/client/iter.md) or using a callback with [Live.add_callback](https://databento.com/docs/api-live/client/add-callback.md). When using [Live.block_for_close](https://databento.com/docs/api-live/client/block-for-close.md) or [Live.wait_for_close](https://databento.com/docs/api-live/client/wait-for-close.md), any `ErrorMsg` records received will be raised in an exception. Clients disconnected with an error are instructed to not reconnect automatically. In the vast majority of cases, reconnecting and resubscribing with the same parameters will lead to the same errors being received again. The error sent to the client will indicate the issue to be fixed prior to resubscribing. --- Source: https://databento.com/docs/api-live/basics/versioning?lang=python ### Versioning Our historical and live APIs and their client libraries adopt `MAJOR.MINOR.PATCH` format for version numbers. These version numbers conform to [semantic versioning](https://semver.org). We are using major version `0` for initial development, where our API is not considered stable. Once we release major version `1`, our public API will be stable. This means that you will be able to upgrade minor or patch versions to pick up new functionality, without breaking your integration. Starting with major versions after `1`, we will provide support for previous versions for one year after the date of the subsequent major release. For example, if version `2.0.0` is released on January 1, 2024, then all versions `1.x.y` of the API and client libraries will be deprecated. However, they will remain supported until January 1, 2025. We may introduce backwards-compatible changes between minor versions in the form of: - New data [encodings](https://databento.com/docs/api-historical/basics/encodings.md) - Additional fields to existing data [schemas](https://databento.com/docs/api-historical/basics/schemas-and-conventions.md) - Additional batch download [customizations](https://databento.com/docs/api-historical/batch.md) Our [Release notes](https://databento.com/docs/release-notes/release-notes-python.md) will contain information about both breaking and backwards-compatible changes in each release. Our API and official client libraries are kept in sync with same-day releases for major versions. For instance, `1.x.y` of the C++ client library will use the same functionality found in any `1.x.y` version of the Python client. Related: [Release notes](https://databento.com/docs/release-notes/release-notes-python.md). --- Source: https://databento.com/docs/api-live/basics/compression?lang=python ### Compression Databento provides options for compressing files from our API. **ZSTD** The `ZSTD` compression option uses the [Zstandard](http://facebook.github.io/zstd/) format. This adds a slight performance cost to encoding and decoding. Read more about [working with Zstandard-compressed files](https://databento.com/docs/conventions/working-with-zstandard.md). **NONE** The `NONE` compression option disables compression entirely, resulting in larger data transfer. By default, live data is uncompressed. --- Source: https://databento.com/docs/api-live/basics/recovering-after-a-disconnection?lang=python ### Recovering after a disconnection When reconnecting to the gateway, clients should [resubscribe](https://databento.com/docs/api-live/client/subscribe.md) to all desired symbols. In order to avoid missing data after a reconnection, there are three main approaches to recovery: - Natural refresh - Intraday replay - Snapshot (MBO only) The best approach to recovery will depend on the client's use case and specific needs. #### Natural refresh To recover via natural refresh, clients can resubscribe to all desired symbols without the `start` or `snapshot` parameters. This means no data will be replayed, and the client will immediately receive the newest messages upon subscribing. This recovery approach is the fastest (since there's no data replay), and is recommended for stateless schemas such as MBP-10, in cases where the client only requires the current state of each instrument. #### Intraday replay To recover via intraday replay, clients should store the last `ts_event` and the number of records received with that last timestamp, per schema and instrument. The `ts_event` and record count should be continuously updated when processing incoming records. When reconnecting, clients should set the `start` parameter of the resubscription to the lowest stored `ts_event` across all instruments for that schema. The gateway will then send all records starting from that timestamp (including records with the exact same `ts_event`). The resubscription may lead to duplicated data being sent to the client. Clients who require that each message is delivered exactly once are instructed to: - Discard all records with a lower `ts_event` than the stored one for the corresponding instrument - Discard the first N records with the same `ts_event` as the stored one for the corresponding instrument, where N is the number of records already seen with that `ts_event` prior to the disconnection. This is important in case there are multiple events with the same `ts_event` and the client is disconnected halfway through processing those events This recovery approach is recommended when clients require the uninterrupted history of records for the desired schema (for example, when using the Trades schema to construct a ticker tape). However, this approach can take a longer time to synchronize with the live stream, depending on how long the client was disconnected. For the CBBO and BBO schemas where filtering is based on `ts_recv`, clients should store the last `ts_recv` per instrument. When reconnecting, clients should set the `start` parameter of the resubscription to the lowest stored `ts_recv` across all instruments. The gateway will then send all records starting from that timestamp (including records with the same `ts_recv`). Since CBBO and BBO are stateless schemas, you should always refer to the most recent record per instrument. #### Snapshot (MBO only) When resubscribing to the MBO schema, clients can request a [snapshot](https://databento.com/docs/conventions/mbo-snapshot.md) to receive the current state of the book after a disconnection. This eliminates the need to replay the missed messages and leads to faster synchronization with the live stream. This recovery approach is generally recommended over intraday replay when using the MBO schema. --- Source: https://databento.com/docs/api-live/basics/maintenance-schedule?lang=python ### Maintenance schedule We restart our live gateways at the following times: - **CME Globex.** Saturday 02:15 CT - **All ICE venues.** Sunday 09:45 UTC - **All other datasets.** Sunday 10:30 UTC All clients will be disconnected during this time. It is recommended to wait 15 minutes after the scheduled restart time before reconnecting. Additionally, we may restart our gateways mid-week. While we generally post these mid-week restarts on our [status page](https://status.databento.com/), we may perform these restarts without notice due to an urgent fix. You should configure your client to handle reconnecting automatically. While some exchanges are closed on Saturday, our systems are still connected to the exchange feeds. The exchange may send test data, and our gateways will disseminate this data to all connected clients. If you are not interested in receiving this test data, we recommend you disconnect after the Friday close and reconnect on Sunday after the scheduled restart. > **Info** > > Any test data sent through the Live API will not be seen in our historical data. --- Source: https://databento.com/docs/api-live/basics/slow-reader-behavior?lang=python ### Slow reader behavior Occasionally, a client session's data processing may lag behind real-time data. This can happen due to multiple reasons, such as a spike in messaging volume from the venue, network fluctuations between the client and the server, or an inability of the client application to process messages fast enough to keep up with the venue's message flow. The live API allows clients to configure different behaviors when this happens via the `slow_reader_behavior` parameter in the [Live constructor](https://databento.com/docs/api-live/client/live.md). This parameter can be set to the values `skip` or `warn`, and defaults to `skip` when not set. This parameter cannot be changed after the session is authenticated. #### Skip If the slow reader behavior is set to `skip`, whenever the gateway detects a client falling behind real-time data, an `ErrorMsg` with error code `SKIPPED_RECORDS_AFTER_SLOW_READING` will be sent to the client and the session will skip records until caught up with real-time data. This behavior only applies to subscriptions for the following schemas: MBP-1, MBP-10, BBO-1s, BBO-1m, CMBP-1, CBBO-1s, and CBBO-1m. Subscriptions to other schemas will always use the `warn` behavior. #### Warn If the slow reader behavior is set to `warn`, whenever the gateway detects a client falling behind real-time data, a `SystemMsg` with system code `SLOW_READER_WARNING` will be sent to the client. The session will not skip any records and will attempt to replay every single record in the subscription. In some cases, setting `warn` as the slow reader behavior can lead to increasing lag between the session and real-time data. #### Data aged out of retention A replay can fall behind far enough that the data it was reading is removed, i.e. it now exceeds the [supported replay window](https://databento.com/docs/api-live/basics/intraday-replay.md). Sessions whose effective behavior is `skip` resume from the earliest retained records. Sessions using `warn`, either by request or because the schema is incompatible with skipping, receive a fatal `ErrorMsg` with error code `REPLAY_DATA_AGED_OUT` naming the earliest available `start` time. --- Source: https://databento.com/docs/api-live/client?lang=python ## Client --- Source: https://databento.com/docs/api-live/client/live?lang=python ### Live To access Databento's live API, first create an instance of the `Live` client. The entire API for a streaming session is exposed through instance methods of the client. Databento's `Live` client is built with Python's [asyncio](https://docs.python.org/3/library/asyncio.html) module and can be easily integrated into asynchronous applications. Note that the API key can be passed as a parameter, which is [not recommended for production applications](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). Instead, you can leave out this parameter to pass your API key via the `DATABENTO_API_KEY` environment variable: #### API method ```python class Live( key: str | None = None, gateway: str | None = None, port: int | None = None, ts_out: bool = False, heartbeat_interval_s: int | None = None, reconnect_policy: ReconnectPolicy | str = ReconnectPolicy.NONE, slow_reader_behavior: SlowReaderBehavior | str | None = None, compression: Compression | str = Compression.NONE, loop: asyncio.AbstractEventLoop | None = None, ) ``` #### Example usage ```python import databento as db # Pass as parameter client = db.Live(key="$YOUR_API_KEY") # Or, pass as `DATABENTO_API_KEY` environment variable client = db.Live() ``` #### Parameters | Parameter | Type | Required | Description | | `key` | str | optional | 32-character API key. Found on your [API keys]($PLATFORM_HOSTNAME/portal/keys) page. If `None` then `DATABENTO_API_KEY` environment variable is used. | | `gateway` | str | optional | The url of the remote gateway to connect to. This is for advanced use. | | `port` | int | optional | The port to connect to the remote gateway on. This is for advanced use. | | `ts_out` | bool, default False | optional | Whether the gateway should append the send timestamp (`ts_out`) to each record. | | `heartbeat_interval_s` | int, default None | optional | The interval in seconds at which the gateway will send [heartbeat records](https://databento.com/docs/api-live/basics/system-messages.md) if no other data records are sent. By default heartbeats will be sent at the gateway's default interval. Must be between 5 and 1800 seconds. | | `reconnect_policy` | ReconnectPolicy or str, default None | optional | The reconnection policy for automatically resuming a live session when an unexpected disconnection occurs. | | `slow_reader_behavior` | SlowReaderBehavior or str, default None | optional | How the gateway should handle a connection unable to keep up with realtime. | | `compression` | Compression or str, default None | optional | Sets the [compression](https://databento.com/docs/api-live/basics/compression.md) mode for the read stream. Uncompressed by default. | | `loop` | asyncio.AbstractEventLoop, default None | optional | The event loop to run the client connection in. If `None`, a shared event loop running on a background thread will be used. | --- Source: https://databento.com/docs/api-live/client/add-callback?lang=python ### Live.add\_callback Add a callback to the live client. This callback must take a single record argument. Refer to the [What's a schema](https://databento.com/docs/schemas/whats-a-schema.md) article for documentation on the fields contained with each record type. A callback will receive error messages from the gateway with the `ErrorMsg` record type. This indicates a problem occurred with the session. Exceptions raised in a callback can be handled explicitly by specifying an `exception_callback`. The callbacks are executed in the order they are added. > **Info** > > It is recommended that callback functions be non-blocking to ensure the networking event loop remains running. #### API method ```python Live.add_callback( record_callback: Callable[[DBNRecord], None], exception_callback: Callable[[Exception], None] | None = None, ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `record_callback` | Callable[[DBNRecord], None] | required | A callback to dispatch records to. The callback must take a single record argument. If no exception callback is provided, any exceptions encountered will be logged and raised as warnings for visibility. | | `exception_callback` | Callable[[Exception], None] | optional | A callback to dispatch exceptions to that are raised when executing `record_callback`. The callback must take a single exception argument. | #### Example usage ```python import databento as db # Create a callback to handle DBN records def user_callback(record: db.DBNRecord) -> None: print(f"callback run for {record}") # Create a callback to handle exceptions from `user_callback` def error_handler(exception: Exception) -> None: print(f"an error occurred {exception}") # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Add the user_callback to the client client.add_callback( record_callback=user_callback, exception_callback=error_handler, # optional error handler ) ``` --- Source: https://databento.com/docs/api-live/client/add-stream?lang=python ### Live.add\_stream Add an output stream to the live client. The client will write binary DBN records to the stream. This stream must be writable in bytes mode, or a path to a file that is writable. A stream will receive error messages from the gateway with the `ErrMsg` record type. This indicates a problem occurred with the session. Exceptions raised in a callback can be handled explicitly by specifying an `exception_callback`. The writes are performed in the order the streams were added. > **See also** > > DBN records are optimized for stream-like processing. > > While pandas DataFrames are not well-suited for this due to their column-oriented format, > they can still be used by first streaming DBN data to a file, then converting to a DataFrame with DBNStore.from_file().to_df(). > See [this example](https://databento.com/docs/examples/basics-live/live-stream-to-file.md) for more information. #### API method ```python Live.add_stream( stream: IO[bytes] | PathLike[str] | str, exception_callback: Callable[[Exception], None] | None = None, ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `stream` | IO[bytes] or PathLike[str] or str | required | An IO stream to write records to, or a writable file path. The stream must support writing bytes. | | `exception_callback` | Callable[[Exception], None] | optional | An error handling callback to process exceptions that are raised when writing to the stream. The callback must take a single exception argument. If no exception callback is provided, any exceptions encountered will be logged and raised as warnings for visibility. | #### Example usage ```python import databento as db # Open a file for writing output = open("output.dbn", mode="wb") # Create a callback to handle exceptions when writing to `output` def error_handler(exception: Exception) -> None: print(f"an error occurred {exception}") # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Add the output file stream to the client client.add_stream( stream=output, exception_callback=error_handler, # optional error handler ) ``` --- Source: https://databento.com/docs/api-live/client/add-reconnect-callback?lang=python ### Live.add\_reconnect\_callback Add a reconnect callback to the live client. This callback must take two arguments. When a reconnection policy is set on the `Live` client, this callback will be executed when a reconnection occurs with two arguments: - The last received record's `ts_event` timestamp, or `start` timestamp from the session `Metadata` if no records were received yet. - The `start` timestamp from the reconnected session's `Metadata`. These values can be used to record any gaps in the received data due to a disconnection. Exceptions raised in a callback can be handled explicitly by specifying an `exception_callback`. The callbacks are executed in the order they are added. > **Info** > It is recommended that callback functions be non-blocking to ensure the networking event loop remains running. #### API method ```python Live.add_reconnect_callback( reconnect_callback: Callable[[pd.Timestamp, pd.Timestamp], None] exception_callback: Callable[[Exception], None] | None = None, ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `reconnect_callback` | Callable[[pd.Timestamp, pd.Timestamp], None] | required | A callback for handling reconnections. The callback must take two arguments. | | `exception_callback` | Callable[[Exception], None] | optional | A callback to dispatch exceptions to that are raised when executing `reconnect_callback`. The callback must take a single exception argument. | #### Example usage ```python import databento as db # Create a callback to handle reconnections def reconnect_callback(start, end) -> None: print(f"reconnection gap from {start} to {end}") # Create a callback to handle exceptions from `reconnect_callback` def error_handler(exception: Exception) -> None: print(f"an error occurred {exception}") # Create a live client with a reconnect policy client = db.Live( key="$YOUR_API_KEY", reconnect_policy="reconnect", ) # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Add the reconnect callback to the client client.add_reconnect_callback( reconnect_callback=reconnect_callback, exception_callback=error_handler, # optional error handler ) ``` --- Source: https://databento.com/docs/api-live/client/block-for-close?lang=python ### Live.block\_for\_close Block until the session closes or a timeout is reached. A session will close when the remote gateway disconnects or after [Live.stop](https://databento.com/docs/api-live/client/stop.md) or [Live.terminate](https://databento.com/docs/api-live/client/terminate.md) are called. In the event the connection is closed unexpectedly, a `BentoError` will be raised. If a `timeout` is specified, [Live.terminate](https://databento.com/docs/api-live/client/terminate.md) will be called when the timeout is reached. When this method unblocks, the session is guaranteed to be closed. #### API method ```python Live.block_for_close( timeout: float | None = None, ) -> None ``` #### Parameters | Parameter | Type | Required | Description | | `timeout` | float | optional | A duration in seconds to wait for the session to close. If `None`, wait forever. | #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Start the session client.start() # Block for the session to close client.block_for_close() ``` --- Source: https://databento.com/docs/api-live/client/start?lang=python ### Live.start Start the session. A client can only be started once, and after a successful connection is made by calling [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md). A client can be gracefully stopped by calling [Live.stop](https://databento.com/docs/api-live/client/stop.md) or forcefully using [Live.terminate](https://databento.com/docs/api-live/client/terminate.md) When iterating records with [Live.\_\_iter\_\_](https://databento.com/docs/api-live/client/iter.md) or [Live.\_\_aiter\_\_](https://databento.com/docs/api-live/client/aiter.md) it is not necessary to call [Live.start](https://databento.com/docs/api-live/client/start.md) and doing so before iterating will raise a `ValueError`. > **Info** > > A session cannot be started more than once. #### API method ```python Live.start() -> None ``` #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Print records as they arrive client.add_callback(print) # Start the streaming session client.start() # Block until disconnection client.block_for_close() ``` --- Source: https://databento.com/docs/api-live/client/stop?lang=python ### Live.stop Stop the session and finish processing received records. A client can only be stopped after a successful connection is made by calling [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md). This method does not block waiting for the connection to close. If this is desired, use [Live.block\_for\_close](https://databento.com/docs/api-live/client/block-for-close.md) or [Live.wait\_for\_close](https://databento.com/docs/api-live/client/wait-for-close.md) to wait for the client to disconnect. The connection will eventually close after calling this method. Once the connection has closed, the `Live` client can be reused but the session state is not preserved. #### API method ```python Live.stop() -> None ``` #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Start the streaming session client.start() # Stop the streaming session client.stop() ``` --- Source: https://databento.com/docs/api-live/client/subscribe?lang=python ### Live.subscribe Add a new subscription to the session. All subscriptions must be for the same `dataset`. The returned integer is the `subscription_id` which will match the id used by the gateway when acknowledging the subscription. Supports multiple subscriptions for different schemas, which enables rich data streams containing mixed record types. Specify an optional `start` time for [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md) with subscriptions made before starting the session. Please note there is no unsubscribe method. Subscriptions end when the client disconnects or by calling [Live.stop](https://databento.com/docs/api-live/client/stop.md) or [Live.terminate](https://databento.com/docs/api-live/client/terminate.md) #### API method ```python Live.subscribe( dataset: Dataset | str, schema: Schema | str, symbols: Iterable[str | int] | str | int = "ALL_SYMBOLS", stype_in: SType | str = "raw_symbol", start: pd.Timestamp | datetime | date | str | int | None = None, snapshot: bool = False, ) -> int ``` #### Parameters | Parameter | Type | Required | Description | | `dataset` | Dataset or str | required | The [dataset](https://databento.com/docs/api-historical/basics/datasets.md) code (string identifier). Must be one of the values from [list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md). | | `schema` | Schema or str | required | The data record [schema](https://databento.com/docs/schemas/whats-a-schema.md). Must be one of the values from [list\_schemas](https://databento.com/docs/api-historical/metadata/metadata-list-schemas.md). | | `symbols` | Iterable[str \ | int] or str or int | optional | The product symbols to subscribe to. If `'ALL_SYMBOLS'` then will select **all** symbols. | | `stype_in` | SType or str, default 'raw_symbol' | optional | The [symbology type](https://databento.com/docs/api-historical/basics/symbology.md) of input `symbols`. Must be one of 'raw_symbol', 'instrument_id', 'parent', or 'continuous'. | | `start` | pd.Timestamp, datetime, date, str or int | optional | The inclusive start of subscription replay. Filters on `ts_event` except for the CBBO and BBO schemas, which filter on `ts_recv`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. Must be within the last 24 hours. Pass `0` to request all available data. Cannot be specified after the [session](https://databento.com/docs/api-live/basics/sessions.md) is [started](https://databento.com/docs/api-live/client/start.md). | | `snapshot` | bool, default False | optional | Request subscription with [snapshot](https://databento.com/docs/conventions/mbo-snapshot.md). Only supported with `mbo` schema. Defaults to `False`. Conflicts with the `start` parameter. | #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) ``` --- Source: https://databento.com/docs/api-live/client/terminate?lang=python ### Live.terminate Terminate the session and stop processing records immediately. A client can only be terminated after a successful connection is made by calling [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md). Unlike [Live.stop](https://databento.com/docs/api-live/client/stop.md), the session will end immediately and received records will no longer be processed. Once terminated, the `Live` client can be reused but the session state is not preserved. #### API method ```python Live.terminate() -> None ``` #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Start the streaming session client.start() # Terminate the streaming session client.terminate() ``` --- Source: https://databento.com/docs/api-live/client/wait-for-close?lang=python ### Live.wait\_for\_close Wait until the session closes or a timeout is reached. A session will close when the remote gateway disconnects, or after [Live.stop](https://databento.com/docs/api-live/client/terminate.md) or [Live.terminate](https://databento.com/docs/api-live/client/terminate.md) are called. In the event the connection is closed unexpectedly, a `BentoError` will be raised. If a `timeout` is specified, [Live.terminate](https://databento.com/docs/api-live/client/terminate.md) will be called when the timeout is reached. When this method unblocks, the session is guaranteed to be closed. #### API method ```python Live.wait_for_close( timeout: float | None = None, ) -> None ``` > **Info** > > This method is a [coroutine](https://docs.python.org/3/library/asyncio-task.html#coroutines) and must be used with an [await](https://docs.python.org/3/reference/expressions.html#await) expression. #### Parameters | Parameter | Type | Required | Description | | `timeout` | float | optional | A duration in seconds to wait for the session to close. If `None`, wait forever. | #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to the trades schema for all ES futures client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Start the session client.start() # Wait for the session to end await client.wait_for_close() ``` --- Source: https://databento.com/docs/api-live/client/aiter?lang=python ### Live.\_\_aiter\_\_ Using [async for](https://docs.python.org/3/reference/compound_stmts.html#the-async-for-statement); records will be yielded as they arrive. Iteration will stop when the connection is closed and all records are processed. This is best for integration into asynchronous applications. Asynchronous iteration is only supported inside a [coroutine](https://docs.python.org/3/library/asyncio-task.html#coroutines). Asynchronous iteration will automatically call [Live.start](https://databento.com/docs/api-live/client/start.md) if the session is connected but it has not been started. Starting iteration after the session has started will cause a `ValueError`. Asynchronous iteration will automatically call [Live.stop](https://databento.com/docs/api-live/client/stop.md) if the iterator is destroyed, such as when an async for loop is escaped with an exception or `break` statement. To prevent this behavior, an iterator can be created explicitly using [aiter()](https://docs.python.org/3/library/functions.html#aiter). #### API method ```python Live.__aiter__() -> AsyncIterator[DBNRecord] ``` #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to a data stream client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Start streaming and asynchronously iterate the records async for record in client: print(record) ``` --- Source: https://databento.com/docs/api-live/client/iter?lang=python ### Live.\_\_iter\_\_ Using [for](https://docs.python.org/3/reference/compound_stmts.html#the-for-statement); records will be yielded as they arrive. Iteration will stop when the connection is closed and all records are processed. This is best for integration into simple synchronous applications. Iteration will automatically call [Live.start](https://databento.com/docs/api-live/client/start.md) if the session is connected but it has not been started. Starting iteration after the session has started will cause a `ValueError`. Iteration will automatically call [Live.stop](https://databento.com/docs/api-live/client/stop.md) if the iterator is destroyed, such as when a for loop is escaped with an exception or `break` statement. To prevent this behavior, an iterator can be created explicitly using [iter()](https://docs.python.org/3/library/functions.html#iter). #### API method ```python Live.__iter__() -> Iterator[DBNRecord] ``` #### Example usage ```python import databento as db # Create a live client client = db.Live(key="$YOUR_API_KEY") # Subscribe to a data stream client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) # Start streaming and synchronously iterate the records for record in client: print(record) ``` --- Source: https://databento.com/docs/api-reference-data?lang=python # API reference - Reference Databento's reference data service can be accessed programmatically over its HTTP API. To make it easier to integrate the API, we also provide official client libraries that simplify the code you need to write. Our HTTP API is designed as a collection of RPC-style methods, which can be called using URLs in the form `https://hist.databento.com/v0/METHOD_FAMILY.METHOD`. Our client libraries wrap these HTTP RPC-style methods with more idiomatic interfaces in their respective languages. --- Source: https://databento.com/docs/api-reference-data/basics?lang=python ## Basics --- Source: https://databento.com/docs/api-reference-data/basics/overview?lang=python ### Overview Our reference API has the following structure: - [Corporate actions](https://databento.com/docs/api-reference-data/corporate-actions.md) provides point-in-time (PIT) corporate actions events with global coverage. - [Adjustment factors](https://databento.com/docs/api-reference-data/adjustment-factors.md) provides end-of-day price adjustment factors for capital events, spanning multiple currencies for the same event. - [Security master](https://databento.com/docs/api-reference-data/security-master.md) provides up-to-date (latest) or point-in-time (PIT) static information on global market securities. --- Source: https://databento.com/docs/api-reference-data/basics/authentication?lang=python ### Authentication Databento uses API keys to authenticate requests. You can view and manage your keys on the [API keys]($PLATFORM_HOSTNAME/portal/keys) page of your portal. Each API key is a 32-character string starting with `db-`. By default, our library uses the environment variable `DATABENTO_API_KEY` as your API key. However, if you pass an API key to the `Reference` constructor through the `key` parameter, then this value will be used instead. Related: [Securing your API keys](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). #### Example usage ```python import databento as db # Establish connection and authenticate client = db.Reference("$YOUR_API_KEY") # Authenticated request print(client.security_master.get_last(symbols="AAPL")) ``` --- Source: https://databento.com/docs/api-reference-data/basics/symbology?lang=python ### Symbology The reference datasets support querying by `raw_symbol` and additionally symbology types listed in the below table (depending on the endpoint). The `raw_symbol` symbology type uses a standardized convention for the `symbols` parameter for requests. This convention is referred to as the "Nasdaq Integrated Platform Suffix" and is used consistently across our datasets, regardless of the suffix originally used by the exchange. - **Standardization.** This approach aligns with our symbology standardization for US equity datasets. For a detailed explanation, see our [symbology standards](https://databento.com/docs/conventions/symbology.md) documentation. - **Local Codes.** The `local_code` field retains the original symbol from the primary exchange. For example, symbols from NYSE-owned feeds such as NYSE Texas use the "CMS Suffix". For additional context on symbol conventions, including the four most common suffix patterns used for US equities, refer to [Nasdaq's guide on ticker symbol conventions](https://www.nasdaqtrader.com/trader.aspx?id=CQSsymbolconvention). The symbology related fields included with every data record are as follows: | Field | Description | | `nasdaq_symbol` | Nasdaq Integrated Platform Suffix convention symbol (standardized `raw_symbol`). | | `local_code` | Local code, typically the original symbol and unique at the listing level (but there are exceptions to this rule). Either an alpha string, or a number. | | `isin` | [**I**nternational **S**ecurities **I**dentification **N**umber (ISIN)](https://www.isin.org/) - ISO 6166. | | `us_code` | US domestic [**C**ommittee on **U**niform **S**ecurities **I**dentification **P**rocedure (CUSIP)](https://en.wikipedia.org/wiki/CUSIP) code. | | `bbg_comp_id` | Bloomberg composite global ID. | | `bbg_comp_ticker` | Bloomberg composite ticker. | | `figi` | **F**inancial **I**nstrument **G**lobal **I**dentifier (FIGI), formerly Bloomberg Global Identifier (BBGID). Listing level identifier. | | `figi_ticker` | Bloomberg exchange-level ticker. | | `listing_id` | Databento-specific identifier that tracks an individual exchange's listing of a security. | | `issuer_id` | Databento-specific identifier shared across all securities issued by the same company. | | `security_id` | Databento-specific identifier at the security level, tracking a security across all of its exchange listings. | > **Info** > > Note a symbol may match multiple securities across a time range if the symbol has been recycled. > **Tip** > > When identifying a listing on a specific venue, we suggest using the `nasdaq_symbol` field, as it aligns > with the normalized `raw_symbol` for instruments. Once you have the ISIN for the security, using > the `isin` symbology type will provide the most accurate results when querying for related records. --- Source: https://databento.com/docs/api-reference-data/basics/dates-and-times?lang=python ### Dates and times Our Python client library has several functions with timestamp arguments. These arguments will have type `pandas.Timestamp | datetime.date | str | int` and support a variety of formats. It's recommended to use [pandas.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), which fully supports timezones and nanosecond-precision. If a `datetime.date` is used, the time is set to midnight UTC. If an `int` is provided, the value is interpreted as [UNIX nanoseconds](https://en.wikipedia.org/wiki/Unix_time). The client library also handles several string-based timestamp formats based on [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html). - `yyyy-mm-dd`, e.g. `"2022-02-28"` (midnight UTC) - `yyyy-mm-ddTHH:MM`, e.g. `"2022-02-28T23:50"` - `yyyy-mm-ddTHH:MM:SS`, e.g. `"2022-02-28T23:50:59"` - `yyyy-mm-ddTHH:MM:SS.NNNNNNNNN`, e.g. `"2022-02-28T23:50:59.123456789"` Timezone specification is also supported. - `yyyy-mm-ddTHH:MMZ` - `yyyy-mm-ddTHH:MM±hh` - `yyyy-mm-ddTHH:MM±hhmm` - `yyyy-mm-ddTHH:MM±hh:mm` **Bare dates** Some parameters require a bare date, without a time. These arguments have type `datetime.date | str` and must either be a `datetime.date` object, or a string in `yyyy-mm-dd` format, e.g. `"2022-02-28"`. --- Source: https://databento.com/docs/api-reference-data/basics/errors?lang=python ### Errors Our reference API uses HTTP response codes to indicate the success or failure of an API request. The client library provides exceptions that wrap these response codes. - `2xx` indicates success. - `4xx` indicates an error on the client side. Represented as a `BentoClientError`. - `5xx` indicates an error with Databento's servers. Represented as a `BentoServerError`. The full list of the response codes and associated causes is as follows: | Code | Message | Cause | | 200 | OK | Successful request. | | 400 | Bad Request | Invalid request. Usually due to a missing, malformed or unsupported parameter. | | 401 | Unauthorized | Invalid username or API key. | | 403 | Forbidden | The API key has insufficient permissions to perform the request. | | 404 | Not Found | A resource is not found, or a requested symbol does not exist. | | 422 | Unprocessable Entity | The request is well formed, but we cannot or will not process the contained instructions. | | 429 | Too Many Requests | API rate limit exceeded. | | 500 | Internal Server Error | Unexpected condition encountered in our system. | | 503 | Service Unavailable | Data gateway is offline or overloaded. | | 504 | Gateway Timeout | Data gateway is available but other parts of our system are offline or overloaded. | #### API method ```python class databento.BentoError(Exception) class databento.BentoHttpError(databento.BentoError) class databento.BentoClientError(databento.BentoHttpError) class databento.BentoServerError(databento.BentoHttpError) ``` #### Example usage ```python import databento as db client = db.Reference("INVALID_API_KEY") try: print(client.security_master.get_last("TEST")) except db.BentoClientError as e: print(e) ``` #### Example response ```plaintext 400 auth_invalid_username_in_basic_auth Invalid username in Basic auth ('INVALID_API_KEY'). documentation: https://databento.com/docs ``` --- Source: https://databento.com/docs/api-reference-data/basics/rate-limits?lang=python ### Rate limits Our reference API allows each IP address up to: - 100 concurrent connections. - 100 [corporate actions](https://databento.com/docs/api-reference-data/corporate-actions.md) requests per second. - 100 [adjustment factors](https://databento.com/docs/api-reference-data/adjustment-factors.md) requests per second. - 100 [security master](https://databento.com/docs/api-reference-data/security-master.md) requests per second. When a request exceeds a rate limit, a `BentoClientError` exception is raised with a 429 error code. **Retry-After** The Retry-After response header indicates how long the user should wait before retrying. If you find that your application has been rate-limited, you can retry after waiting for the time specified in the Retry-After header. If you are using Python, you may use the time.sleep function as seen below to wait for the time specified in the Retry-After header. e.g. `time.sleep(int(response.headers("Retry-After", 1)))` This code snippet works best for our current APIs with their rate limits. Future APIs may have different rate limits, and might require a different default time delay. --- Source: https://databento.com/docs/api-reference-data/client?lang=python ## Client --- Source: https://databento.com/docs/api-reference-data/client/reference?lang=python ### Reference To access Databento's reference API, first create an instance of the `Reference` client. The entire API is exposed through instance methods of the client. Note that the API key can be passed as a parameter, which is [not recommended for production applications](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). Instead, you can leave out this parameter to pass your API key via the `DATABENTO_API_KEY` environment variable: Currently, only the `'bo1'` gateway is supported for historical data. #### API method ```python class Reference( key: str | None = None, gateway: HistoricalGateway | str = 'bo1', ) ``` #### Example usage ```python import databento as db # Pass as parameter client = db.Reference("$YOUR_API_KEY") # Or, pass as `DATABENTO_API_KEY` environment variable client = db.Reference() ``` #### Parameters | Parameter | Type | Required | Description | | `key` | str | optional | 32-character API key. Found on your [API keys]($PLATFORM_HOSTNAME/portal/keys) page. If `None` then `DATABENTO_API_KEY` environment variable is used. | | `gateway` | databento.HistoricalGateway or str | optional | Site of historical gateway to connect to. Currently only `'bo1'` is supported. If `None` then will connect to the default historical gateway. | --- Source: https://databento.com/docs/api-reference-data/corporate-actions?lang=python ## Corporate actions --- Source: https://databento.com/docs/api-reference-data/corporate-actions/corporate-actions-get-range?lang=python ### Reference.corporate\_actions.get\_range Make a streaming request for corporate actions time series data from Databento. Each record represents a unique corporate actions event at the [listing](https://databento.com/docs/faqs/instruments-and-products.md#listings) level. The specified `index` will be used to filter the time range and order the records. It will also be set as the index of the resulting data frame. > **See also** > > [Corporate actions](https://databento.com/docs/schemas/corporate-actions.md) dataset guide for further details. #### API method ```python Reference.corporate_actions.get_range( start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, index: str = "event_date", symbols: Iterable[str] | str | None = None, stype_in: SType | str = "raw_symbol", events: Iterable[str] | str | None = None, countries: list[str] | str | None = None, exchanges: list[str] | str | None = None, security_types: list[str] | str | None = None, flatten: bool = True, pit: bool = False, allocate_isins: bool = True, ) -> pd.DataFrame ``` #### Parameters | Parameter | Type | Required | Description | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range based on `index`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range based on `index`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. If `None`, then will return **all** data available after `start`. | | `index` | str, default 'event\_date' | optional | The index column to use for filtering the `start` and `end` time range and for record ordering. Use any of 'event\_date', 'ex\_date' or 'ts\_record'. Records with a `NULL` value in the specified `index` column will be excluded. | | `symbols` | list or str | optional | The symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols by default. | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-reference-data/basics/symbology.md) of input `symbols`. Use any of 'raw\_symbol', 'nasdaq\_symbol', 'isin', 'us\_code', 'bbg\_comp\_id', 'bbg\_comp\_ticker', 'figi', 'figi\_ticker', 'listing\_id', 'issuer\_id', 'security\_id'. | | `events` | list or str | optional | The event types to filter for. Takes any number of [event](https://databento.com/docs/conventions/corporate-actions-events.md)s per request. If not specified then will select **all** event types by default. See [EVENT](https://databento.com/docs/conventions/reference-data-enums.md#event) enum. | | `countries` | list or str | optional | The listing countries to filter for. Takes any number of country codes per request as two letter ISO 3166-1 alpha-2 strings. If not specified then will select **all** countries by default. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `exchanges` | list or str | optional | The (listing) exchanges to filter for. Takes any number of exchanges per request. If not specified then will select **all** exchanges by default. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `security_types` | list or str | optional | The security types to filter for. If not specified then will select **all** security types by default. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `flatten` | bool, default True | optional | If nested JSON objects within the `date_info`, `rate_info`, and `event_info` fields should be flattened into separate columns in the resulting data frame. | | `pit` | bool, default False | optional | Determines whether to retain all historical records or only the latest records. If True, all historical records for each `event_unique_id` will be retained, preserving the complete point-in-time history. If False (default), the data frame will include only the most recent record for each `event_unique_id` based on the `ts_record` timestamp. | | `allocate_isins` | bool, default True | optional | Whether the request should allocate any new ISINs for plans that are ISIN-limited. The request will drop any rows that would create new allocations. | #### Returns `pd.Dataframe` The data converted into a DataFrame. > **See also** > > Corporate actions [field descriptions](https://databento.com/docs/schemas/corporate-actions.md#fields) for further details. #### Example usage ```python import databento as db client = db.Reference("$YOUR_API_KEY") df_raw = client.corporate_actions.get_range( symbols=["AAPL"], stype_in="raw_symbol", start="2024-01", events=["SHOCH"], countries=["US"], ) print(df_raw.head()) ``` #### Example response ```plaintext event_unique_id ts_record event_id listing_id listing_group_id security_id issuer_id event_action event event_subtype event_date_label event_created_date effective_date ex_date record_date record_date_id related_event related_event_id global_status listing_status listing_source listing_date delisting_date issuer_name security_type security_description primary_exchange exchange operating_mic symbol nasdaq_symbol local_code isin us_code bbg_comp_id bbg_comp_ticker figi figi_ticker listing_country register_country trading_currency multi_currency segment_mic_name segment_mic mand_volu_flag rd_priority lot_size par_value par_value_currency payment_date duebills_redemption_date from_date to_date registration_date start_date end_date open_date close_date start_subscription_date end_subscription_date option_election_date withdrawal_rights_from_date withdrawal_rights_to_date notification_date financial_year_end_date exp_completion_date payment_type option_id serial_id default_option_flag rate_currency ratio_old ratio_new fraction outturn_style outturn_security_type outturn_security_id outturn_isin outturn_us_code outturn_local_code outturn_bbg_comp_id outturn_bbg_comp_ticker outturn_figi outturn_figi_ticker min_offer_qty max_offer_qty min_qualify_qty max_qualify_qty min_accept_qty max_accept_qty tender_strike_price tender_price_step option_expiry_time option_expiry_tz withdrawal_rights_flag withdrawal_rights_expiry_time withdrawal_rights_expiry_tz expiry_time expiry_tz ts_created old_outstanding_date new_outstanding_date old_shares_outstanding new_shares_outstanding event_date 2024-01-02 U-40685533904-135825 2024-01-12 13:12:12+00:00 E-5533904-SHOCH L-135825 LG-35825 S-33449 I-30017 I SHOCH NaN effective_date 2024-01-12 2024-01-02 None None NaN SHOCH NaN A L M 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW US US USD False Global Select XNGS M NaN 100 0.00001 USD None None None None None None None None None None None None NaN None None None None NaN NaN NaN False NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaT NaN NaN NaT NaN NaT NaN 2024-07-24 06:32:10.375440+00:00 2023-10-20T00:00:00 2024-01-02T00:00:00 15552752000 15461896000 2024-01-19 U-40685575281-135825 2024-02-05 03:37:49+00:00 E-5575281-SHOCH L-135825 LG-35825 S-33449 I-30017 I SHOCH NaN effective_date 2024-02-03 2024-01-19 None None NaN SHOCH NaN A L M 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW US US USD False Global Select XNGS M NaN 100 0.00001 USD None None None None None None None None None None None None NaN None None None None NaN NaN NaN False NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaT NaN NaN NaT NaN NaT NaN 2024-07-24 22:44:12.284177+00:00 2024-01-02T00:00:00 2024-01-19T00:00:00 15461896000 15441881000 2024-04-19 U-40685752069-135825 2024-05-06 13:21:00+00:00 E-5752069-SHOCH L-135825 LG-35825 S-33449 I-30017 I SHOCH NaN effective_date 2024-05-06 2024-04-19 None None NaN SHOCH NaN A L M 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW US US USD False Global Select XNGS M NaN 100 0.00001 USD None None None None None None None None None None None None NaN None None None None NaN NaN NaN False NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaT NaN NaN NaT NaN NaT NaN 2024-07-25 01:52:32.148864+00:00 2024-01-19T00:00:00 2024-04-19T00:00:00 15441881000 15334082000 2024-07-19 U-40685933868-135825 2024-08-05 03:41:52+00:00 E-5933868-SHOCH L-135825 LG-35825 S-33449 I-30017 I SHOCH NaN effective_date 2024-08-03 2024-07-19 None None NaN SHOCH NaN A L M 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW US US USD False Global Select XNGS M NaN 100 0.00001 USD None None None None None None None None None None None None NaN None None None None NaN NaN NaN False NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaT NaN NaN NaT NaN NaT NaN 2024-08-05 06:00:56.552900+00:00 2024-04-19T00:00:00 2024-07-19T00:00:00 15334082000 15204137000 ``` --- Source: https://databento.com/docs/api-reference-data/corporate-actions/corporate-actions-list-events?lang=python ### Reference.corporate\_actions.list\_events Requests documentation for all supported corporate action event types, including their fields, subtypes, and applicable calendar dates. #### API method ```python Reference.corporate_actions.list_events() -> dict[str, dict] ``` #### Returns `dict[str, dict]` The raw JSON response as a dict keyed by event code (e.g. `AGM`), where each value is: | Field | Type | Description | | `calendar_dates` | list[dict] or None | The calendar dates relevant to the event. | | `category` | str or None | The category of the event (`distribution`, `distribution_debit`, `legal_action`, `proposal`, `reorganisation`, `static_reference`, `tax_related`, `other`). | | `code` | str or None | The code for the event (e.g. `AGM`). | | `description` | str or None | Description of the event. | | `fields` | list[dict] or None | The fields relevant to the event. | | `level` | str | The level the event applies to (`country`, `issuer`, `listing`, `security`). | | `name` | str | Name of the event. | | `participation` | str or None | Whether the event is mandatory, voluntary, or a mix of the two (`mandatory`, `voluntary`, `mandatory_voluntary`). | | `subtypes` | list[dict] or None | The subtypes relevant to the event. | > **See also** > > [Corporate actions events](https://databento.com/docs/datasets/corporate-actions.md#events) guide for information in text form. #### Example usage ```python import databento as db client = db.Reference("$YOUR_API_KEY") event_docs = client.corporate_actions.list_events() print(event_docs["AGM"]) ``` #### Example response ```json { "code": "AGM", "name": "Company Meeting", "level": "issuer", "category": "proposals", "participation": "voluntary", "description": "Annual General meeting of shareholders is held once a year, usually after the end of the financial year. Also includes other meeting types such as Extraordinary General Meeting, Special General Meeting, Court Ordered Meeting, and Postal Ballot.", "subtypes": [ { "code": "AGM", "description": "Annual General Meeting" }, { "code": "BHM", "description": "Bond Holder Meeting" }, { "code": "CGM", "description": "Court Ordered General Meeting" }, { "code": "EGM", "description": "Extraordinary General Meeting" }, { "code": "GM", "description": "General Meeting" }, { "code": "SGM", "description": "Special General Meeting" } ], "calendar_dates": [ { "name": "event_date", "alias": "meeting_date" }, { "name": "effective_date", "alias": "meeting_date" } ], "fields": [ { "group": "event_info", "name": "meeting_number", "description": "Company Meeting Number" }, { "group": "event_info", "name": "meeting_time", "description": "Local time for Start of Meeting" }, { "group": "event_info", "name": "address_line1", "description": "Address line 1" }, { "group": "event_info", "name": "address_line2", "description": "Address line 2" }, { "group": "event_info", "name": "address_line3", "description": "Address line 3" }, { "group": "event_info", "name": "address_line4", "description": "Address line 4" }, { "group": "event_info", "name": "address_line5", "description": "Address line 5" }, { "group": "event_info", "name": "address_line6", "description": "Address line 6" }, { "group": "event_info", "name": "city", "description": "City" }, { "group": "event_info", "name": "country", "description": "Country Code - ISO 3166-1" } ] } ``` --- Source: https://databento.com/docs/api-reference-data/corporate-actions/corporate-actions-list-enums?lang=python ### Reference.corporate\_actions.list\_enums Requests documentation for the full set of enumerations used across corporate actions data, like event actions and statuses. #### API method ```python Reference.corporate_actions.list_enums() -> dict[str, list[dict]] ``` #### Returns `dict[str, list[dict]]` The raw JSON response as a dict keyed by enum group name (e.g. `ACTION`). Each value is a list of variants: | Field | Type | Description | | `code` | str or None | Code for the enum variant. | | `description` | str | Description of the enum variant. | > **See also** > > [Reference data enums](https://databento.com/docs/conventions/reference-data-enums.md) for a lookup table of possible enumeration field values. #### Example usage ```python import databento as db client = db.Reference("$YOUR_API_KEY") enum_docs = client.corporate_actions.list_enums() print(enum_docs["ACTION"]) ``` #### Example response ```json [ { "code": "C", "description": "Cancelled" }, { "code": "D", "description": "Deleted" }, { "code": "I", "description": "Inserted" }, { "code": "P", "description": "Payment details Cancelled by Issuer" }, { "code": "Q", "description": "Payment details Deleted by Data Supplier" }, { "code": "U", "description": "Updated" } ] ``` --- Source: https://databento.com/docs/api-reference-data/adjustment-factors?lang=python ## Adjustment factors --- Source: https://databento.com/docs/api-reference-data/adjustment-factors/adjustment-factors-get-range?lang=python ### Reference.adjustment\_factors.get\_range Make a streaming request for adjustment factors time series data from Databento. Each record represents a unique corporate actions event with an adjustment factor at the [listing](https://databento.com/docs/faqs/instruments-and-products.md#listings) level. The `ex_date` column will be used to filter the time range and order the records. It will also be set as the index of the resulting data frame. > **See also** > > [Adjustment factors](https://databento.com/docs/datasets/adjustment-factors.md) dataset guide for further details. #### API method ```python Reference.adjustment_factors.get_range( start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, symbols: Iterable[str] | str | None = None, stype_in: SType | str = "raw_symbol", countries: list[str] | str | None = None, security_types: list[str] | str | None = None, allocate_isins: bool = True, ) -> pd.DataFrame ``` #### Parameters | Parameter | Type | Required | Description | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range based on `ex_date`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range based on `ex_date`. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. If `None`, then will return **all** data available after `start`. | | `symbols` | list or str | optional | The symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols by default. | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-reference-data/basics/symbology.md) of input `symbols`. Use any of 'raw\_symbol', 'nasdaq\_symbol', 'isin', 'us\_code', 'security\_id'. | | `countries` | list or str | optional | The listing countries to filter for. Takes any number of country codes per request as two letter ISO 3166-1 alpha-2 strings. If not specified then will select **all** countries by default. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `security_types` | list or str | optional | The security types to filter for. If not specified then will select **all** security types by default. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `allocate_isins` | bool, default True | optional | Whether the request should allocate any new ISINs for plans that are ISIN-limited. The request will drop any rows that would create new allocations. | #### Returns `pd.Dataframe` The data converted into a DataFrame. > **See also** > > Adjustment factors [field descriptions](https://databento.com/docs/schemas/adjustment-factors.md#fields) for further details. #### Example usage ```python import databento as db client = db.Reference("$YOUR_API_KEY") df_raw = client.adjustment_factors.get_range( symbols=["MSFT"], stype_in="raw_symbol", start="2019", end="2020", countries=["US"], ) print(df_raw.head()) ``` #### Example response ```plaintext security_id event_id event issuer_name security_type ... dividend_currency frequency option detail ts_created ex_date ... 2019-02-20 S-39827 E-1443840-DIV DIV Microsoft Corporation EQS ... USD INT 1 INT Dividend (cash) of USD0.46/EQS 2024-09-23 00:25:46.589794+00:00 2019-05-15 S-39827 E-1550148-DIV DIV Microsoft Corporation EQS ... USD FNL 1 FNL Dividend (cash) of USD0.46/EQS 2024-09-23 00:28:54.651339+00:00 2019-08-14 S-39827 E-1617896-DIV DIV Microsoft Corporation EQS ... USD INT 1 INT Dividend (cash) of USD0.46/EQS 2024-09-23 00:32:47.230682+00:00 2019-11-20 S-39827 E-1669990-DIV DIV Microsoft Corporation EQS ... USD INT 1 INT Dividend (cash) of USD0.51/EQS 2024-09-23 00:36:33.357489+00:00 [4 rows x 27 columns] ``` --- Source: https://databento.com/docs/api-reference-data/security-master?lang=python ## Security master --- Source: https://databento.com/docs/api-reference-data/security-master/security-master-get-last?lang=python ### Reference.security\_master.get\_last Make a streaming request for the latest security master data from Databento. Each record represents a unique [listing](https://databento.com/docs/faqs/instruments-and-products.md#listings) for a security. The resulting data frame will be indexed and sorted by `ts_effective`. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. #### API method ```python Reference.security_master.get_last( symbols: Iterable[str] | str | None = None, stype_in: SType | str = "raw_symbol", countries: list[str] | str | None = None, security_types: list[str] | str | None = None, allocate_isins: bool = True, ) -> pd.DataFrame ``` #### Parameters | Parameter | Type | Required | Description | | `symbols` | list or str | optional | The symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols by default. | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-reference-data/basics/symbology.md) of input `symbols`. Use any of 'raw\_symbol', 'nasdaq\_symbol', 'isin', 'us\_code', 'bbg\_comp\_id', 'bbg\_comp\_ticker', 'figi', 'figi\_ticker', 'listing\_id', 'issuer\_id', 'security\_id'. | | `countries` | list or str | optional | The listing countries to filter for. Takes any number of country codes per request as two letter ISO 3166-1 alpha-2 strings. If not specified then will select **all** countries by default. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `security_types` | list or str | optional | The security types to filter for. If not specified then will select **all** security types by default. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `allocate_isins` | bool, default True | optional | Whether the request should allocate any new ISINs for plans that are ISIN-limited. The request will drop any rows that would create new allocations. | #### Returns `pandas.Dataframe` The data converted into a DataFrame. > **See also** > > Security master [field descriptions](https://databento.com/docs/schemas/security-master.md#fields) for further details. #### Example usage ```python import databento as db client = db.Reference("$YOUR_API_KEY") df_raw = client.security_master.get_last( symbols=["AAPL"], countries=["US"], ) print(df_raw.head()) ``` #### Example response ```plaintext ts_record listing_id listing_group_id security_id issuer_id listing_status listing_source listing_created_date listing_date delisting_date issuer_name security_type security_description primary_exchange exchange operating_mic symbol nasdaq_symbol local_code isin us_code bbg_comp_id bbg_comp_ticker figi figi_ticker fisn lei sic cik gics naics cic cfi incorporation_country listing_country register_country trading_currency multi_currency segment_mic_name segment_mic structure lot_size par_value par_value_currency voting vote_per_sec shares_outstanding shares_outstanding_date ts_created ts_effective 2024-07-19 00:00:00+00:00 2024-08-05 02:41:52+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 15204137000 2024-07-19 2024-11-11 22:41:29.670132+00:00 ``` --- Source: https://databento.com/docs/api-reference-data/security-master/security-master-get-range?lang=python ### Reference.security\_master.get\_range Make a streaming request for security master point-in-time (PIT) time series data from Databento. Each record represents the status of a unique [listing](https://databento.com/docs/faqs/instruments-and-products.md#listings) for a security with an effective date. The specified `index` will be used to filter the time range and order the records. It will also be set as the index of the resulting data frame. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. #### API method ```python Reference.security_master.get_range( start: pd.Timestamp | datetime | date | str | int, end: pd.Timestamp | datetime | date | str | int | None = None, index: str = "ts_effective", symbols: Iterable[str] | str | None = None, stype_in: SType | str = "raw_symbol", countries: list[str] | str | None = None, security_types: list[str] | str | None = None, allocate_isins: bool = True, ) -> pd.DataFrame ``` #### Parameters | Parameter | Type | Required | Description | | `start` | pd.Timestamp, datetime, date, str, or int | required | The inclusive start of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. | | `end` | pd.Timestamp, datetime, date, str, or int | optional | The exclusive end of the request range. Takes [pd.Timestamp](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html), [Python datetime](https://docs.python.org/3/library/datetime.html#datetime-objects), [Python date](https://docs.python.org/3/library/datetime.html#date-objects), [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) string, or [UNIX timestamp](https://en.wikipedia.org/wiki/Unix_time) in nanoseconds. Assumes UTC as timezone unless otherwise specified. If `None`, then will return **all** data available after `start`. | | `index` | str, default 'ts\_effective' | optional | The index column to use for filtering the `start` and `end` time range and record ordering. Use any of 'ts\_effective' or 'ts\_record'. | | `symbols` | list or str | optional | The symbols to filter for. Takes up to 2,000 symbols per request. If more than 1 symbol is specified, the data is merged and sorted by time. If `'ALL_SYMBOLS'` or `None` then will select **all** symbols by default. | | `stype_in` | SType or str, default 'raw\_symbol' | optional | The [symbology type](https://databento.com/docs/api-reference-data/basics/symbology.md) of input `symbols`. Use any of 'raw\_symbol', 'nasdaq\_symbol', 'isin', 'us\_code', 'bbg\_comp\_id', 'bbg\_comp\_ticker', 'figi', 'figi\_ticker', 'listing\_id', 'issuer\_id', 'security\_id'. | | `countries` | list or str | optional | The listing countries to filter for. Takes any number of country codes per request as two letter ISO 3166-1 alpha-2 strings. If not specified then will select **all** countries by default. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `security_types` | list or str | optional | The security types to filter for. If not specified then will select **all** security types by default. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `allocate_isins` | bool, default True | optional | Whether the request should allocate any new ISINs for plans that are ISIN-limited. The request will drop any rows that would create new allocations. | #### Returns `pd.Dataframe` The data converted into a DataFrame. > **See also** > > Security master [field descriptions](https://databento.com/docs/schemas/security-master.md#fields) for further details. #### Example usage ```python import databento as db client = db.Reference("$YOUR_API_KEY") df_raw = client.security_master.get_range( symbols=["AAPL"], countries=["US"], start="2005-01-01", ) print(df_raw.head(10)) ``` #### Example response ```plaintext ts_record listing_id listing_group_id security_id issuer_id listing_status listing_source listing_created_date listing_date delisting_date issuer_name security_type security_description primary_exchange exchange operating_mic symbol nasdaq_symbol local_code isin us_code bbg_comp_id bbg_comp_ticker figi figi_ticker fisn lei sic cik gics naics cic cfi incorporation_country listing_country register_country trading_currency multi_currency segment_mic_name segment_mic structure lot_size par_value par_value_currency voting vote_per_sec shares_outstanding shares_outstanding_date ts_created ts_effective 2004-11-19 00:00:00+00:00 2004-12-06 15:09:30+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 402057856 2004-11-19 2024-11-11 22:41:24.780487+00:00 2005-01-25 00:00:00+00:00 2005-02-03 16:00:32+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 408585970 2005-01-25 2024-11-11 22:41:24.780487+00:00 2005-02-28 00:00:00+00:00 2005-03-01 15:28:04+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 817171940 2005-02-28 2024-11-11 22:41:24.780487+00:00 2005-04-22 00:00:00+00:00 2005-05-06 14:06:41+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 823933916 2005-04-22 2024-11-11 22:41:24.780487+00:00 2005-07-22 00:00:00+00:00 2005-08-04 14:39:03+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 829829296 2005-07-22 2024-11-11 22:41:24.780487+00:00 2005-11-18 00:00:00+00:00 2005-12-02 15:25:18+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 842767948 2005-11-18 2024-11-11 22:41:24.780487+00:00 2006-01-25 00:00:00+00:00 2006-02-07 15:32:26+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 848612359 2006-01-25 2024-11-11 22:41:24.780487+00:00 2006-04-26 00:00:00+00:00 2006-05-09 14:36:54+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 850508144 2006-04-26 2024-11-11 22:41:24.780487+00:00 2006-12-13 00:00:00+00:00 2007-01-02 15:42:39+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Computer Inc. EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 859273757 2006-12-13 2024-11-11 22:41:24.780487+00:00 2007-01-11 00:00:00+00:00 2007-01-11 00:00:00+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 859273757 2006-12-13 2024-11-11 22:41:29.670132+00:00 ``` --- Source: https://databento.com/docs/architecture [//]: # 'template: one-column-L2' [//]: # 'template: examples-nav' # Architecture Learn about Databento's architecture and its implications on timestamping, data quality, and performance optimization. | Title | Subtitle | Image | Supported languages | | [Databento architecture](https://databento.com/docs/architecture/architecture-diagram.md) | Databento's self-hosted infrastructure design and how our data is captured. | ![Architecture image](https://databento.com/docs/architecture.svg.md) | Data capture, Data quality | | [Timestamping](https://databento.com/docs/architecture/timestamping-guide.md) | Details on our nanosecond-resolution timestamps and PTP time synchronization. | ![Timestamping image](https://databento.com/docs/timestamping.png.md) | Data capture, Latency | | [Locations and network connectivity](https://databento.com/docs/architecture/locations-network-connectivity.md) | Compare our Internet, dedicated, and peering connectivity options by use case. | ![Locations and connectivity image](https://databento.com/docs/connectivity.svg.md) | Connectivity, Latency | | [Dedicated connectivity](https://databento.com/docs/architecture/dedicated-connectivity-guide.md) | Private connectivity and colocation options for higher throughput or lower latency. | ![Dedicated connectivity image](https://databento.com/docs/dedicated-connectivity.svg.md) | Connectivity, Latency | | [Databento NTP service](https://databento.com/docs/architecture/ntp-service.md) | Synchronize your clock against our free NTP service. | ![NTP diagram image](https://databento.com/docs/ntp-service.svg.md) | Latency | | [Performance optimization](https://databento.com/docs/architecture/performance-optimization.md) | Frequently-asked questions on locations, setup, performance, hardware, and more. | ![Performance FAQ image](https://databento.com/docs/optimization.svg.md) | Connectivity, Latency | --- Source: https://databento.com/docs/architecture/architecture-diagram # Databento architecture Databento's infrastructure is designed with the following considerations: - **Redundancy and high availability.** There are at least two redundant servers, connections, and network devices in every hop of the critical path between us and your application. - **Self-hosted.** All our servers are self-hosted. We only rely on a cloud provider -- AWS -- for our public DNS servers, the CDN for our website, and independent monitoring of our service uptime. - **Low latency.** Our live data architecture is optimized for low-latency use cases, such as algorithmic trading. - **High data quality.** We employ several techniques to achieve lossless capture and highly accurate timestamping. ### Live data architecture Our live data architecture is composed of the following elements: - Colocation and direct connectivity to venue extranet - Low latency switches and interconnects - Lossless capture servers - Zero-copy messaging - Real-time data gateways - SmartNIC-accelerated load balancing, firewall, and routing - Direct Server Return (DSR) for load balancing - Multiple enterprise IP transit providers - Inline historical data capture and storage Users connecting to us over a public network from within the same data center or metro area, using our [Raw API](https://databento.com/docs/api-live.md) and [Databento Binary Encoding](https://databento.com/docs/conventions/databento-binary-encoding.md), can expect to receive data less than 1 millisecond after the venue disseminates the original data from their matching engine. ![Databento architecture diagram](https://databento.com/docs/databento-architecture.svg.md) ### Colocation and direct connectivity to venue extranet Our capture servers and real-time data gateways are generally colocated in the same primary data center as the venue's matching engine. We receive the data through a pair of direct connections (also called **handoffs**) to the venue's private network. This minimizes both the latency of our service and the variance of the network topology between us and the matching engine, yielding highly deterministic timestamps. The venue will sometimes employ **fiber equalization** to ensure the same latency is achieved at handoffs across multiple data centers. In these cases, we may colocate our capture servers in any one of the data centers with equalized latency. The only exception to primary data center colocation is when there are microstructural reasons for our trading users to prefer receiving the data elsewhere. These may include a central location for fragmented venues (like US equities), or proximity to a demarcation point for long-haul connectivity. > **Info** > > You can find the exact locations of our capture servers for each venue from > our [list of venues](https://databento.com/docs/datasets.md). ### Low latency switches and interconnects Depending on the specific venue, the data takes one to three switch hops after the handoff to reach our capture server. One of those switch hops, at most, may be a cross-connect within the same data center. We use a mix of Layer 1 and 3 switches, with 4 nanosecond and sub-300 nanosecond port-to-port latencies, respectively. `ts_in_delta` represents the time it takes (in nanoseconds) for a data message to travel from the venue to our capture server's network card. > **Info** > > You can find the schemas that include the `ts_in_delta` field from our > [list of schemas](https://databento.com/docs/schemas.md). ### Lossless capture servers Most venues disseminate their data over UDP multicast and provide a pair of multicast streams to offer protection against packet loss. Each of our capture servers receives both streams and compares packet sequence numbers to fill data gaps in the first stream. We call this **packet arbitration** or **A/B arbitration**. We perform a second layer of packet arbitration between our capture servers to further reduce the risk of data loss. This is a proprietary part of our system that we call **X/Y arbitration**. As such, our setup receives at least four redundant streams of the data. Most software-based capture processes are unable to sustain 100% line rate capture during the 10 to 40 gigabit bursts that we constantly encounter from the data feeds. To address this, we offload the capture of the packets to an FPGA on each capture server, which reduces packet loss down to the order of one in a billion packets per stream. Combined with our packet arbitration process across four redundant streams, our process significantly mitigates the risk of data loss on the billions of packets that we receive daily on full venue feeds. ### Zero-copy messaging We employ a brokerless messaging pattern with a zero-copy internal messaging protocol to support our low latency, high throughput requirements. Once processed in our capture servers, the data only takes a single sub-300 nanosecond switch hop to our real-time data gateways. ### Real-time data gateways The data is preprocessed in real time to support the various real-time customizations available on our live service. This includes parsing, normalization, and book construction. Depending on the venue, about 5 to 35 microseconds are spent in our capture server including through these preprocessing steps. Our real-time data gateways are responsible for live data distribution. This includes negotiation of TCP sessions, inline data customization, transcoding, and handling of requests that vary from client to client. Our data spends about 5 microseconds in this layer before it takes another switch hop either to a customer's link (for customers connected over private cross-connect or peering) or to a network gateway that handles load balancing, firewall, and BGP routing (for customers connected over internet). > **Info** > > We also allow users to bypass our firewall with a > private cross-connect or peering to achieve **sub-41.3 microsecond** > latencies. Learn more from our [Databento locations and connectivity options](https://databento.com/docs/architecture/locations-network-connectivity.md) > guide or [contact us]($PLATFORM_HOSTNAME/support) for details. The time that the data terminates on the venue's handoff to the time that our processed data leaves any of our real-time data gateways is provided in our data as `ts_out` - `ts_recv`. > **Info** > > Receiving `ts_out` is enabled through a parameter at the authentication stage of the Raw API > and in the constructors of the Live clients. ### SmartNIC-accelerated load balancing, firewall, and routing We employ DPDK-based, SmartNIC-accelerated network gateways that process packets up to 100 gigabit line rate with a median latency of 64 microseconds and 99th percentile latency of 311 microseconds. ### Direct Server Return (DSR) for load balancing In a traditional load balancing setup both incoming and outgoing traffic pass through a load balancer adding latency and another potential bottleneck for a high-volume streaming application. With DSR, incoming client traffic still passes through our load balancers; however, now our client servers respond and push data directly to clients bypassing the load balancer hop altogether. ### Multiple enterprise IP transit providers To maintain stable routes from public internet, we multiplex connections from multiple IP transit providers and private peering partners on our network gateways. ### Inline historical data capture and storage We capture our historical data on the same servers that normalize and disseminate our live data in real time. This ensures our historical data has the same timestamps as our live data. Historical data is stored on a self-hosted, highly available storage cluster. To protect data integrity, our storage system has built-in detection and self-healing of silent bit rot. The stored data is then made available through our historical API servers after a short delay, though licensing restrictions may limit when you can access this data. To ensure high uptime, our historical API servers are deployed as multiple instances in our cluster. > **See also** > > You can read our docs on [precision, accuracy, and resolution](https://databento.com/docs/architecture/timestamping-guide.md#precision-accuracy-and-resolution) > and [timestamps on Databento and how to use them](https://databento.com/docs/architecture/timestamping-guide.md#nanosecond-resolution-timestamps) > for more details about our data integrity and timestamping accuracy. --- Source: https://databento.com/docs/architecture/timestamping-guide # Timestamping ![Databento timestamping diagram](https://databento.com/docs/timestamps.svg.md) ### Precision, accuracy, and resolution In our documentation, we quantify these terms as follows: - **Precision.** The maximum deviation between timestamps delivered in the data and any reference time. - **Accuracy.** The maximum deviation between timestamps delivered in the data and Coordinated Universal Time (UTC). - **Resolution.** An arbitrary unit of time. The minimum increment of timestamps delivered in the data. - **Jitter.** Closely related to precision. The maximum deviation of a clock's periodicity from its intended periodicity. > **Info** > > Unfortunately, common speech uses the opposite sign convention. > A time source with **high precision** or **accuracy** is said to have low > maximum deviation. Likewise, **high resolution** refers to a small minimum > timestamp increment. A clock with **high jitter** has low precision. The following diagram helps to visualize these terms: ![Databento timestamping diagram](https://databento.com/docs/precision-vs-accuracy-vs-resolution.svg.md) ### Nanosecond resolution timestamps You may have seen other data sources that claim to provide nanosecond-*resolution* timestamps. But this is a meaningless claim on its own, as any device is capable of nanosecond-resolution timestamps! You can guess the current time and write a number arbitrarily down by hand to the nanosecond resolution - that doesn't mean that your body is capable of nanosecond-*precision* timestamping. Likewise, you will find that most sources of financial data that carry nanosecond-resolution timestamps will use software-based timestamping that relies on CPU registers. There are a host of issues with this: - The precision of CPU registers varies substantially with thermal conditions. - The timestamping application is subject to sources of high jitter, such as interrupts, kernel scheduler events, context switching, and interference from other applications etc. This is especially the case with data providers that use cloud-based virtual machines to capture the data. To safeguard against these issues, we offload our timestamping to hardware, for instance, FPGA, which achieves more deterministic results. Depending on the techniques used, software timestamps may have a precision of anywhere from 100 nanoseconds to over 10 milliseconds. In contrast, our hardware timestamps achieve a precision from 4 to 10 nanoseconds (depending on the market). ### Time synchronization with Precision Time Protocol You will also find that most sources of financial data rely on Network Time Protocol (NTP) to synchronize their clocks. In practice, software timestamps relying on NTP synchronization against public NTP servers can deviate from UTC up to the order of 100 milliseconds. Typical implementations of Precision Time Protocol (PTP) will achieve accuracy from sub-microsecond to somewhere under 50 microseconds. At each data center, Databento uses a pair of GPS grandmaster clocks with PTP class 6 and clock accuracy under 25 ns against UTC as the time source. Our data servers achieve sub-microsecond synchronization to these GPS clocks. ### Accuracy, precision, and resolution of Databento timestamps Databento's data is intended to be suitable for high fidelity simulation, including backtesting of passive and arbitrage strategies, order routing, transaction cost analysis, and execution research. Our receive timestamps (`ts_recv`) have the following properties: - **Precision.** We use hardware timestamping to achieve a precision from 4 to 10 nanoseconds. - **Accuracy.** We use PTP time synchronization against GPS receiver-based time sources that are backed up by rubidium atomic oscillators to ensure holdover accuracy. Our timestamps achieve an accuracy from under 100 nanoseconds to under 1 microsecond, depending on the capture server's location. - **Resolution.** All of our timestamps are in nanosecond resolution. Any exception to the above for a given dataset is stated in the [detail page](https://databento.com/docs/schemas.md) of the dataset. Such exceptions generally arise when we have to backfill data for other sources or use different hardware for a particular market. There is no guarantee that timestamps embedded by original publisher or market (`ts_event`, `ts_in_delta`) will have high accuracy, precision, or resolution. Some markets still use millisecond resolution timestamps, inherited from legacy FIX protocol. Many markets will use NTP time synchronization and software timestamps. Often, our timestamps will have even higher precision and accuracy than those provided by the market itself. --- Source: https://databento.com/docs/architecture/locations-network-connectivity # Locations and network connectivity ![North America locations](https://databento.com/docs/locations.svg.md) ### Overview The following table provides an overview of the various connectivity options supported by Databento and their recommended use cases. | Connectivity option | Services | Best for | Ports | Latency (90th) | | [Internet](https://databento.com/docs/architecture/locations-network-connectivity.md#connecting-via-internet) | Live, Historical | Lowest cost | | 0.5+ ms | | Dedicated: [Interconnect with AWS, Google Cloud or Microsoft Azure](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#interconnect-with-aws-google-cloud-or-microsoft-azure) | Live, Historical | Mission-critical, cloud-based applications requiring high uptime or consuming entire feed(s) | 1G | 1.7+ ms | | Dedicated: [Interconnect at proximity hosting location](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#interconnect-at-proximity-hosting-location) | Live, Historical | Mission-critical, self-hosted applications requiring high uptime or consuming entire feed(s) | 1G | 0.5+ ms | | Dedicated: [Cross-connect with any colocation or managed services provider (MSP)](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#cross-connect-with-any-colocation-or-managed-services-provider-msp) | Live | Applications requiring lowest latency | 10G, 25G | 42.4 μs | | Dedicated: [Colocation with Databento](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#colocation-with-databento) | Live | Applications requiring lowest latency | 10G, 25G | 42.4 μs | | [Peering](https://databento.com/docs/architecture/locations-network-connectivity.md#connecting-via-peering) | Live, Historical | Colocation, hosting and extranet providers, MSPs | 10G, 25G | 0.5+ ms | ### Connecting via internet By default, our client libraries will connect to our gateways over internet, using our public DNS hostnames. Databento's services are designed to be efficient when consumed over internet. We use a mix of compression and normalization techniques to minimize bandwidth requirements of our data services, so that you can even consume full venue feeds--which would typically require a 10+ gigabit cross-connect--over a regular internet connection. Hosting your application in close geographical proximity to our gateways is the first step you can take towards ensuring a fast and stable connection. To help you make that determination, here are the locations of our gateways as well as hostnames that you can run a ping or traceroute against: | Service | Nearest metro | Data center | Address | Public DNS hostname | | Historical (BO1) | Boston | CoreSite BO1 | 70 Inner Belt Rd, Somerville, MA | `hist.databento.com` | | Live (DC3) | Chicago | CyrusOne Aurora I | 2905 Diehl Rd, Aurora, IL 60502 | `dc3.databento.com` | | Live (NY4) | New York | Equinix NY4 | 755 Secaucus Rd, Secaucus, NJ 07094 | `ny4.databento.com` | There is no strong guarantee that the shortest route between your application and our gateways will be taken even if they're situated in the same data center. To address this issue and achieve stable routes across the internet, Databento relies on blended bandwidth and private peering agreements which are summarized in the table below. If minimizing network variance is important for your use case, we recommend using any of these providers. | Service | Data center | Transit providers and private peers | | Historical (BO1) | CoreSite BO1 | Lumen, NTT, Telia, Cogent, Verizon, Google, Microsoft, Akamai | | Live (DC3) | CyrusOne Aurora I | Lumen, Zayo | | Live (NY4) | Equinix NY4 | Lumen, Hurricane Electric | If your security policies only allow traffic from designated IPs, ensure that you can route to the entire range of Databento's public IP addresses, from **209.127.152.0** to **209.127.159.255** (or **209.127.152.0/21** in CIDR notation). We are allocated exclusive ownership and usage of these public IPs by ARIN. You should also allow the following UDP and TCP port ranges used by our services through your access lists and firewalls. The destination ports are port numbers that Databento's servers listen on. | Protocol | Destination ports | | NTP | 123 | | HTTPS (historical) | 443 | | FTP | 21, 61000-61004 | | UDP | 13000-13050 | | TCP (live) | 13000-13050 | ### Connecting via dedicated connectivity See our [dedicated connectivity guide](https://databento.com/docs/architecture/dedicated-connectivity-guide.md) for more details. ### Connecting via peering Databento supports private peering over BGP at CyrusOne Aurora and the Equinix NY2/4/5/6 facilities for our live data, and at Markley One Summer Street for historical data. Our BGP ASN is **AS400138** and more info can be found on our [PeeringDB profile](https://www.peeringdb.com/net/32761). We do not support public peering at an IX at this time. [Contact support]($PLATFORM_HOSTNAME/contact) to set up private peering with Databento. --- Source: https://databento.com/docs/architecture/dedicated-connectivity-guide # Dedicated connectivity ![North America locations](https://databento.com/docs/locations.svg.md) Databento provides private, dedicated connectivity for customers who require predictable latency, high throughput, and higher uptime. The following table provides an overview of the four dedicated connectivity options supported by Databento and the recommended use case of each. | Connectivity option | Services | Best for | Ports | Latency (90th) | | [Interconnect with AWS, Google Cloud or Microsoft Azure](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#interconnect-with-aws-google-cloud-or-microsoft-azure) | Live, Historical | Mission-critical, cloud-based applications requiring high uptime or consuming entire feed(s) | 1G | 1.7+ ms | | [Interconnect at proximity hosting location](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#interconnect-at-proximity-hosting-location) | Live, Historical | Mission-critical, self-hosted applications requiring high uptime or consuming entire feed(s) | 1G | 0.5+ ms | | [Cross-connect with any colocation or managed services provider (MSP)](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#cross-connect-with-any-colocation-or-managed-services-provider-msp) | Live | Applications requiring lowest latency | 10G, 25G | 42.4 μs | | [Colocation with Databento](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#colocation-with-databento) | Live | Applications requiring lowest latency | 10G, 25G | 42.4 μs | [Contact support]($PLATFORM_HOSTNAME/contact) if you need dedicated connectivity or any customized connectivity solution. ### Interconnect with AWS, Google Cloud or Microsoft Azure This is the recommended setup to achieve 1.5 to 7 ms latency with Databento and the most cost-effective option of our four dedicated connectivity solutions. A 1 Gb connection is provided here and adequate for consuming entire feeds. This setup leverages our existing interconnections to AWS, Google Cloud, and Microsoft Azure on-ramps at Equinix CH1 and NY5. A dedicated layer 3 connection is installed and ensures that our data traffic goes through your cloud provider's backbone to reach your cloud services in any availability region or zone. Note that connecting to the cloud over a dedicated interconnect may only achieve similar *median* latency as connecting over public internet, as Databento has a highly-optimized IP network with diverse routes across tier 1 ISPs. However, this solution is expected to achieve better tail latencies. | Total NRC | Total MRC | Latency\* (90th) | | $300 | $750 | 1.7 ms, DC3 to Azure US N. Central. 6.8 ms, DC3 to GCP us-central1. | \* Varies with site. First byte of data in at our boundary switch to last byte read from your client socket. **Advantages** - Ease of setup, especially if your infrastructure is already AWS, Google Cloud, or Microsoft Azure. - It avoids contention on public internet routes, ensuring stable tail latencies and sufficient bandwidth between you and our live data gateways. - Simplicity--redundancy is built-in with BGP, which will fall back on our public internet routes if the dedicated connection is down, and once set up, you can connect to our gateways using our public DNS hostnames. - You can use public cloud services, which are more cost-effective compared to conventional hosting and colocation services; it's cheaper to spin up redundant servers on public cloud. - Your cloud servers may be situated in any availability region or zone to use this solution, giving you many flexible hosting options. - Most cost-effective of our four dedicated connectivity options. **Disadvantages** - Most other financial services providers, e.g. your broker, trading venues, extranet providers, etc. only provide connectivity at certain points-of-presence (PoPs) and don't support connections to public cloud. If you need to connect to such other external services besides Databento, you'll most likely use us to arrange additional cross-connects and backhaul. If these services require much more than 1 Gb of bandwidth, it then becomes more cost-effective to pursue our other dedicated connectivity options. - Higher latency than our other connectivity options. ### Interconnect at proximity hosting location This is similar to the setup for an [Interconnect with AWS, Google Cloud or Microsoft Azure](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#interconnect-with-aws-google-cloud-or-microsoft-azure), except that we'll connect to your own infrastructure at a proximity hosting site instead of a public cloud site. At the moment, the following proximity hosting locations are supported: - **Equinix CH1/4**, 350 E Cermak Ave. - **Equinix NY1**, 165 Halsey St. - **Equinix NY2**, 275 Hartz Way. - **Equinix NY9**, 111 8th Ave. - **Equinix LD4**, 2 Buckingham Ave, Slough, UK. - **Equinix FR2**, Kruppstrasse 121-127, Frankfurt, Germany. - **LSE**, 1 Earl St, London, UK. - **Interxion LON-1**, 11 Hanbury St, London, UK. | Total NRC | Total MRC | Latency\* (90th) | | $300 | $750+ | 0.59 ms, NY4 to NY1. 1.05 ms, DC3 to CH1. | \* Varies with site. First byte of data in at our boundary switch to last byte read from your client socket. ### Cross-connect with any colocation or managed services provider (MSP) This is the minimum recommended setup to achieve sub-50 μs latency with Databento. We're carrier-neutral and vendor-neutral--this means you may use any colocation provider or MSP that allows you to run a cross-connect to us at either CyrusOne Aurora I (DC3) or Equinix NY4/5. At the moment, our Raw API for live data only supports TCP transport and a single port is sufficient for reliable transmission with this setup. Latency varies with the site and data feed. The estimate below is based on our slowest path, which includes the following hops: - Arista 7050SX3-48YC, 800 ns - Arista 7260X3, 450 ns - Arista 7060SX2, 565 ns - Mellanox Spectrum SN2410, 680 ns | Total NRC | Total MRC | Latency\* (90th) | | $300 | $2,177.50 | 42.4 μs | \* First byte of data in at our boundary switch to last byte of data out onto your cross-connect. ### Colocation with Databento > **Info** > > This option is only available to customers with an existing annual, flat-rate > subscription for Databento Live. This is similar to [cross-connecting with any colocation or MSP](https://databento.com/docs/architecture/dedicated-connectivity-guide.md#cross-connect-with-any-colocation-or-managed-services-provider-msp), and achieves similar latency. The main difference is that you're colocating with us directly. | Total NRC | Total MRC | Latency\* (90th) | | $1,500 | $2,048.75 | 42.4 μs | \* First byte of data in at our boundary switch to last byte of data out on your data port. **Advantages** - Allows us to provide more end-to-end support and debug a wider range of issues. - Provides you with predictable latency that matches our test bench. - No cross-connect to Databento required. You're in the same rack as our live gateway and this skips hops on your colocation provider or MSP's edge border leaf or spine switches. **Disadvantages** - Not suitable if you anticipate a complex environment with more than 3 cross-connects, you have frequently-changing requirements, or you need a diverse selection of on-net counterparties and direct venue connectivity. > **See also** > > [Locations and network connectivity](https://databento.com/docs/architecture/locations-network-connectivity.md) and [Performance optimization](https://databento.com/docs/architecture/performance-optimization.md). --- Source: https://databento.com/docs/architecture/ntp-service # Databento NTP service Network Time Protocol (NTP) is an internet protocol commonly used to synchronize time between computer systems over a network. The Databento NTP service is a free public time service, based on NTP, that's available to everyone over internet and public cloud. ### Who should use this Databento's NTP service is ideal for financial users using proximity hosting services and public cloud providers like AWS, Azure, and GCP, and for users whose servers are located in the Chicago, New York, and New Jersey metro regions. While more accurate time synchronization practices exist for these users, the next marginal improvement for these users over our NTP service will require dedicated hosting or cross-connects, which are significantly more costly. All live data users connecting to Databento over internet are strongly recommended to use this time service to ensure that their local system clock is synchronized against Databento's internal time source and that latency estimates taken between local timestamps and Databento timestamps are more accurate. ### Getting started Anyone can synchronize their clocks against our time service by pointing their NTP client at **ntp.databento.com**. Alternatively, you can pick specific servers from this pool, as follows: | Server | Data center | Location | | `ntp-01.dc3.databento.com` | CyrusOne Aurora I | 2905 Diehl Rd, Aurora, IL 60502 | | `ntp-01.ny4.databento.com` | Equinix NY4 | 755 Secaucus Rd, Secaucus, NJ 07094 | | `ntp-02.ny4.databento.com` | Equinix NY4 | 755 Secaucus Rd, Secaucus, NJ 07094 | ### Recommended configuration Here are some recommendations for obtaining good accuracy on your system clock against our NTP service: - Use a high quality NTP client implementation, such as [chrony](https://chrony-project.org/). - In most cases, you should only step the clock in the first few updates, before your application starts. You can configure this with the `makestep` directive in your chrony configuration: ``` makestep 1 3 ``` - Ensure your system clock is only adjusted by **slewing** while your application is running. This means that your clock is adjusted by slowing down or speeding up instead of stepping, which ensures monotonic timestamps. By default, chrony adjusts the clock by slewing. - However, if your system does not have a real-time clock or is a virtual machine which can be suspended and resumed with an incorrect time, it may be necessary to allow stepping on every clock update, as follows: ``` makestep 1 -1 ``` In summary, you can add the following lines to your chrony configuration file, found at `/etc/chrony.conf` or `/etc/chrony/chrony.conf`: ``` pool ntp.databento.com iburst minpoll 4 maxpoll 4 # Step the system clock instead of slewing it if the adjustment is larger than # one second, but only in the first three clock updates makestep 1 3 ``` You can check that chrony is now using your new NTP sources with the `chronyc sources` command. ### Architecture - How it works Since the margin of error in time synchronization is largely a function of distance and hops to the time source, Databento's NTP service leverages our network of colocated servers at various financial data centers to help users obtain more accurate time. The Databento NTP service is composed of a pool of Stratum 1 NTP servers at the CyrusOne Aurora I and Equinix NY4 data centers. In contrast, common time providers usually situate their time servers in general-purpose data centers that are further from popular colocation and proximity hosting sites used by financial firms. Another important reason to use Databento's time service is the nature of its clock source. The time servers for our service are synchronized against the same PTP time source used for supplemental receive and outbound timestamps in our data, thus our time servers have a margin of error under 200 nanoseconds: ``` MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== #* PTP 0 0 377 1 +9ns[ +11ns] +/- 159ns ``` In turn, our PTP source is based on a pair of grandmaster clocks with under 25 nanosecond root-mean-squared error against UTC. Other public NTP services do not provide any guarantees about the quality of their clock sources--even if you were to obtain a time sync to their NTP servers with sub-millisecond tolerance, you may still be off from UTC by tens of milliseconds. --- Source: https://databento.com/docs/architecture/performance-optimization # Performance optimization ### Locations **What's DC3?** DC3 refers to the legacy name of the CME data center and the primary colocation site for its matching engine, Aurora DC3, before it was acquired by CyrusOne and renamed Aurora I. Since we have some data predating this acquisition, we still use this site name for naming consistency. **Where are DC3, NY4, and NY5 located?** DC3 is located at 2905 Diehl Road, Aurora, IL 60502. NY4 and NY5 are located at 755 and 800 Secaucus Rd, Secaucus NJ 07094 respectively. ### Setup **Do I need to install or license any special software to make use of dedicated connectivity?** No, you do not require any special software. Databento's Raw API and official Python, C++ and Rust client libraries support dedicated connectivity. Our live data offering is a managed service, meaning you don't have to run our software as a process on your machine. **Do I need any network configuration to make use of dedicated connectivity?** We'll assist you in configuring network routes to our gateways depending on the option that you pick. For cross-connects and colocation, you'll pass in a private IP to take a dedicated route. For dedicated interconnects, you'll rely on BGP to select the shortest path. ### Performance **Does latency matter for my use case? If so, is Databento fast enough?** Most electronic trading participants do not see an appreciable benefit from going below 50 us or even 1 ms, so our dedicated connectivity options and normalized feed performance should be more than adequate for most of our users. If you need even lower latency, there are specialized solutions tailored to your use case. However, to our knowledge, the next fastest solution will cost at least thrice the cost. This is intentional: Databento is designed to be at the pareto-frontier between latency and cost, i.e. each of our dedicated connectivity options is the fastest commercially-available solution at its price point. Being on the sweet spot between latency and pricing puts us in a position to solve many issues. For instance, Databento is great for initial exploration in cases where you have a catch-22 of being unable to quantify if you need lower latency until you start trading. Also, through us, users with cloud-based infrastructure can achieve comparable performance to colocated, bare metal infrastructure at a fraction of the cost. **Why does Databento Live use TCP for transport? Don't trading venues use UDP for raw feeds? Isn't TCP slower than UDP?** At this time, we're only supporting TCP so that all of our customers get reliable transmission and congestion control without additional complexity in the client implementation. On a side note, TCP is potentially faster than UDP for customers who're connecting to us over long-haul, and we've made considerable efforts to optimize our Raw API and measure its performance. Raw financial feeds often use UDP multicast because of its simplicity and potential for latency optimization on networks with excess bandwidth, as in the case when you have a direct connection to the trading venue's extranet. However, TCP is more efficient in environments with limited bandwidth, network congestion, and a large proportion of small messages--because the TCP protocol allows for buffering of the data messages to fill a full network segment. In fact, some popular ECNs and OTC trading venues have to support customers connecting over the internet and have TCP-based feeds as their primary, raw data feed. **Do you lease out server hardware to your customers?** We only lease out hardware to customers who colocate with us directly. ### Hardware **Do you offer VMs or virtual private servers (VPSes)?** We don't currently offer VMs or VPSes. **Do you have a recommended server spec?** Databento has very low minimum system requirements, so the hardware specification will depend more on your application and we do not have any hard recommendations. Most of our customers have servers with a dedicated NIC from AMD, Mellanox, or Intel. ### Fees and terms **Do you charge all of the fees listed, including for services not provided directly by Databento?** We only charge the fees with Databento listed in the "Paid to" column. Other listed fees are estimates and payable to the provider. **What's the term length for any of these dedicated connectivity options?** We require a 12-month contract. --- Source: https://databento.com/docs/conventions # Standards and conventions Frequently-used fields, normalization conventions, symbology, encodings, and more. | Title | Subtitle | Image | Supported languages | | [Common fields, types, and enums](https://databento.com/docs/conventions/common-fields-enums-types.md) | Date and timestamp formats, price formats, record types, flags, and more. | ![Script image](https://databento.com/docs/fields.svg.md) | Terminology, List of fields | | [Normalization](https://databento.com/docs/conventions/normalization.md) | The reasons behind the design of Databento's normalization format. | ![Normalization image of databases](https://databento.com/docs/normalization.svg.md) | Terminology | | [Symbology](https://databento.com/docs/conventions/symbology.md) | A guide to our raw symbol, instrument ID, parent, and continuous symbology types. | ![Symbology image](https://databento.com/docs/conventions-symbology.svg.md) | Symbology | | [Databento Binary Encoding](https://databento.com/docs/conventions/databento-binary-encoding.md) | Fast, zero-copy message encoding and storage format for market data. | ![Encoding terms image](https://databento.com/docs/dbn.svg.md) | Encoding | | [Zstandard (zstd)](https://databento.com/docs/conventions/working-with-zstandard.md) | This lossless compression algorithm is recommended for historical requests. | ![Image placeholder](https://databento.com/docs/zstd.svg.md) | Compression, Historical only | | [MBO snapshot](https://databento.com/docs/conventions/mbo-snapshot.md) | Reconstruct the order book with MBO snapshot. | ![MBO snapshot image](https://databento.com/docs/live-snapshot.svg.md) | MBO snapshot, Live, Historical | | [Reference data enums](https://databento.com/docs/conventions/reference-data-enums.md) | Specifies descriptions for enumeration field values in reference data responses. | ![Enums image](https://databento.com/docs/reference-data-enums.svg.md) | Terminology | --- Source: https://databento.com/docs/conventions/common-fields-enums-types # Common fields, enums and types ### Publishers, datasets, and venues We use a few different terms to describe our data: - A **dataset** is a source of data. - A **venue** is an exchange, OTC market (e.g., ATS, ECN) or reporting entity. - A **publisher** is a specific venue from a specific dataset. > **See also** > > Read our [Venues and publishers](https://databento.com/docs/faqs/venues-and-publishers.md) section for a more detailed explanation. ### Publisher identifiers All of our schemas include a `publisher_id` field, which is a unique numeric ID assigned by Databento to each publisher. A full list of publishers can be found using the [metadata.list\_publishers](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md) endpoint. ### Venue and dataset identifiers Each publisher is also assigned a string identifier (e.g., `OPRA.PILLAR.XCBO`), composed of two parts: - **Dataset ID** (e.g., `OPRA.PILLAR`). This is used as the `dataset` argument in any API or client method. Dataset IDs can be found on the Databento portal on each dataset's [details page](https://databento.com/docs/portal/data-catalog.md#dataset-details-page) or via the [metadata.list\_datasets](https://databento.com/docs/api-historical/metadata/metadata-list-datasets.md) endpoint. - **Venue** (e.g., `XCBO`). For most markets, this is its [ISO 10383 MIC code](https://www.iso20022.org/market-identifier-codes), which is guaranteed to be four characters long. For entities without a MIC code, this string is arbitrarily assigned by Databento and will also be four characters long. ### Instrument identifiers All of our schemas contain an `instrument_id` field which is a numeric ID that maps to a given instrument. In most cases, this numeric ID is assigned by the publisher. For publishers that do not assign this value, we create a synthetic mapping for it. `instrument_id` is only guaranteed to be unique within a given day. Some publishers provide a different instrument ID on different days for the same underlying instrument. Other publishers may use the same instrument ID for different underlying instruments at different points in time. Depending on the use case, it may be easier to work with other symbology types such as `raw_symbol`. Our [symbology](https://databento.com/docs/conventions/symbology.md) documentation outlines the various symbology types we support. ### Timestamps All the timestamps in our data are expressed as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time), i.e. UNIX timestamps. All timestamp fields are prefixed with `ts_`. Some of our timestamps are encoded as the difference, i.e. delta, relative to another timestamp. Such timestamp fields are suffixed with `_delta`. We provide four types of timestamps, through the following fields: - Event timestamps, `ts_event` - Publisher sending timestamp, `ts_in_delta` - Databento receive timestamp, `ts_recv` - Databento sending timestamp (live only), `ts_out` `UNDEF_TIMESTAMP` (`UINT64_MAX`, 18446744073709551615) is used to denote a null or undefined timestamp. The event and publisher sending timestamps are provided by the publisher (or market), and we provide their original values without any adjustment. ### ts\_event Most users will only need the event timestamp, i.e. `ts_event`. For market data, this represents the time that the event is received by the matching engine (tag 60 in FIX encoding). The exact location where this timestamp is taken varies with matching engine architecture of each market. Some markets will handle different subsets of instruments on independent order gateways, while other markets will load balance the same subset of instruments across independent order gateways. Some markets take the event timestamp at the time it is received on the independent order gateways, while others may take this timestamp at the time it reaches a FIFO matching queue. In the former case, the clocks on independent order gateways are often not properly synchronized to the same clock source. Since we do not adjust the publisher's timestamps, any non-monotonicity in the original data will remain. ### ts\_in\_delta The publisher sending timestamp represents the time when the data message associated with an event is sent (tag 52 in FIX encoding). We encode this information in `ts_in_delta`, which expresses the number of nanoseconds between the Databento receive timestamp (`ts_recv`) and the publisher sending timestamp. To get the sending timestamp itself, simply subtract `ts_in_delta` from `ts_recv`. Since the publisher and Databento are not necessarily synchronized to the same clock source, `ts_in_delta` may be negative. `ts_in_delta` is a 32-bit signed integer. The minimum will clamp to `INT32_MIN` and the maximum will clamp to `INT32_MAX`, even if the true value exceeds these limits. Some markets do not provide both match event timestamps and sending timestamps. Often, they will provide only one of the two. In such cases, we take it that the event timestamp and sending timestamp assume the same value. As such, `ts_event` will be the provided timestamp and `ts_in_delta` will be equal to the difference between `ts_recv` and `ts_event`. ### ts\_recv Unless otherwise specified, Databento receive timestamps, i.e. `ts_recv`, are synchronized against UTC with sub-microsecond accuracy. Moreover, these receive timestamps are always guaranteed to be monotonic for any given symbol. These receive timestamps rely on hardware timestamping on the network adapter and are synchronized against a GPS clock source using PTP. The clock is corrected by slewing the time, which prevents discrete jumps backwards in time. In other words, our local receive timestamps are guaranteed to be monotonic for any given symbol. `ts_recv` is also adjusted for leap seconds. The local receive timestamp is not immediately adjusted intraday when a leap second is introduced. Instead, the leap second update is applied at the end of the market session. ### ts\_out For live data, we optionally include a timestamp of our data before it leaves our data gateways. This information is encoded as `ts_out`. Both `ts_out` and `ts_recv` are synced to the same GPS clock source. Subtracting `ts_recv` gives the number of nanoseconds spent in our system. Some records may share a `ts_out` value captured before those records are encoded. ### Index timestamp All schemas have a primary timestamp that should be used for sorting records as well as indexing into any symbology data structure. This index timestamp will be `ts_recv` if it exists in the schema, otherwise it will be `ts_event`. When requesting historical data, the data will be filtered based on the index timestamp. When requesting data in CSV and JSON encodings, the first field will be set to this index timestamp. Additionally, for schemas that contain `ts_recv`, the second field will be set to `ts_event`. ### Encodings We support [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md), [CSV](https://datatracker.ietf.org/doc/html/rfc4180), and [JSON](https://datatracker.ietf.org/doc/html/rfc8259) encodings for our data. DBN is an extremely fast message encoding and storage format for normalized market data. All official Databento client libraries use DBN under the hood, both as a data interchange format and for in-memory representation of data. DBN is also the default encoding for all Databento APIs, including live data streaming, historical data streaming, and batch flat files. Our batch download system also supports CSV and JSON encodings. ### Time zone By default, all of our data is set in UTC time zone. Likewise, our site displays all dates and times in UTC by default. ### Dates and times We use the [ISO 8601 date and time format](https://www.iso.org/iso-8601-date-and-time-format.html) to express dates and times used as parameters to our APIs. All dates and times used as parameters are in UTC by default. The "reduced precision" concept in the ISO 8601 standard allows for dates and times to be represented with varying levels of detail. Any number of values may be dropped from any of the date and time representations, but in the order from the least to the most significant. For example, "2024-05" corresponds to "2024-05-01T00:00:00". Any parameter that takes an ISO 8601 timestamp can instead be given a timestamp in nanoseconds since the UNIX epoch, as described in the above section. All of our timestamp parameters are start-inclusive and end-exclusive. ### Forward filling end parameters For our APIs that take an optional `end` parameter as an ISO 8601 string, we will implement the following behavior when the end parameter is not provided: We will *forward fill* any date or time components of the associated `start` parameter that are omitted. This "rounds up" the `start` timestamp for use as the `end` timestamp, and is done for more concise usage. Examples of this behavior are shown below. > **Info** > > We will only *forward fill* timestamps with less than one-second resolution. | Start timestamp | Effective start timestamp | Forward filled end timestamp | | "2024" | "2024-01-01T00:00:00" | "2025-01-01T00:00:00" | | "2024-03" | "2024-03-01T00:00:00" | "2024-04-01T00:00:00" | | "2024-03-10" | "2024-03-10T00:00:00" | "2024-03-11T00:00:00" | | "2024-03-10T01" | "2024-03-10T01:00:00" | "2024-03-10T02:00:00" | | "2024-03-10T00:01" | "2024-03-10T00:01:00" | "2024-03-10T00:02:00" | For example, a query for the entire month of March 2024 can be specified with `start="2024-03"` without an `end`. ### rtype An rtype or record type is an unsigned 8-bit discriminant in the header of every [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md) record that indicates the type of record structure. Each [schema](https://databento.com/docs/schemas.md) has one rtype and by extension one record structure associated with it. > **Info** > > Some rtypes are not associated with a schema and are only present in live data. | Name | Hex | Decimal | Description | | MBP-0 | `0x00` | 0 | A market-by-price record with a book depth of 0. Used for the trades schema. | | MBP-1 | `0x01` | 1 | A market-by-price record with a book depth of 1. Used for the TBBO and MBP-1 schemas. | | MBP-10 | `0x0A` | 10 | A market-by-price record with a book depth of 10. | | Status | `0x12` | 18 | An exchange status record. | | Definition | `0x13` | 19 | An instrument definition record. | | Imbalance | `0x14` | 20 | An order imbalance record. | | Error | `0x15` | 21 | An [error record](https://databento.com/docs/api-live/basics/errors.md) from the live gateway. | | Symbol mapping | `0x16` | 22 | A [symbol mapping record](https://databento.com/docs/conventions/symbology.md#symbolmappingmsg) from the live gateway. | | System | `0x17` | 23 | A [non-error record](https://databento.com/docs/api-live/basics/system-messages.md) from the live gateway. | | Statistics | `0x18` | 24 | A statistics record from the publisher. | | OHLCV-1s | `0x20` | 32 | An OHLCV record at a 1-second cadence. | | OHLCV-1m | `0x21` | 33 | An OHLCV record at a 1-minute cadence. | | OHLCV-1h | `0x22` | 34 | An OHLCV record at an hourly cadence. | | OHLCV-1d | `0x23` | 35 | An OHLCV record at a daily cadence. | | MBO | `0xA0` | 160 | A market-by-order record. | | CMBP-1 | `0xB1` | 177 | A consolidated market-by-price record with a book depth of 1. | | CBBO-1s | `0xC0` | 192 | A consolidated market-by-price record with a book depth of 1 at a 1-second cadence. | | CBBO-1m | `0xC1` | 193 | A consolidated market-by-price record with a book depth of 1 at a 1-minute cadence. | | TCBBO | `0xC2` | 194 | A consolidated market-by-price record with a book depth of 1 with only trades. | | BBO-1s | `0xC3` | 195 | A market-by-price record with a book depth of 1 at a 1-second cadence. | | BBO-1m | `0xC4` | 196 | A market-by-price record with a book depth of 1 at a 1-minute cadence. | ### Prices Prices are expressed as signed integers in fixed-precision format, whereby every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. For example, a price of 5411750000000 corresponds to 5411.75 (decimal format). When requesting data via batch download in CSV and JSON encodings, you can optionally choose for prices to be returned in decimal format. If you are requesting data using the online portal, you can select **Decimal prices** in the [Advanced customization](https://databento.com/docs/portal/batch-download.md#advanced-customization) section. Otherwise, you can specify the `pretty_px` parameter in [batch.submit\_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) using the client libraries. Additionally, our client libraries support functionality to view prices in decimal format. In certain scenarios--such as calendar spreads in futures--prices can be negative. `UNDEF_PRICE` is used to denote a null or undefined price. It will be equal to 9223372036854775807 (`INT64_MAX`) when using the fixed-precision integer format. When expressed in decimal format, it will be equal to `null` in JSON, or `""` (an empty string) in CSV. ### Side The `side` field contains information about the side of an order event. It's meaning will vary depending on the `action` field. - When `action` is **T**rade: - `A` - The trade aggressor was a seller - `B` - The trade aggressor was a buyer - `N` - No side specified - When `action` is **F**ill: - `A` - A resting sell order was filled - `B` - A resting buy order was filled - `N` - No side specified - When `action` is **A**dd, **M**odify, and **C**ancel: - `A` - A resting sell order updated the book - `B` - A resting buy order updated the book - `N` - No side specified - When `action` is clea**R** book, `side` will always be `N` `side` can be `N` in the following cases: - The source does not disseminate a side for trades. - Trades happening during opening and closing auctions - Trades against non-displayed orders - Trades involving implied orders - Off-exchange trades The [Venues and datasets](https://databento.com/docs/datasets.md) section provides more information regarding the specific cases for each dataset where no side will be specified. ### Action The `action` field contains information about the type of order event contained in the message. | Name | Value | Action | | Add | `A` | Insert a new order into the book. | | Modify | `M` | Change an order's price and/or size. | | Cancel | `C` | Fully or partially cancel an order from the book. | | Clear | `R` | Remove all resting orders for the instrument. | | Trade | `T` | An aggressing order traded. Does not affect the book. | | Fill | `F` | A resting order was filled. Does not affect the book. | | None | `N` | No action: does not affect the book, but may carry `flags` or other information. | ### Flags The `flags` field is a [bit field](https://en.wikipedia.org/wiki/Bit_field) that contains information about the message. The meaning of each bit is as follows: | Flag | Value | Decimal | Description | | `F_LAST` | `1 << 7` | 128 | Marks the last record in a single event for a given `instrument_id`. | | `F_TOB` | `1 << 6` | 64 | Top-of-book message, not an individual order. | | `F_SNAPSHOT` | `1 << 5` | 32 | Message sourced from a replay, such as a snapshot server. | | `F_MBP` | `1 << 4` | 16 | Aggregated price level message, not an individual order. | | `F_BAD_TS_RECV` | `1 << 3` | 8 | The `ts_recv` value is inaccurate due to clock issues or packet reordering. | | `F_MAYBE_BAD_BOOK` | `1 << 2` | 4 | An unrecoverable gap was detected in the channel. | | `F_PUBLISHER_SPECIFIC` | `1 << 1` | 2 | Semantics depend on the `publisher_id`. Refer to the relevant [dataset supplement](https://databento.com/docs/datasets.md) for more details. | | | `1 << 0` | 1 | Reserved for internal use can safely be ignored. May be set or unset. | Multiple flags can be set on a single message, in which case the field value is the sum of the individual flag values. For example, a `flags` value of 160 means `F_LAST` and `F_SNAPSHOT` are set: 160 = 128 + 32. Test for an individual flag with a bitwise AND: `flags & 128` is non-zero when `F_LAST` is set, `flags & 64` is zero when `F_TOB` is not. > **Warning** > > Comparing the field with a single flag's value, such as `flags == 128` to check for `F_LAST`, will miss any record that has other flags set too. ### Top-of-book datasets Some datasets are built on feeds from vendors that only provide top-of-book information (best bid and offer). Top-of-book messages are normalized into a pair of MBO records with the **A**dd action and the `F_TOB` flag (`0x40`, 64) set. Typically for these datasets, there is no information available about the passive side of trades, so there are no **F**ill records and the `side` of the **T**rade record is always set to **N**one. The removal of a price level is normalized as an **A**dd action with a size of 0 and price of `UNDEF_PRICE` (`INT64_MAX`, 9223372036854775807) or NaN in Python. This indicates there's currently no quotes for that side. Other schemas, such as MBP-1, Trades, and OHLCV remain the same for top-of-book datasets. ### Market-by-price datasets Some datasets are built on feeds from vendors that only provide market-by-price information (with limited depth). Messages adding/modifying/deleting a price level are normalized into an MBO record with the **A**dd/**M**odify/**C**ancel action, with the `size` field containing the full quantity at that level and the `F_MBP` flag (`0x10`, 16) set. A price level can be identified from the combination of the `side` and the `price`. The `order_id` field should be ignored for those messages. If the upstream feed has a maximum depth, an additional record with **C**ancel action will be sent whenever a price level falls outside the maximum depth - even if there are still orders at that price level. Typically for these datasets, there is no information available about the passive side of trades, so there are no **F**ill records. MBP-10 will only include depth up to the depth provided by the publisher. The remaining levels will always be empty. Other schemas, such as Trades and OHLCV are otherwise unaffected. --- Source: https://databento.com/docs/conventions/normalization # Normalization **Normalization** refers to the process of exporting data in their various source formats to a single, unified format. Such a unified format is often called a **normalization format** (or **normalized schema**). The primary reason for normalizing market data is to abstract away differences between source formats, making the data easier to work with. The normalization process is one of the most likely places where inaccuracies or data errors are introduced. This article describes these issues, the trade-offs for addressing them, and the reasons behind the design of Databento's normalization schema. ### Examples of normalized data - For example, Nasdaq's proprietary TotalView data feed has a protocol with its own message format and provides market-by-order data, while IEX's proprietary TOPS data feed has a completely different protocol with another message format and provides top-of-book data. These are examples of **raw data** formats. - When you consume market data from a data redistributor's feed, the redistributor will have its own protocol and message format, distinct from the venues'. This is an example of **normalized data**. - The most sophisticated trading firms will generally collect data directly from their sources and normalize them to a proprietary format. ### Common issues found in normalized market data There are many ways in which normalization can introduce data errors, lossiness or performance issues. | Issue | Definition | Examples | | **Incompatible schema** | The source schema and normalized schema are mismatched. | A direct market feed with an order-based schema is normalized to a vendor's schema that only provides aggregated market depth. | | **Truncated timestamps** | A direct market feed which originally includes nanosecond-resolution timestamps is normalized to a schema that truncates the timestamps to a lower resolution. | Some vendors have a legacy data schema designed for older FIX dialects, forcing them to truncate nanosecond-resolution timestamps found in modern markets to millisecond resolution. | | **Discarded timestamps** | A direct market feed which originally includes more than one timestamp field is normalized to a schema that discards that timestamp. This introduces imprecision when the normalized data is used for strategy backtesting. | A proprietary exchange feed may include both match (Tag 60) and sending (Tag 52) timestamps but a vendor's schema may preserve only one of the two. | | **Discarded or remapped sequence numbers** | The normalized schema either discards the original message sequence numbers or remaps them to a vendor's own message sequence numbers. | This creates problems if you need to resolve post-trade issues with the market or your broker, as it makes it harder to identify the exact event. | | **Loss of price precision** | The normalized schema represents prices in a type that loses precision. | Many vendors use floating point representation for prices, losing precision past 6 decimal places. This can create issues for trading Japanese yen spot rates, fixed income instruments and cryptocurrencies. | | **Loss of null semantics** | The normalized schema represents null values in a way that changes the meaning. | Some data feeds will represent null prices with zeros or a negative value like -1. This can introduce errors downstream if the price is interpreted to be non-null. This is also problem if your application needs to handle both asset classes that can have negative prices (such as futures and spreads) and asset classes which cannot. | | **Loss of packet structure** | The normalized schema does not preserve packet-level structure. | Many markets publish multiple events within a packet. Without packet-level structure, it may create the appearance of artificial trading opportunities between any two events within a packet. | | **Lossy or irreversible symbology mappings** | The normalized schema adopts a proprietary symbology that is different from the original source's symbology. Sometimes, such proprietary symbology cannot be mapped back to the original. | Some vendors adopt a symbology system that only includes lead months of futures contracts, causing the far month contracts to be discarded. | | **Lossy abstraction** | The normalized schema does not adequately standardize information across multiple datasets, resulting in the end user needing to understand the specifications of the various source schemas anyway in order to determine the lost information. | This often happens when normalizing less commonly used features such as matching engine statuses or instrument definitions. This puts significant burden on the user to study the specifications of various data feeds to understand the lost information. | | **Statelessness** | The normalized schema provides incremental changes but does not provide snapshots or replay of order book state. | This presents an issue when using the normalized data in real-time, as the user loses information in the event of a disconnection or late join. | | **Coalescing** | The normalized schema aggregates the information at a lower granularity. | A vendor may coalesce a feed of tick data with second bar aggregates or subsample a source feed. | | **Conflation** | A normalized feed batches multiple updates into one at some lower frequency, to alleviate bandwidth limitations. Often present along with coalescing. | This is a common practice for retail brokerages, whose data feeds are designed more for display use and consumption over sparse WAN links. | | **Dropped packets** | A normalized data feed deliberately discards data when the network or system is unable to keep up. | This is often present when the source feed or upstream parts of the vendor's infrastructure uses UDP for transmission. | | **Buffering** | The data server sends stale data either because there is insufficient network bandwidth or the client is reading too slowly. The client misinterprets the stale data, either obscuring this effect or injecting incorrect timestamps. | This often manifests when the data feed uses TCP for transmission - which is a common practice when disseminating data over WAN links. | | **Ex post cleaning** | A data source is cleaned, during the normalization process, using future information. This enhances the historical data with artificial information that may not have been actionable in real-time. | The data may be reordered; trades that were canceled after the end of market session may be removed, or prices may be adjusted with information from a future rollover or dividend event. | | **Schema bloat** | A normalized schema represents some data fields with types that take up unnecessary space or make the data more difficult to compress, which increases storage costs and reduces application performance. | Common cases of this include representing timestamps as ISO 8601 strings or prices as strings, especially on vendor feeds that use JSON encoding. | Our normalization schema is designed to mitigate most of these issues. ### Why use normalized data? Though it may seem counterintuitive, some degree of lossiness introduced during normalization can be preferable. A normalized schema that has too many data fields, as a result of trying to preserve information from too many different source schemas, is hard to use. Here are some ways in which lossiness can be useful: - Discarding unnecessary data fields can reduce storage and bandwidth requirements, and improve application performance. For example, many strategies execute at time scales where extra timestamps are unnecessary. - Most status or reference data events are irrelevant for any given business use case. For example, many users only trade during the regular market session and their applications do not need to be aware of special matching conditions that are more typically found outside of regular hours or during pre-market. - Floating point prices can be easier to use and the modeling error introduced by them could be negligible compared to other, more likely sources of error for the given use case. - Order book snapshots can be unnecessary on liquid products whose orders turnover very quickly, as pre-existing orders in the snapshot will eventually be filled or canceled - a process which is commonly referred to as *natural refresh*. It should be noted that normalized data is not necessarily going to be smaller or have a simpler specification than the source data. If your use case only requires a single dataset, there may be complexity using normalized data whose schema was designed to accommodate differences between multiple datasets. We normalize to our proprietary [Databento Binary Encoding (DBN)](https://databento.com/docs/conventions/databento-binary-encoding.md). --- Source: https://databento.com/docs/conventions/symbology # Symbology Financial datasets usually contain symbols or [product](https://databento.com/docs/faqs/instruments-and-products.md) identifiers. The mapping of symbols to their corresponding product names can be extracted from our [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) as well as the metadata packaged with our data. Databento supports four symbology types, also referred to as **stypes**. They are: [raw_symbol](https://databento.com/docs/conventions/symbology.md#raw-symbol), [instrument_id](https://databento.com/docs/conventions/symbology.md#instrument-id), [parent](https://databento.com/docs/conventions/symbology.md#parent), and [continuous](https://databento.com/docs/conventions/symbology.md#continuous). We include methods for mapping between symbology types and resolving symbols under the [symbology](https://databento.com/docs/api-historical/symbology.md) family of methods. We do not retroactively reassign symbols in our historical data. Symbols found in our historical data are exactly as they appeared in the live data at the original event time. For example, if a stock symbol was changed due to a corporate action, we preserve the original symbol for data before the event and the new symbol for data after the event. This approach guarantees that the historical data looks identical to the live data at the original time, and encourages our users to write their integration in a manner that handles historical and live data in the same way. A symbol can be reused and point to two different instruments on two different dates. ### Supported symbology combinations When requesting data, such as with the [timeseries.get_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md) or [batch.submit_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md) endpoints, an input (`stype_in`) and output (`stype_out`) symbology type are specified. Not all symbology types are supported for output and some symbology types are not available in certain datasets. | `stype_in` ↓ / `stype_out` → | `instrument_id` | `raw_symbol` | `parent` | `continuous` | | `instrument_id` | ✓ | ✓ | | | | `raw_symbol` | ✓ | | | | | `parent` | ✓ | | | | | `continuous` | ✓ | | | | All datasets support bidirectional conversion between `raw_symbol` and `instrument_id`. The table below outlines the datasets that support [parent symbology](https://databento.com/docs/conventions/symbology.md#parent) and [continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous) (futures contracts only). | `stype_in` | `parent` | `continuous` | | [GLBX.MDP3](https://databento.com/docs/datasets/glbx-mdp3.md) | ✓ | ✓ | | [IFEU.IMPACT](https://databento.com/docs/datasets/ifeu-impact.md) | ✓ | ✓ | | [IFLL.IMPACT](https://databento.com/docs/datasets/ifll-impact.md) | ✓ | ✓ | | [IFUS.IMPACT](https://databento.com/docs/datasets/ifus-impact.md) | ✓ | ✓ | | [NDEX.IMPACT](https://databento.com/docs/datasets/ndex-impact.md) | ✓ | ✓ | | [OPRA.PILLAR](https://databento.com/docs/datasets/opra-pillar.md) | ✓ | | | [XCBF.PITCH](https://databento.com/docs/datasets/xcbf-pitch.md) | ✓ | ✓ | | [XEEE.EOBI](https://databento.com/docs/datasets/xeee-eobi.md) | ✓ | ✓ | | [XEUR.EOBI](https://databento.com/docs/datasets/xeur-eobi.md) | ✓ | ✓ | ### Raw symbol Raw symbols are the original string symbols used by the publisher in the source data. This can be useful for environments with direct market connectivity. Examples of raw symbols include `AAPL`, `ESH3`, etc. This symbology is used by setting the `stype_in=raw_symbol` parameter in the API. #### CMS vs. Nasdaq symbology in US equities In US equities, the same security may carry slightly different ticker symbols across venues. The two competing conventions are **Nasdaq** and **CMS** (CTA/UTP Consolidated Market System). For multi-class shares, units, warrants, preferred shares, and similar securities, the conventions differ in the suffix delimiter and codes used: | Security | Nasdaq | CMS | | Berkshire Hathaway Class B | `BRK.B` | `BRK B` | | AAC Holdings warrant | `AAC+` | `AAC WS` | | Heico Class A | `HEI.A` | `HEI A` | | Moog Class A | `MOG.A` | `MOG A` | The reference for the suffix mappings is the [Nasdaq Symbol Convention](https://www.nasdaqtrader.com/Trader.aspx?id=CQSSymbolConvention) document. `raw_symbol` returns the symbol as published by the source, so the convention varies with the dataset. The table below shows the convention for each US equities dataset: | Dataset | Native convention | | [ARCX.PILLAR](https://databento.com/docs/datasets/arcx-pillar.md) | CMS | | [BATS.PITCH](https://databento.com/docs/datasets/bats-pitch.md) | Nasdaq | | [BATY.PITCH](https://databento.com/docs/datasets/baty-pitch.md) | Nasdaq | | [EDGA.PITCH](https://databento.com/docs/datasets/edga-pitch.md) | Nasdaq | | [EDGX.PITCH](https://databento.com/docs/datasets/edgx-pitch.md) | Nasdaq | | [EPRL.DOM](https://databento.com/docs/datasets/eprl-dom.md) | Nasdaq | | [EQUS.MINI](https://databento.com/docs/datasets/equs-mini.md) | Nasdaq | | [EQUS.SUMMARY](https://databento.com/docs/datasets/equs-summary.md) | Nasdaq | | [IEXG.TOPS](https://databento.com/docs/datasets/iexg-tops.md) | Nasdaq | | [MEMX.MEMOIR](https://databento.com/docs/datasets/memx-memoir.md) | CMS | | [OCEA.MEMOIR](https://databento.com/docs/datasets/ocea-memoir.md) | CMS | | [XASE.PILLAR](https://databento.com/docs/datasets/xase-pillar.md) | CMS | | [XBOS.ITCH](https://databento.com/docs/datasets/xbos-itch.md) | Nasdaq | | [XCHI.PILLAR](https://databento.com/docs/datasets/xchi-pillar.md) | CMS | | [XCIS.TRADESBBO](https://databento.com/docs/datasets/xcis-tradesbbo.md) | CMS | | [XNAS.BASIC](https://databento.com/docs/datasets/xnas-basic.md) | Nasdaq | | [XNAS.ITCH](https://databento.com/docs/datasets/xnas-itch.md) | Nasdaq | | [XNYS.PILLAR](https://databento.com/docs/datasets/xnys-pillar.md) | CMS | | [XPSX.ITCH](https://databento.com/docs/datasets/xpsx-itch.md) | Nasdaq | ### Instrument ID Instrument IDs are the unique numeric ID assigned to each instrument by the publisher. Most venues use such numeric IDs under the hood. Numeric IDs have the benefit of taking less space than most string symbols. However, numeric IDs can be difficult to work with, especially as some publishers remap them daily. This symbology is used by setting the `stype_in=instrument_id` parameter in the API. ### Parent Parent symbology is a smart symbology feature that allows you to easily refer to groups of related symbols using a single root symbol. The root symbols are sourced from the `asset` field of the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). All futures for a root symbol can be referenced using the parent symbol `[ROOT].FUT`, for options: `[ROOT].OPT`. For example, `ES.FUT` refers to all E-mini S&P 500 futures and futures spreads and `ES.OPT` refers to all quarterly E-mini S&P 500 options and option spreads. The type of instrument will be specified in the `instrument_class` field. When requesting data using futures parent symbology, this field will indicate whether the instrument is a future or futures spread. When requesting data using options parent symbology, it will indicate whether the instrument is a call or put. A full list of variants can be found in the [instrument class documentation](https://databento.com/docs/schemas/instrument-definitions.md#instrument-class). This symbology is used by setting the `stype_in=parent` parameter in the API. ### Continuous > **Info** > Our continuous contract symbology is a notation that maps to an actual, tradable instrument on any given date. > The continuous contract prices returned are the original, unadjusted prices. We don't create a synthetic time > series by back-adjusting the prices to remove jumps during rollovers. Continuous contract symbology is a smart symbology feature that allows a single symbol to refer to different instruments over time. For example, continuous contract symbology allows you to query a single symbol that changes or *rolls* forward before expiration. For futures outrights, we use the format `[ROOT].[ROLL_RULE].[RANK]` to refer to continuous contracts that change over time according to a roll rule and rank. Like with parent symbology, the root symbol corresponds with the `asset` field of the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). `RANK` is a zero-indexed integer, and `ROLL_RULE` is either `c`, `n`, or `v` from the table below. This symbology is used by setting the `stype_in=continuous` parameter in the API. It is not currently possible to select continuous contracts through our web portal. | Roll rule | Code | Overview | Example | | Calendar | `c` | Refers to the offset from the closest expiration or front month. | On September 28, 2022 `NG.c.0` referred to the October NG future (NGV2) and `NG.c.1` referred to the November future (NGX2). However, because the October contract expired at the end of trading on September 28 and the continuous smart symbol would be rolled forward, on September 29, 2022, `NG.c.0` then referred to the November future (NGX2) and `NG.c.1` referred to the December future (NGZ2). | | Open interest | `n` | Will rank the expirations by the open interest at the previous day's close. | `CL.n.1` refers to the CL future with the second-highest open interest. | | Volume | `v` | Will rank the expirations by the previous day's trading volume. | `ZN.v.0` refers to the ZN future with the most volume. | ### All symbols It is possible to request all symbols within a dataset without providing them explicitly. This is done by specifying `ALL_SYMBOLS` with `stype_in=raw_symbol` or `stype_in=parent` in the API. When requesting all symbols using [timeseries.get_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md) symbology data is not provided. This means that for the CSV and JSON encodings the parameter `map_symbols=True` is not allowed. For the DBN encoding, the metadata header will not contain symbology mappings. When requesting all symbols using [batch.submit_job](https://databento.com/docs/api-historical/batch/batch-submit-job.md), the [symbology.json](https://databento.com/docs/portal/download-center.md#support-files) support file will not contain symbology mappings. ### Symbology.resolve endpoint Databento offers symbology resolution for free in our [symbology.resolve endpoint](https://databento.com/docs/api-historical/symbology/symbology-resolve.md) and in our client libraries. This endpoint can be used to request mappings from one symbology type to another and contains all the data necessary to perform these conversions. | Field | Description | | `result` | The symbology mapping result. For each requested symbol, a list of symbology mappings is provided. | | `symbols` | The requested symbols. | | `stype_in` | The requested input symbology type. | | `stype_out` | The requested output symbology type. | | `start_date` | The requested symbology start date, as an ISO 8601 date string. | | `end_date` | The requested symbology end date, as an ISO 8601 date string. | | `partial` | The list of symbols, if any, that partially resolved inside the start date and end date interval. | | `not_found` | The list of symbols, if any, that failed to resolve inside the start date and end date interval. | | `message` | A short message indicating the overall symbology result. Can be one of: "OK", "Not found", or "Partially resolved". | | `status` | A numerical status field indicating the overall symbology result. Can be one of: 0 (OK), 1 (Partially resolved), or 2 (Not found). | ### Symbology support file For some batch downloads, symbology information for the job is contained in `*.symbology.json` support files. This file is automatically included when the batch job files do not contain symbology information, such as when requesting CSV or JSON encodings when symbol mapping is not requested. Below is a sample file. It's contents are directly obtained from the [symbology.resolve](https://databento.com/docs/api-historical/symbology/symbology-resolve.md) endpoint: ```json { "result": { "ES.c.0": [ { "d0": "2023-01-01", "d1": "2023-03-19", "s": "206299"}, { "d0": "2023-03-19", "d1": "2023-06-01", "s": "95414"} ] }, "symbols": ["ES.c.0"], "stype_in": "continuous", "stype_out": "instrument_id", "start_date": "2023-01-01", "end_date": "2023-06-01", "partial": [], "not_found": [], "message": "OK", "status": 0 } ``` Examining this sample we can see a requested mapping of the `stype_in` ("continuous") `symbol` ("ES.c.0") to `stype_out` ("instrument_id") over the date range `start_date` ("2023-01-01") to the `end_date` ("2023-06-01"). We can check the `message` ("OK") and `status` (0) fields to confirm that our request was successful over the entire date interval. Additionally, the `not_found` and `partial` fields are empty. Most importantly, the `result` field contains our symbology mappings keyed by each input `symbol`. Each `symbol` entry in the `result` mapping will contain a list of entries. These entries contain a start date in the `d0` field, and an end date in the `d1` field for the mapping. The `s` field contains the output symbol. | Continuous contract | Start date (d0) | End date (d1) | Instrument ID (s) | | ES.c.0 | 2023-01-01 | 2023-03-19 | 206299 | | ES.c.0 | 2023-03-19 | 2023-06-01 | 95414 | ### SymbolMappingMsg Databento's live data publishes symbology information using the `SymbolMappingMsg`. This message will always contain the input symbol and the resolved output symbol. The record header of the `SymbolMappingMsg` will always contain the `instrument_id`. See our [DBN encoding](https://databento.com/docs/conventions/databento-binary-encoding.md) article for more information on our binary format. | Field | Type | Description | | `stype_in` | uint8_t | The input symbology type (DBN version 2 only). | | `stype_in_symbol` | char\[symbol_cstr_len\] | The input symbol from the subscription, where `symbol_cstr_len` is specified in the [Metadata](https://databento.com/docs/conventions/databento-binary-encoding.md#metadata). | | `stype_out` | uint8_t | The output symbology type (DBN version 2 only). Will always be `raw_symbol`. | | `stype_out_symbol` | char\[symbol_cstr_len\] | The output symbol from the subscription, where `symbol_cstr_len` is specified in the [Metadata](https://databento.com/docs/conventions/databento-binary-encoding.md#metadata). | | `start_ts` | uint64_t | The start of the mapping interval expressed as the number of nanoseconds since the UNIX epoch. | | `end_ts` | uint64_t | The end of the mapping interval expressed as the number of nanoseconds since the UNIX epoch. | --- Source: https://databento.com/docs/conventions/databento-binary-encoding # Databento Binary Encoding **Databento Binary Encoding (DBN)** is an extremely fast message encoding and storage format for normalized market data. The DBN specification includes a simple, self-describing metadata header and a fixed set of struct definitions, which enforce a standardized way to normalize market data. All official Databento client libraries use DBN under the hood, both as a data interchange format and for in-memory representation of data. DBN is also the default encoding for all Databento APIs, including live data streaming, historical data streaming, and batch flat files. ### Getting started with DBN The easiest way to get started with DBN is through any of the official Databento client libraries, which support reading and writing DBN files. See: - The [`DBNStore.from_file`](https://databento.com/docs/api-historical/helpers/dbn-store-from-file.md?lang=python) and [`DBNStore.to_file`](https://databento.com/docs/api-historical/helpers/dbn-store-to-file.md?lang=python) methods for [Python](https://github.com/databento/databento-python) - The [`DbnFileStore::Replay`](https://databento.com/docs/api-historical/helpers/dbn-store.md?lang=cpp) method for [C++](https://github.com/databento/databento-cpp) - The [`AsyncDbnStore::from_file`](https://databento.com/docs/api-historical/helpers/dbn-store-from-file.md?lang=rust) method, and [`databento::dbn::encode`](https://docs.rs/dbn/latest/dbn/encode/index.html) module for [Rust](https://github.com/databento/databento-rs) Other resources are also available: - The [dbn](https://crates.io/crates/dbn) Rust crate is the reference implementation of DBN and provides a library for decoding and encoding DBN, and also for converting from DBN to CSV and JSON. - [dbn-cli](https://crates.io/crates/dbn-cli) is a command line tool that can be used to read DBN files; transcode DBN to CSV or JSON; print the output, or write it to disk. See [Installing dbn-cli](https://databento.com/docs/conventions/databento-binary-encoding.md#installing-dbn-cli) below for setup instructions. ### Installing dbn-cli Pre-built `dbn` binaries are attached to every release of the [databento/dbn](https://github.com/databento/dbn/releases) repository on GitHub. Builds are available for the following targets: - Linux: `x86_64` and `aarch64`, both `gnu` and `musl` - macOS: `x86_64` and `aarch64` - Windows: `x86_64` and `aarch64` Each release ships a `dbn--.tar.gz` archive (or `.zip` on Windows) together with a `.sha256` checksum file. Download the archive for your platform, verify the checksum, extract the archive, and put the `dbn` binary somewhere on your `PATH`. Alternatively, if you have a Rust toolchain installed, you can build and install from source with [Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html): ```sh cargo install dbn-cli ``` ### Why should you use DBN? The key advantages of using DBN are: - **End-to-end.** DBN can be used to store and transport normalized data across all components of a typical trading system. It can fulfill the requirements of a file format for efficient storage; a message encoding for fast real-time streaming, and an in-memory representation of market data for a low latency system--all at once. This simplifies your trading system and eliminates the use of multiple serialization formats. It also ensures that market data is immutable, lossless, and consistent as it passes between components. - **Use the same code for historical and live.** Our client libraries exploit the end-to-end aspect of DBN and allow you to use the same code in historical and live; you can write an event-driven trading platform that runs the exact same code in backtest and production trading. - **Zero-copy.** DBN data is structured the same way whether in-memory, on the wire, or on disk. Thus, the data gets read and written directly as-is, without costly encoding or decoding steps that move the data into CPU and back out. - **Symbology metadata.** The DBN protocol includes a lightweight header that provides metadata for interpreting and using the market data payload, such as symbology mappings, so a DBN file is self-sufficient for many use cases. - **Highly compressible.** DBN strictly uses fixed lengths and offsets for all fields. This layout enables typical compression algorithms, such as zstd and lz4, to achieve high compression ratios. - **Optimized for modern CPUs.** The predictable layout of DBN records also allows for highly-optimized, sequential access patterns that take advantage of instruction pipelining and prefetching on modern CPUs. The struct definitions are also deliberately designed so that most records fit into a single cache line. - **Extremely fast.** DBN achieves extremely fast speeds on reads and writes. Most use cases of DBN are compression-bound or I/O-bound while only using a single CPU core. DBN has been used in environments with 6.1 microseconds median internal latency; we've also seen user-reported benchmarks of full order book replay over 19.1 million messages per second using our C++ client library on a Google Cloud VM. - **Normalization format.** Using DBN also automatically means you're adopting its normalization format. While there are many ways to normalize data, our team arrived at these best practices after many years of combined experience at top-tier trading firms and integrating dozens of trading venues. For example, DBN allows you to replay full order book data at I/O bound, backtest with passive orders in precise sequence, and losslessly achieve much of what's possible with raw packet captures with several factors of improvement in speed and storage requirements. ### Layout A valid DBN stream or file has two parts, beginning with **metadata**, and immediately followed by **records**. The following diagram shows the field layout of the DBN encoding: #### Version 1 ```text 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | magic string = "DBN" | version = 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + dataset + | | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | schema | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | start (UNIX nanos) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | end (UNIX nanos) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | limit (max records) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | reserved | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | stype_in | stype_out | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ts_out | | +-+-+-+-+-+-+-+-+ + | | + + | | + + | | + + | | + + | | + + | reserved (47 bytes of padding) | + + | | + + | | + + | | + + | | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | schema_definition_length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | schema_definition (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | symbols_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | symbols (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | partial_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | partial (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | not_found_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | not_found (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | mappings_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | mappings (variable) | | ... | +-+-+-+-+-+-+-+-+-+-end metadata; begin body--+-+-+-+-+-+-+-+-+-+ | records | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ``` #### Versions 2 and above ```text 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | magic string = "DBN" | version | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + dataset + | | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | schema | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | start (UNIX nanos) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | end (UNIX nanos) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | limit (max records) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | stype_in | stype_out | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ts_out | symbol_cstr_len | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | + + | | + + | | + + | | + + | | + + | | + + | reserved (53 bytes of padding) | + + | | + + | | + + | | + + | | + + | | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | schema_definition_length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | schema_definition (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | symbols_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | symbols (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | partial_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | partial (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | not_found_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | not_found (variable) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | mappings_count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | mappings (variable) | | ... | +-+-+-+-+-+-+-+-+-+-end metadata; begin body--+-+-+-+-+-+-+-+-+-+ | records | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ``` ### Metadata Metadata is included at the beginning of every DBN stream or file. Basic information is found at the start of the metadata, followed by optional [symbology](https://databento.com/docs/api-historical/symbology.md) mappings. The metadata contains all of the parameters needed to construct a request for the exact same data via Databento's historical API. Likewise, if you fetched DBN-encoded data from Databento's historical API, the metadata header will contain the parameters of your original request. The following table describes the metadata fields, in the order of appearance. All fields are little-endian. | Field | Type | Description | | `version` | char\[4\] | `"DBN"` followed by the version of DBN the file is encoded in as a `u8`. | | `length` | uint32\_t | The length of the remaining metadata header, i.e. excluding `version` and `length`. | | `dataset` | char\[16\] | The dataset ID (string identifier). | | `schema` | uint16\_t | The data record schema. `u16::MAX` indicates a potential mix of schemas and record types, which will always be the case for live data. | | `start` | uint64\_t | The start time of query range in UNIX epoch nanoseconds. | | `end` | uint64\_t | The end time of query range in UNIX epoch nanoseconds. `u64::MAX` indicates no end time was provided. | | `limit` | uint64\_t | The maximum number of records to return. 0 indicates no limit. | | `stype_in` | uint8\_t | The symbology type of input symbols. `u8::MAX` indicates a potential mix of types, such as with live data. | | `stype_out` | uint8\_t | The symbology type of output symbols. | | `ts_out` | uint8\_t | Whether each record has an appended gateway send timestamp. | | `symbol_cstr_len` | uint16\_t | The number of bytes in fixed-length string symbols, including a null terminator byte. Version 2 only, symbol strings are always 22 in version 1. | | `schema_definition_length` | uint32\_t | Number of bytes in the schema definition. | | `schema_definition` | uint8\_t\[schema\_definition\_length\] | Self-describing schema to be implemented in the future. | | `symbols_length` | uint32\_t | Number of symbols in the original query. | | `symbols` | char\[symbols\_length\]\[symbol\_cstr\_len\] | The symbols from the original query. | | `partial_length` | uint32\_t | The number of symbols partially resolved. | | `partial` | char\[partial\_length\]\[symbol\_cstr\_len\] | The partially resolved symbols. | | `not_found_length` | uint32\_t | The number of unresolved symbols. | | `not_found` | char\[not\_found\_length\]\[symbol\_cstr\_len\] | The unresolved symbols. | | `mappings_length` | uint32\_t | The number of symbols at least partially resolved. | | `mappings` | SymbolMapping\[mappings\_length\] | The `SymbolMapping`s, one for each resolved symbol. | where `SymbolMapping` has the following structure: | Field | Type | Description | | `raw_symbol` | char\[symbol\_cstr\_len\] | The symbol requested symbol `stype_in`. | | `interval_length` | uint32\_t | The number of `MappingInterval`s in `intervals`. | | `intervals` | MappingInterval\[interval\_length\] | The `MappingInterval`s associated with `raw_symbol`. | and where `MappingInterval` has the following structure: | Field | Type | Description | | `start_date` | uint32\_t | The start date of the interval, as a YYYYMMDD integer. | | `end_date` | uint32\_t | The end date of the interval, as a YYYYMMDD integer. | | `symbol` | char\[symbol\_cstr\_len\] | The symbol in `stype_out` to which `raw_symbol` corresponds for the interval between `start_date` and `end_date`, where `symbol_cstr_len` is specified earlier in the Metadata. This is often `instrument_id` because it is the default `stype_out`. | ### Records The metadata is immediately followed by DBN records. A valid DBN stream or file contains zero or more records. All records begin with the same 16-byte `RecordHeader` with the following structure: | Field | Type | Description | | `length` | uint8\_t | The length of the record in 32-bit words. | | `rtype` | uint8\_t | The record type. Each schema corresponds with a single `rtype` value. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. | | `instrument_id` | uint32\_t | The numeric instrument ID. | | `ts_event` | uint64\_t | The event timestamp as the number of nanoseconds since the [UNIX epoch](https://en.wikipedia.org/wiki/Unix_time). | See the [Schemas and data formats](https://databento.com/docs/schemas.md) section for a full list of fields for the record associated with each schema. ### Versioning We use the version field in the metadata header to signal changes to the structure of record types and metadata. #### Version 2 The following was changed: - Metadata: - Sets `version` to 2 - Adds `symbol_cstr_len` field - Rearranges padding - The fixed-length strings for symbology are now defined to have `symbol_cstr_len` characters (currently 71), whereas in version 1 they always had 22 - `InstrumentDefMsg` (definition schema): - `raw_symbol` now has `symbol_cstr_len` characters (71) - Rearranges padding - `SymbolMappingMsg` (live symbology): - `stype_in_symbol` and `stype_out_symbol` now have `symbol_cstr_len` characters (71) - Adds `stype_in` and `stype_out` fields - Removes padding - `ErrorMsg` (gateway errors in live) - Adds space to `err` for longer error messages - Adds `code` and `is_last` fields - `SystemMsg` (non-error gateway messages in live) - Add space to `msg` for longer messages - Adds `code` field #### Version 3 This set of changes adds support for strategy legs to the definition schema and an expanded `quantity` field in the statistics schemas. - Added 8-byte alignment padding to the end of metadata - Expanded `quantity` to 64 bits in `StatMsg` (statistics schema) - `InstrumentDefMsg` (definition schema): - A definition record will be created for each strategy leg - Adds the following leg fields: - `leg_count` - `leg_index` - `leg_instrument_id` - `leg_raw_symbol` - `leg_side` - `leg_underlying_id` - `leg_instrument_class` - `leg_ratio_qty_numerator` - `leg_ratio_qty_denominator` - `leg_ratio_price_numerator` - `leg_ratio_price_denominator` - `leg_price` - `leg_delta` - Expands `asset` to 11 bytes - Expands `raw_instrument_id` to 64 bits to support publishers that use larger IDs - Removal of statistics-schema related fields `trading_reference_price`, `trading_reference_date`, and `settl_price_type` - Removal of the status-schema related field `md_security_trading_status` > **Info** > CSV and JSON are affected by the new fields. Currently, version 3 is used for the `EQUS.SIP`, `GLBX.MDP3`, `IFEU.IMPACT`, `IFLL.IMPACT`, `IFUS.IMPACT`, `NDEX.IMPACT`, `OCEA.MEMOIR`, `XCBF.PITCH`, `XEEE.EOBI`, and `XEUR.EOBI` datasets. All other datasets use version 1. The DBN crate and client libraries will continue to support decoding earlier versions. **Upgrading to versions** DBN files can be upgraded to the latest version with the `dbn` CLI tool by passing the `--upgrade` or `-u` flag. ```sh dbn version1.dbn --output version3.dbn --upgrade ``` ### Comparison with other encodings and formats DBN is designed specifically for normalized market data. It adopts a fixed set of struct definitions, also called **message schemas**, for this purpose. It's important to note that DBN is not a general-purpose serialization format like Simple Binary Encoding (SBE) or Google Protocol Buffers (protobufs), which provide a flexible schema definition language. Unlike these formats, DBN doesn't offer tools for generating decoders or encoders from any user-specified schema. When comparing DBN to most encodings or serialization formats, a critical difference is that DBN is a zero-copy encoding. Moreover, what makes DBN most unique is that it's simultaneously intended for three common use cases in a trading system: file format, real-time messaging format, and in-memory representation. This is a very specific convergence of use cases that manifests frequently in financial trading systems. Other encodings or formats typically used in situations where DBN would be a suitable replacement include: - SBE - Apache Parquet - Apache Arrow - Feather Comparing to these, DBN is intended to be good at all three abovementioned use cases of a trading system--so you don't have to mix multiple serialization formats in one system--while the others tend to excel only in one or two use cases. A single format for all use cases carries a more important benefit for trading than just the performance upside that comes with minimizing copies; it ensures that market data is immutable when it passes through your trading system. The following diagram helps you visualize the difference between a potential trading system that uses DBN compared to a typical trading system that doesn't. | Typical trading environment (top chart) | Trading environment using DBN (bottom chart) | | Market data in multiple message, file, and in-memory formats | Market data in a single format | | Multiple layers of serialization and deserialization | No transformation of data | | Incurs risk of inconsistent state between components using market data | Eliminates risk of inconsistent state between components using market data | | Complex, slow code | Simple, fast code | ![Typical trading environment](https://databento.com/docs/typical-trading-environment.svg.md)

Typical trading environment

![Trading environment using DBN](https://databento.com/docs/trading-environment-using-dbn.svg.md)

Trading environment using DBN

Immutable market data makes it easy to align live trading with post-production logs and historical data; it makes it easy to use the same code for live trading, backtesting, and exploratory research; it also makes it easy to write GUIs that need accurately synchronize order events with market data events, especially market data events that triggered those order events. In short, using the same encoding or format everywhere ensures that state is synchronized throughout distributed parts of your trading system. For these reasons, most mature trading firms eventually end up implementing their own proprietary encoding that resembles DBN. The following table summarizes other key comparisons: | | **DBN** | **SBE** | **Parquet** | **Arrow** | | **Schema definition** | Fixed schemas | XML | Thrift, Avro, Protobuf | Arrow object model | | **Layout** | Sequential | Sequential | Column-oriented | Column-oriented | | **Zero copy** | Yes | Yes | No | Limited support | | **Suitable for real-time messaging** | Yes | Yes | No | No | | **Suitable as file format** | Yes | Yes | Yes | Through Feather | | **Metadata** | Yes | No, user-defined | No, user-defined | No | | **Sequential read** | Fastest | Fast | Moderate | Moderate | | **Sequential write** | Fastest | Fast | Slowest | Moderate (Feather) | | **Compressed size** | Small | Moderate | Smallest | Largest (Feather) | | **Transcoding to CSV** | Yes | No | Through pandas | Yes | | **Transcoding to JSON** | Yes | No | Through pandas | No | | **Mapping to pandas** | Yes | No | Yes | Yes | | **Package size (lines of code)** | 16.0k (v0.14.0) | 55.7k (v1.27.0) | 108.5k (v1.12.3) | 1.6M (v12.0.0) | | **Language support** | Python, C++, Rust, C bindings | C++, Java, C# | 11+ languages | 11+ languages | | **Use case** | Market data (storage, replay, research, real-time messaging, normalization, OMS, EMS, GUIs) | Direct venue connectivity | Storage file format | Data exploration | ### Frequently asked questions **Isn't this basically a bunch of raw structs? What's so special about this?** Yes, pretty much! And it's not exactly novel--in our experience, most top-tier trading firms will have something similar already implemented, along with proprietary tooling to support it. The significance of DBN is that we're open sourcing the whole toolset, with many best practices for normalization and performance optimization, so that you don't have to reinvent the wheel. Another purpose of DBN is that it provides a standardized data interchange format that can be used for high-throughput, low latency streaming between a data provider like us and you. At the time of the initial release of DBN, we're not aware of any data provider that adopts a binary flat file or messaging format with similar zero-copy semantics. Even if you don't want to use DBN exactly, it's a lightweight specification so it's easy to mimic some of its practices or fork our reference implementation for your own use case. **When should you not use DBN?** - When you depend on many tools and frameworks in the Apache ecosystem. Many data processing tools have native support for Apache Parquet and Arrow. However in our experience, mature trading environments generally use fewer general-purpose computation frameworks. In these cases, DBN is still excellent as an interchange format for receiving and storing data from Databento, and we still support [converting DBN to pandas dataframes](https://databento.com/docs/api-historical/helpers/dbn-store-to-df.md?lang=python) and hence Arrow format. - If you don't use Databento, only ever plan on trading on one trading venue, have already written parsers for the raw feeds, and have direct extranet connectivity, then there's a strong argument for just using the original wire protocol like ITCH and even rolling your own, thinner normalization format. - If you have an academic or toy project and only plan on working with historical data. Many such projects employ relatively small amounts of data and don't require live data. In these circumstances, it makes sense to just store the data in your own, thinner binary format or a binary format with some structure like Parquet or HDF5. - If you have to support many teams with one platform, with different trading styles and business functions, of which many of them only require low frequency data. In this situation, the performance benefits of fixed schemas become much less important within your organization, and the flexibility becomes more important. It's also quite likely in this situation that you have to constantly update your normalization format for new exploratory workflows. In those cases, DBN is still excellent as an interchange format for receiving and storing data from Databento, but your firm will likely benefit from converting DBN data into flexible formats downstream. **Encoding, serialization format, protocol--what's the difference?** There are slight differences in these terms but DBN is all three at once. One way to look at DBN is that it's an OSI layer 6 presentation protocol, much like SBE--except that DBN is much stricter about message schemas, whereas SBE is flexible. Data written in accordance to such a protocol can be persisted to disk, so it can also serve as a storage format; it can also be written on the wire as a message encoding or wire format. SBE excels as a message encoding but is less often used as a storage format, whereas DBN accommodates both equally well. **Why is the reference implementation written in Rust? Can I use it?** The majority of Databento's infrastructure is written in Rust, C, and Python, and the reference implementation of DBN is a rewrite of the original C implementation and was originally written with internal use in mind. Rust's C ABI makes it interoperable across multiple languages with thin bindings, and its memory management model makes it safe and performant. It's simple to integrate the Rust DBN library into your Python application, as seen in our [Python client library](https://github.com/databento/databento-python). **Why is DBN sequential and not column-oriented? Aren't modern column-oriented layouts more optimized for querying?** The sequential layout of DBN makes it more performant for real-time messaging use cases. Column-oriented formats do have the theoretical potential for more optimization in non-real time use cases, but this depends on the actual implementation. Our reference DBN implementation is heavily optimized and still on par with column-oriented formats like Apache Arrow on common use cases for historical market data. **Am I locked-in to a proprietary binary format here?** No, the DBN reference implementation is open-sourced under the permissive [Apache 2.0 License](https://github.com/databento/dbn/blob/main/LICENSE). We also provide transcoders to convert your DBN data into CSV and JSON. **How is DBN being used currently?** We store upwards of 4 PB and over 30 trillion records of normalized historical data in DBN internally at Databento. Every single message that passes through our infrastructure gets encoded in DBN--over billions of messages per day, at single-port peak messaging rates over 60 Gbps, spanning multiple asset classes and over 1.8 million instruments on any given day. It is used for all of our data schemas, including full order book, tick-by-tick trades, top of the book, OHLCV aggregates, venue statistics, instrument definitions, and more. Most of users, including some of the world's largest hedge funds and market making firms, are already using DBN through our client libraries, putting it through multiple production use cases that involve real-time streaming and historical flat files. --- Source: https://databento.com/docs/conventions/working-with-zstandard # Zstandard (zstd) Zstandard is a fast, lossless compression algorithm that offers high compression ratios, making it suitable for storing and transmitting large amounts of data. This compression algorithm is recommended by Databento for all historical streaming and batch downloads, as it minimizes the amount of data transmitted over the network and stored on file systems. You can select Zstandard compression by using the `zstd` option through the client libraries or the HTTP API. Once you have obtained Zstandard compressed data from Databento, you have a number of options for decompressing: - **Databento client libraries.** Decompression from Zstandard is handled internally. Refer to the [API reference](https://databento.com/docs/api-historical.md) for usage details - **dbn-cli.** The Databento CLI tool for working with DBN data also includes support for decompressing `.zst` files. See [Installing dbn-cli](https://databento.com/docs/conventions/databento-binary-encoding.md#installing-dbn-cli) for setup instructions - **7-Zip.** To decompress using [7-Zip](https://www.7-zip.org/) for Windows, you'll need to install the [7-Zip Zstandard edition](https://github.com/mcmilk/7-Zip-zstd). Then right-click on the `.zst` file, navigate to the 7-Zip menu, and select 'Extract here' or 'Extract to \' - **zstd CLI.** Used directly from the command line (described below), or via your own bash scripts ### Installing Zstandard The Zstandard (`zstd`) library can be installed on most operating systems, as detailed below: **macOS** brew install zstd **Linux** Debian/Ubuntu: sudo apt-get update sudo apt-get install zstd CentOS/RHEL: sudo dnf install zstd Arch: sudo pacman -S zstd **Windows** Using Chocolatey: choco install zstd Alternatively, you can also download the Zstandard binaries from the [official releases page on GitHub](https://github.com/facebook/zstd/releases) and add the directory to your `PATH`. You can also install from source by following the instructions in the [Zstandard GitHub repository](https://github.com/facebook/zstd). Verify your installation by checking the version of Zstandard: zstd --version ### Decompressing Once Zstandard is installed on your machine, you can utilize its command line interface (CLI) for various operations, including decompression and recompression. Below, we'll explore commands specifically for decompressing Zstandard-compressed files in different scenarios. To view all available options in the Zstandard CLI, you can run the help command: zstd -h **Decompress to a file** To decompress a file that has been compressed with Zstandard, use the `-d` (or `--decompress`) option followed by the filename. For example, if you have a file named `data.zst`, you can decompress it with the following command: zstd -d data.zst This will create a decompressed file named data in the same directory. > **Tip** > > You can decompress a Zstandard-compressed Databento batched data file by running the following command: > > `zstd -d glbx-mdp3-20231201.trades.csv.zst` > > This will result in the decompressed file, `glbx-mdp3-20231201.trades.csv`, being created in the current directory. If you want to decompress the file to a specific output file, you can use the `-o` option: zstd -d data.zst -o decompressed_data.txt This command will decompress `data.zst` into a file named `decompressed_data.txt`. By default, `zstd` will *not* overwrite existing files. If you need to decompress and overwrite any existing files, you can use the `-f` (force) option: zstd -d -f data.zst This will decompress data.zst and overwrite any existing file with the same name as the output. **Decompressing multiple files** You can also decompress multiple files at once by specifying multiple filenames: zstd -d file1.zst file2.zst file3.zst **Using wildcards for batch decompression** In a directory with multiple `.zst` files, you can decompress all of them using a wildcard (`*`). This is particularly useful for batch processing: zstd -d *.zst This command will decompress all files in the current directory with the `.zst` extension. > **Tip** > > Running this command in a batch download directory will decompress all .zst files at once. **Decompressing to standard output** If you want to decompress a file and output the contents directly to the terminal (standard output), use the `--stdout` option: zstd -d --stdout data.zst This could be useful for piping the decompressed output to other programs in bash scripts. --- Source: https://databento.com/docs/conventions/mbo-snapshot # MBO snapshots An [MBO](https://databento.com/docs/schemas/mbo.md) snapshot represents the order book at a specific point in time, including all outstanding buy and sell orders. The snapshot is streamed as a sequence of MBO records to insert new orders in the book (a sequence of **A**dd [Actions](https://databento.com/docs/conventions/common-fields-enums-types.md#action)). The snapshot records preserve the priority order (per instrument) at each price level, enabling accurate reconstruction of the order book. All snapshot records are marked with the `F_SNAPSHOT` and `F_BAD_TS_RECV` [flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags) (`ts_recv` is set to the snapshot generation timestamp). An instrument's snapshot starts with a clea**R** action, followed by zero or more **A**dd Actions. The snippet below shows the result of a snapshot request for 2 instruments. ```python MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718539204593519451 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 0 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718539204593519451 }, order_id: 6413364814613, price: 5560.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 752 } ... MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718582343502504063 }, order_id: 6413384952694, price: 5655.000000000, size: 2, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175066 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718582343502957759 }, order_id: 6413384952695, price: 5691.000000000, size: 2, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175067 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718539204593519451 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 0 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718539204593519451 }, order_id: 6413256341927, price: 4650.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 519 } ... MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718582366242085265 }, order_id: 6413384879630, price: 5612.500000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175633 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718582366260973309 }, order_id: 6413384879629, price: 5632.500000000, size: 1, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175760 } ``` The snapshot for an empty order book has a single MBO record with clea**R** action, marked with the flags `F_SNAPSHOT` and `F_LAST`, as in the snippet below. ```python MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 14160, ts_event: 1718117495809255541 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'R', side: 'N', ts_recv: 18446744073709551615, ts_in_delta: 0, sequence: 0 } ``` MBO snapshots are available from both the historical and live APIs. ### Historical API We offer MBO snapshot through the [Historical API](https://databento.com/docs/api-historical.md) for venues that follow a weekly session structure ([CME Globex MDP 3.0](https://databento.com/docs/datasets/glbx-mdp3.md)), and for venues whose daily trading sessions cross 00:00:00 UTC ([ICE Europe Commodities iMpact](https://databento.com/docs/datasets/ifeu-impact.md)). The order book snapshot is generated at 00:00:00 UTC each weekday (Monday-Friday). Snapshot records are streamed through the [Historical API](https://databento.com/docs/api-historical/timeseries.md) when the requested interval includes midnight UTC for a given weekday. The snippet below shows an example of a timeseries request in Python whose result includes an MBO snapshot. ```python client.timeseries.get_range( dataset="GLBX.MDP3", symbols="ES.c.2", stype_in="continuous", schema="mbo", start="2024-06-16T23:58:50", end="2024-06-17T00:00:10", ) ``` ```python MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582336782043745 }, order_id: 6413384952478, price: 5564.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582336782569129, ts_in_delta: 12306, sequence: 174718 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582336782045281 }, order_id: 6413384879627, price: 5560.500000000, size: 1, flags: LAST (130), channel_id: 0, action: 'M', side: 'B', ts_recv: 1718582336782613211, ts_in_delta: 12838, sequence: 174721 } ... MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582388400474957 }, order_id: 6413384953470, price: 5562.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582388400587693, ts_in_delta: 14247, sequence: 176276 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718539204593519451 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 0 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718539204593519451 }, order_id: 6413383913050, price: 5500.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 936 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718539204593519451 }, order_id: 6413373623913, price: 5475.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 936 } ... MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582388400474957 }, order_id: 6413384953470, price: 5562.750000000, size: 1, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 176276 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582400066482471 }, order_id: 6413384953619, price: 5564.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582400066580869, ts_in_delta: 14145, sequence: 176515 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582401163834877 }, order_id: 6413384953619, price: 5564.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'C', side: 'A', ts_recv: 1718582401163946845, ts_in_delta: 13326, sequence: 176774 } ... MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582409155703017 }, order_id: 6413384953440, price: 5413.250000000, size: 1, flags: LAST (130), channel_id: 0, action: 'M', side: 'B', ts_recv: 1718582409155796258, ts_in_delta: 12560, sequence: 178707 } ``` ### Live API Users can request an MBO snapshot through the live API to obtain the recent order book state without replaying the whole trading session. The following sequence of messages is streamed from a live session after a snapshot subscription. 1. [Symbol mapping messages](https://databento.com/docs/api-live/basics/symbology.md) 2. Snapshot records (clear book and outstanding orders at each price level) 3. Real-time records > **Warning** > > The order book of a given instrument is not guaranteed to be in a complete and valid state after the last snapshot record because it might not correspond to the final event record (indicated with the `F_LAST` [flag](https://databento.com/docs/conventions/common-fields-enums-types.md#flags)). > If the last snapshot record does not have the `F_LAST` flag set, the order book will not be valid until the next MBO record with the `F_LAST` flag set. The example below shows the live stream for a snapshot subscription for a single instrument, followed by a real-time record. ```python SymbolMappingMsg { hd: RecordHeader { length: 44, rtype: SymbolMapping, publisher_id: 0, instrument_id: 118, ts_event: 1721732190956747490 }, stype_in: 255, stype_in_symbol: "ES.c.0", stype_out: 255, stype_out_symbol: "ESU4", start_ts: 18446744073709551615, end_ts: 18446744073709551615 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152358684229 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 0 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514530616, price: 5547.500000000, size: 8, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1022 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514530330, price: 5547.250000000, size: 7, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1022 } ... MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413506441952, price: 5628.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'A', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1101 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413500305436, price: 5628.250000000, size: 6, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'A', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1058 } ... MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152290217707 }, order_id: 6413522310536, price: 5613.250000000, size: 3, flags: SNAPSHOT | BAD_TS_RECV (42), channel_id: 0, action: 'A', side: 'A', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 13076493 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152290632121 }, order_id: 6413522309988, price: 5512.500000000, size: 6, flags: LAST | SNAPSHOT | BAD_TS_RECV (170), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 13076496 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152365904111 }, order_id: 6413522310535, price: 5613.250000000, size: 1, flags: LAST (130), channel_id: 0, action: 'C', side: 'A', ts_recv: 1721732152366001513, ts_in_delta: 14406, sequence: 13076497 } MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152365916169 }, order_id: 6413522310537, price: 5613.000000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152366014389, ts_in_delta: 12865, sequence: 13076498 } ... ``` ### Timestamp fields on snapshot messages The [timestamp](https://databento.com/docs/conventions/common-fields-enums-types.md#timestamps) fields for snapshot messages are described below. - `ts_event`: unchanged. The clea**R** book MBO record from the Live API snapshot contains the snapshot generation timestamp instead - `ts_in_delta`: always 0 - `ts_recv`: set to the snapshot generation timestamp (indicated with the `F_BAD_TS_RECV` flag) - `ts_out`: unchanged ### Public client support for MBO snapshot Live and historical snapshot features are available on all our official client libraries ([Python](https://github.com/databento/databento-python), [C++](https://github.com/databento/databento-cpp), [Rust](https://github.com/databento/databento-rs)), as well as through our [Raw API](https://databento.com/docs/api-live.md). An example of a live snapshot subscription using our client libraries can be found in [this article](https://databento.com/docs/examples/basics-live/live-snapshot.md). --- Source: https://databento.com/docs/conventions/reference-data-enums # Reference data enums Exchanges, event types, other fields included in Databento's corporate actions data and reference data API. Over 60 event types, including splits, dividends, adjustment factors, and listings. The following tables specify the descriptions for all reference data enums found in responses. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ### ACTION | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `C` | Cancelled | | `D` | Deleted | | `I` | Inserted | | `P` | Payment details Cancelled by Issuer | | `Q` | Payment details Deleted by Data Supplier | | `U` | Updated | ### CNTRY | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `AA` | Supranational | | `AD` | Andorra | | `AE` | United Arab Emirates | | `AF` | Afghanistan | | `AG` | Antigua and Barbuda | | `AI` | Anguilla | | `AL` | Albania | | `AM` | Armenia | | `AO` | Angola | | `AR` | Argentina | | `AS` | American Samoa | | `AT` | Austria | | `AU` | Australia | | `AW` | Aruba | | `AZ` | Azerbaijan | | `BA` | Bosnia and Herzegovina | | `BB` | Barbados | | `BD` | Bangladesh | | `BE` | Belgium | | `BF` | Burkina Faso | | `BG` | Bulgaria | | `BH` | Bahrain | | `BI` | Burundi | | `BJ` | Benin | | `BM` | Bermuda | | `BN` | Brunei Darussalam | | `BO` | Bolivia | | `BR` | Brazil | | `BS` | Bahamas | | `BT` | Bhutan | | `BV` | Bouvet Island | | `BW` | Botswana | | `BY` | Belarus | | `BZ` | Belize | | `CA` | Canada | | `CC` | Cocos Islands | | `CD` | Congo Democratic Republic | | `CF` | Central African Republic | | `CG` | Congo | | `CH` | Switzerland | | `CI` | Ivory Coast | | `CK` | Cook Islands | | `CL` | Chile | | `CM` | Cameroon | | `CN` | China | | `CO` | Colombia | | `CR` | Costa Rica | | `CU` | Cuba | | `CV` | Cape Verde | | `CW` | Curaçao | | `CX` | Christmas Islands | | `CY` | Cyprus | | `CZ` | Czech Republic | | `DE` | Germany | | `DJ` | Djibouti | | `DK` | Denmark | | `DM` | Dominica | | `DO` | Dominican Republic | | `DZ` | Algeria | | `EC` | Ecuador | | `EE` | Estonia | | `EG` | Egypt | | `EH` | Western Sahara | | `ES` | Spain | | `ET` | Ethiopia | | `EU` | Europe | | `FI` | Finland | | `FJ` | Fiji | | `FK` | Falkland Islands | | `FM` | Micronesia | | `FO` | Faroe Islands | | `FR` | France | | `GA` | Gabon | | `GB` | United Kingdom | | `GD` | Grenada | | `GE` | Georgia | | `GF` | French Guyana | | `GG` | Guernsey | | `GH` | Ghana | | `GI` | Gibraltar | | `GL` | Greenland | | `GM` | Gambia | | `GN` | Guinea | | `GP` | Guadeloupe | | `GQ` | Equatorial Guinea | | `GR` | Greece | | `GT` | Guatemala | | `GU` | Guam | | `GW` | Guinea-Bissau | | `GY` | Guyana | | `HK` | Hong Kong | | `HM` | Heard Island & McDonald Island | | `HN` | Honduras | | `HR` | Croatia | | `HT` | Haiti | | `HU` | Hungary | | `ID` | Indonesia | | `IE` | Ireland | | `IL` | Israel | | `IM` | Isle of Man | | `IN` | India | | `IO` | British Indian Ocean territory | | `IQ` | Iraq | | `IR` | Iran | | `IS` | Iceland | | `IT` | Italy | | `JE` | Jersey | | `JM` | Jamaica | | `JO` | Jordan | | `JP` | Japan | | `KE` | Kenya | | `KG` | Kyrgyz Republic | | `KH` | Cambodia | | `KI` | Kiribati | | `KM` | Comoros | | `KN` | St. Kitts & Nevis | | `KP` | Korea (North) | | `KR` | Korea (South) | | `KW` | Kuwait | | `KY` | Cayman Islands | | `KZ` | Kazakhstan | | `LA` | Laos | | `LB` | Lebanon | | `LC` | St. Lucia | | `LI` | Liechtenstein | | `LK` | Sri Lanka | | `LR` | Liberia | | `LS` | Lesotho | | `LT` | Lithuania | | `LU` | Luxembourg | | `LV` | Latvia | | `LY` | Libya | | `MA` | Morocco | | `MC` | Monaco | | `MD` | Moldova | | `ME` | Montenegro | | `MG` | Madagascar | | `MH` | Marshall Islands | | `MK` | North Macedonia | | `ML` | Mali | | `MM` | Myanmar | | `MN` | Mongolia | | `MO` | Macao | | `MP` | Northern Mariana Islands | | `MQ` | Martinique | | `MR` | Mauritania | | `MS` | Montserrat | | `MT` | Malta | | `MU` | Mauritius | | `MV` | Maldives | | `MW` | Malawi | | `MX` | Mexico | | `MY` | Malaysia | | `MZ` | Mozambique | | `NA` | Namibia | | `NC` | New Caledonia | | `NE` | Niger | | `NF` | Norfolk Island | | `NG` | Nigeria | | `NI` | Nicaragua | | `NL` | Netherlands | | `NO` | Norway | | `NP` | Nepal | | `NR` | Nauru | | `NU` | Niue | | `NZ` | New Zealand | | `OM` | Oman | | `PA` | Panama | | `PE` | Peru | | `PF` | French Polynesia | | `PG` | Papua New Guinea | | `PH` | Philippines | | `PK` | Pakistan | | `PL` | Poland | | `PM` | St. Pierre and Miquelon | | `PN` | Pitcairn | | `PR` | Puerto Rico | | `PS` | Palestine | | `PT` | Portugal | | `PW` | Palau | | `PY` | Paraguay | | `QA` | Qatar | | `RE` | Reunion | | `RO` | Romania | | `RS` | Serbia | | `RU` | Russia | | `RW` | Rwanda | | `SA` | Saudi Arabia | | `SB` | Solomon Islands | | `SC` | Seychelles | | `SD` | Sudan | | `SE` | Sweden | | `SG` | Singapore | | `SH` | St. Helena | | `SI` | Slovenia | | `SJ` | Svalbard and Jan Mayen | | `SK` | Slovak Republic | | `SL` | Sierra Leone | | `SM` | San Marino | | `SN` | Senegal | | `SO` | Somalia | | `SR` | Suriname | | `ST` | Sao Tome and Principe | | `SV` | El Salvador | | `SY` | Syria | | `SZ` | Eswatini | | `TC` | Turks and Caicos Islands | | `TD` | Chad | | `TF` | French Southern Territories | | `TG` | Togo | | `TH` | Thailand | | `TJ` | Tajikistan | | `TK` | Tokelau | | `TM` | Turkmenistan | | `TN` | Tunisia | | `TO` | Tonga | | `TP` | East Timor | | `TR` | Turkiye | | `TT` | Trinidad and Tobago | | `TV` | Tuvalu | | `TW` | Taiwan | | `TZ` | Tanzania | | `UA` | Ukraine | | `UG` | Uganda | | `UM` | United States Minor Outlying | | `US` | United States of America | | `UY` | Uruguay | | `UZ` | Uzbekistan | | `VA` | Vatican City | | `VC` | St. Vincent & the Grenadines | | `VE` | Venezuela | | `VG` | Virgin Islands (UK) | | `VI` | Virgin Islands (US) | | `VN` | Vietnam | | `VU` | Vanuatu | | `WF` | Wallis & Futuna Islands | | `WS` | Samoa | | `XG` | Shanghai SC | | `XH` | Hong Kong SC | | `XZ` | Shenzhen SC | | `YE` | Yemen | | `YT` | Mayotte | | `ZA` | South Africa | | `ZM` | Zambia | | `ZW` | Zimbabwe | | `ZZ` | Unclassified | ### CUREN | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `AED` | UAE Dirham | | `AFN` | Afghanis | | `ALL` | Albanian Lek | | `AMD` | Armenian Dram | | `ANG` | Netherlands Antilles Guilders | | `AOA` | Angola Kwanza | | `ARS` | Argentine Peso | | `AUD` | Australian Dollar | | `AWG` | Aruban Guilder | | `AZN` | Azerbaijani Manat | | `BAM` | Convertible Marks | | `BBD` | Barbados Dollar | | `BDT` | Bangladesh Taka | | `BGN` | Bulgarian Lev | | `BHD` | Bahraini Dinar | | `BIF` | Burundi Franc | | `BMD` | Bermuda Dollar | | `BND` | Brunei Dollar | | `BOB` | Boliviano | | `BOV` | Mvdol | | `BRL` | Brazilian Real | | `BSD` | Bahamas Dollar | | `BTN` | Bhutanese Ngultrum | | `BWP` | Botswana Pula | | `BYN` | Belarusian Ruble (New) | | `BZD` | Belize Dollar | | `CAD` | Canadian Dollar | | `CDF` | Congolese Franc | | `CHF` | Swiss Francs | | `CLF` | Chilean Unidad de Fomento | | `CLP` | Chilean Peso | | `CNY` | Chinese Yuan Renminbi | | `COP` | Colombian Peso | | `COU` | Colombian (Unidad de Valor Real) | | `CRC` | Costa Rican Colon | | `CUP` | Cuban Peso | | `CVE` | Cape Verde Escudo | | `CYP` | Cypriot Pound | | `CZK` | Czech Koruna | | `DJF` | Djibouti Franc | | `DKK` | Danish Kroner | | `DOP` | Dominican Peso | | `DZD` | Algerian Dinar | | `ECS` | Ecuador Sucre | | `EEK` | Estonian Kroon | | `EGP` | Egyptian Pound | | `ERN` | Eritrean Nakfa | | `ETB` | Ethiopian Birr | | `EUR` | Euros | | `FJD` | Fiji Dollar | | `FKP` | Falklands Pounds | | `GBP` | Pound Sterling | | `GBX` | GB Pence | | `GEL` | Georgian Lari | | `GHS` | Ghanaian Cedi | | `GIP` | Gibraltar Pounds | | `GMD` | Gambian Dalasi | | `GNF` | Guinean Franc | | `GRD` | Greek Drachma | | `GTQ` | Guatemalan Quetzal | | `GYD` | Guyana Dollar | | `HKD` | Hong Kong Dollar | | `HNL` | Honduras Lempira | | `HRK` | Croatian Kuna | | `HTG` | Haiti Gourde | | `HUF` | Hungarian Forint | | `IDR` | Indonesian Rupiah | | `ILS` | Israeli New Shekel | | `INR` | Indian Rupees | | `IQD` | Iraqi Dinar | | `IRR` | Iranian Rial | | `ISK` | Icelandic Krona | | `JMD` | Jamaican Dollar | | `JOD` | Jordanian Dinar | | `JPY` | Japanese Yen | | `KES` | Kenyan Shilling | | `KGS` | Kyrgyzstan Som | | `KHR` | Cambodian Riel | | `KMF` | Comoro Franc | | `KPW` | North Korean Won | | `KRW` | Korean Won | | `KWD` | Kuwaiti Dinar | | `KYD` | Cayman Islands Dollar | | `KZT` | Kazakhstan Tenge | | `LAK` | Lao Liberation Kip | | `LBP` | Lebanese Pound | | `LKR` | Sri Lankan Rupee | | `LRD` | Liberian Dollar | | `LSL` | Lesotho Loti | | `LTL` | Lithuanian Litas | | `LYD` | Libyan Dinar | | `MAD` | Moroccan Dirham | | `MDL` | Moldovan Leu | | `MGA` | Malagasy Ariary | | `MKD` | Macedonian Denar | | `MMK` | Myanmar Kyat | | `MNT` | Mongolian Tugrik | | `MOP` | Macau Pataca | | `MRO` | Mauritanian Ouguiya | | `MUR` | Mauritius Rupee | | `MVR` | Maldivian Rufiyaa | | `MWK` | Malawi Kwacha | | `MXN` | Mexican Nuevo Peso | | `MXV` | Mexican Unidad de Inversion (UDI) | | `MYR` | Malaysian Ringgit | | `MZN` | Mozambique Metical | | `NAD` | Namibian Dollar | | `NGN` | Nigerian Naira | | `NIO` | Nicaraguan Cordoba Oro | | `NOK` | Norwegian Krone | | `NPR` | Nepalese Rupee | | `NZD` | New Zealand Dollar | | `OMR` | Omani Rial | | `PAB` | Panama Balboa | | `PEN` | Peruvian Nuevo Sol | | `PGK` | Papua New Guinea Kina | | `PHP` | Philippines Peso | | `PKR` | Pakistan Rupee | | `PLN` | Polish Złoty (New) | | `PYG` | Paraguay Guarani | | `QAR` | Qatar Rial | | `RON` | Romanian Leu (New) | | `RSD` | Serbian Dinars | | `RUB` | Russian Ruble (New) | | `RWF` | Rwandan Franc | | `SAR` | Saudi Arabian Riyal | | `SBD` | Solomon Islands Dollar | | `SCR` | Seychelles Rupee | | `SDD` | Sudanese Dinar | | `SDG` | Sudanese Pound | | `SEK` | Swedish Kroner | | `SGD` | Singapore Dollar | | `SHP` | St. Helena Pounds | | `SLL` | Sierra Leone | | `SOS` | Somalia Shilling | | `SRD` | Surinam Dollar | | `STD` | Sao Tome and Principe Dobra | | `SVC` | El Salvador Colon | | `SYP` | Syrian Pound | | `SZL` | Swaziland Lilangeni | | `THB` | Thai Baht | | `TJS` | Tajikistani Somoni | | `TMM` | Turkmenistan Manat | | `TND` | Tunisian Dinar | | `TOP` | Tonga Pa`anga | | `TRY` | Turkish Lira (New) | | `TTD` | Trinidad and Tobago Dollar | | `TWD` | Taiwan Dollar | | `TZS` | Tanzanian Shilling | | `UAH` | Ukrainian Hryvnia | | `UGX` | Ugandan Shilling | | `USD` | US Dollar | | `USX` | US Cents | | `UYI` | Uruguay Peso (Index Linked) | | `UYU` | Uruguayan Peso | | `UYW` | Uruguayan Unidad Previsional (Pension Unit) | | `UZS` | Uzbekistan Sum | | `VEF` | Venezuela Bolivares Fuertes | | `VES` | Venezuela Sovereign Bolivar | | `VND` | Vietnamese Dong | | `VUV` | Vanuatu Vatu | | `WST` | Samoan Tala | | `XAF` | CFA Franc (BEAC) | | `XCD` | Caribbean Dollar | | `XDR` | International Monetary Fund | | `XFU` | UIC-Franc | | `XOF` | CFA Franc (BCEAO) | | `XPF` | CFP Franc | | `XTS` | Codes for testing purposes | | `XXX` | Codes for transactions/no currencies involved | | `YER` | North Yemen Rial | | `ZAC` | South African Cents | | `ZAR` | South African Rand | | `ZMK` | Zambian Kwacha | | `ZMW` | Zambian New Kwacha | | `ZRN` | New Zaire | | `ZWD` | Zimbabwe Dollar | | `ZWG` | Zimbabwe Gold | | `ZWL` | Zimbabwean Dollar | ### DRTYPE | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `ADR` | American DR | | `BDR` | Belgium DR | | `CDI` | Chess Depository Interest | | `CDR` | Cedear | | `DR` | Generic DR | | `EDR` | European DR | | `FDR` | Finland DR | | `GDR` | Global DR | | `HDR` | Ghanian DR | | `IDR` | International DR | | `NDR` | Netherland Bearer DR | | `NVD` | Non-Voting DR | | `NYS` | New York Shares | | `PDR` | Philippine DR | | `RDR` | Russian DR | | `SDR` | Swedish DR | | `SDS` | Singapore DR | | `TDR` | Taiwan DR | | `UDR` | United Kingdom DR | | `ZDR` | Zimbabwe DR | ### EVENT | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `AGM` | Company Meeting | | `ANN` | Announcement | | `ARR` | Arrangement | | `ASSM` | Assimilation | | `BB` | Buyback | | `BBCC` | Bloomberg Composite ID Change | | `BBEC` | Bloomberg Exchange ID Change | | `BKRP` | Bankruptcy | | `BON` | Bonus Issue | | `BR` | Bonus Rights | | `CALL` | Call | | `CAPRD` | Capital Reduction | | `CLSAC` | Class Action | | `CONSD` | Consolidation | | `CONV` | Conversion | | `CTX` | Certificate of Exchange | | `CURRD` | Currency Redenomination | | `DIST` | Distribution | | `DIV` | Dividend | | `DIVEB` | Dividend - Equity Bifurcated | | `DIVRC` | Dividend Reclassification | | `DMRGR` | Demerger | | `DRCHG` | Depository Receipt Change | | `DRIP` | Dividend Reinvestment Plan | | `DVST` | Divestment | | `ENT` | Entitlement Issue | | `FRANK` | Franking | | `FSPLT` | Forward Split, currently only used in US events | | `FTT` | Financial Transaction Tax | | `FYCHG` | Financial Year Change | | `ICC` | International Code Change | | `INCHG` | Country of Incorporation Change | | `ISCHG` | Issuer Name Change | | `LCC` | Local Code Change | | `LIQ` | Liquidation | | `LSTAT` | Listing Status | | `LTCHG` | Lot Change | | `MKCHG` | Market Segment Change | | `MRGR` | Merger | | `NLIST` | New Listing | | `ODDLT` | Odd Lot Offer | | `PID` | Property Income Distribution | | `PO` | Purchase Offer | | `PRCHG` | Primary Exchange Change | | `PRF` | Preferential Offer | | `PVRD` | Par Value Redenomination | | `RCAP` | Return of Capital | | `RD` | Record Date | | `REDEM` | Redemption | | `RSPLT` | Reverse Split, currently only used in US events | | `RTS` | Rights | | `SCCHG` | Security Name Change | | `SCSWP` | Security Swap | | `SD` | Sub Division | | `SDCHG` | Sedol Change | | `SECRC` | Security Re-classification | | `SHOCH` | Shares Outstanding Change | | `SOFF` | Spin-Off, currently only used in US events | | `TKOVR` | Takeover | | `WAREX` | Warrant Exercise | ### EVENTSUBTYPE | Event [//]: # (column: sortable) | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `AGM` | `AGM` | Annual General Meeting | | `AGM` | `BHM` | Bond Holder Meeting | | `AGM` | `CGM` | Court Ordered General Meeting | | `AGM` | `EGM` | Extraordinary General Meeting | | `AGM` | `GM` | General Meeting | | `AGM` | `SGM` | Special General Meeting | | `BB` | `DUTCHAUCT` | Dutch Auction | | `BON` | `DPRCPDIV` | Depository Receipt Dividend | | `CONSD` | `CONSD` | Reverse Split, currently only used in US events | | `CONV` | `BB` | Buyback | | `CONV` | `CALL` | Call Option Exercised | | `CONV` | `CLAIMSET` | Claim Settled | | `CONV` | `DEFPY` | Default Payment | | `CONV` | `DR` | Drawings | | `CONV` | `DRL` | Drawings by lottery | | `CONV` | `ECONV` | Early Conversion | | `CONV` | `MAT` | Maturity | | `CONV` | `ORD` | Ordinary | | `CONV` | `PUT` | Put Option Exercised | | `CONV` | `REDEMCLAIM` | Redemption Claim | | `CONV` | `WRTDN` | Write Down | | `DIST` | `DIST` | Spin-Off, currently only used in US events | | `DIV` | `CAPDIST` | Capital Distribution | | `DIV` | `CAPGAIN` | Capital Gain | | `DIV` | `DIV` | Forward Split, currently only used in US events | | `DIV` | `INTDIV` | Derived from Interest Payment | | `DIVEB` | `DT` | Tax Free Dividend Component | | `DIVEB` | `RCAP` | Return of Capital Component | | `DIVEB` | `ROD` | Repayment of Debt Component | | `DIVIF` | `DIVACC` | Dividend Accumulation | | `DIVIF` | `DIVINC` | Dividend Income | | `DIVIF` | `INT` | Interest Basis Unknown | | `DIVIF` | `INTACC` | Interest Accumulation | | `DIVIF` | `INTINC` | Interest Income | | `DIVIF` | `PRO` | Property Basis Unknown | | `DIVIF` | `PROACC` | Property Accumulation | | `DIVIF` | `PROINC` | Property Income | | `DMRGR` | `DMRGR` | Spin-Off, currently only used in US events | | `ENT` | `NRENRTS` | Non Renounceable Rights | | `ENT` | `OPOFF` | Open Offer | | `ENT` | `POFF` | Priority Offer | | `ENT` | `SPP` | Share Purchase Plan | | `FRANK` | `F` | Fully franked | | `FRANK` | `N` | Not known | | `FRANK` | `P` | Partially franked | | `FRANK` | `U` | Unfranked | | `FSPLT` | `BON` | Bonus | | `FSPLT` | `DIV` | Dividend | | `FSPLT` | `SD` | Subdivision | | `RCAP` | `CAPRD` | Capital Reduction | | `RCAP` | `LIQ` | Liquidation | | `RCAP` | `RES` | Reserves | | `RCAP` | `SOA` | Sale of Assets | | `RCAP` | `SPA` | Share Premium Account | | `REDEM` | `AMT` | Amortisation | | `REDEM` | `BBED` | Buyback Early Deadline | | `REDEM` | `BBRD` | Buyback Regular Deadline | | `REDEM` | `CORR` | Correction | | `REDEM` | `CU` | Clean Up | | `REDEM` | `ER` | Early Redemption | | `REDEM` | `MWC` | Make Whole Call | | `REDEM` | `PF` | Purchase Fund | | `REDEM` | `RM` | Residual Maturity | | `RSPLT` | `CONSD` | Consolidation | | `RTS` | `DPRCPDIV` | Depository Receipt Dividend | | `SD` | `SD` | Forward Split, currently only used in US events | | `SOFF` | `DIST` | Distribution | | `SOFF` | `DMRGR` | Demerger | | `TKOVR` | `MRGR` | Merger | | `TKOVR` | `TEND` | Tender Offer | | `TKOVR` | `TENDMRGR` | Tender resulting in Merger | | `TKOVR` | `TKOVRMINI` | Mini-Takeover | | `TKOVR` | `UKWNSUBTYP` | Insufficient data to assign a TKOVR event subtype | ### EXCHANGE | Name [//]: # (column: sortable) | Country [//]: # (column: sortable) | Exchange [//]: # (column: sortable) | MIC [//]: # (column: sortable) | | A2X Markets | South Africa | `ZAA2X` | `A2XX` | | AIAF - Fixed Income Market | Spain | `ESAIAF` | `XDRF` | | Abu Dhabi Securities Exchange | United Arab Emirates | `AEADSM` | `XADS` | | Algiers Stock Exchange | Algeria | `DZASE` | `XALG` | | Amman Stock Exchange | Jordan | `JOASE` | `XAMM` | | Angola Securities Exchange | Angola | `AOASE` | `XBDV` | | Aquis Stock Exchange | United Kingdom | `GBOFX` | `AQSE` | | Argentina Open Electronic Exchange | Argentina | `AROEX` | `XMAB` | | Armenia Securities Exchange | Armenia | `AMASE` | `XARM` | | Astana International Exchange | Kazakhstan | `KZAIX` | `AIXK` | | Asuncion Stock Exchange | Paraguay | `PYBVA` | `XVPA` | | Athens Exchange | Greece | `GRASE` | `ASEX` | | Australian Securities Exchange | Australia | `AUASX` | `XASX` | | B3 - Brasil Bolsa Balcao | Brazil | `BRBVSP` | `BVMF` | | BME Growth | Spain | `ESMAB` | `GROW` | | BSE | India | `INBSE` | `XBOM` | | BX Swiss | Switzerland | `CHBRN` | `XBRN` | | Bahamas International Securities Exchange | Bahamas | `BSBSE` | `XBAA` | | Bahrain Bourse | Bahrain | `BHBSE` | `XBAH` | | Baku Stock Exchange | Azerbaijan | `AZBSE` | `BSEX` | | Banja Luka Stock Exchange | Bosnia and Herzegovina | `BABLSX` | `XBLB` | | Barbados Stock Exchange | Barbados | `BBSEB` | `XBAB` | | Barcelona Stock Exchange | Spain | `ESBSE` | `XBAR` | | Beijing Stock Exchange | China | `CNBJSE` | `BJSE` | | Beirut Stock Exchange | Lebanon | `LBBSE` | `XBEY` | | Belarus Currency and Stock Exchange | Belarus | `BYBCSE` | `BCSE` | | Belgrade Stock Exchange | Serbia | `RSBEL` | `XBEL` | | Berlin Stock Exchange | Germany | `DEBSE` | `XBER` | | Bermuda Stock Exchange | Bermuda | `BMBSE` | `XBDA` | | Bilbao Stock Exchange | Spain | `ESBBSE` | `XBIL` | | Bolivian Stock Exchange | Bolivia | `BOBVB` | `XBOL` | | Bolsa Electronica de Chile | Chile | `CLBEC` | `XBCL` | | Bolsa Mexicana de Valores | Mexico | `MXMSE` | `XMEX` | | Bolsa Nacional de Valores | Costa Rica | `CRBNV` | `XBNV` | | Bolsa de Valores de Lima | Peru | `PEBVL` | `XLIM` | | Borsa Italiana | Italy | `ITMSE` | `XMIL` | | Botswana Stock Exchange | Botswana | `BWBSE` | `XBOT` | | Bratislava Stock Exchange | Slovakia | `SKBSE` | `XBRA` | | Bucharest Stock Exchange | Romania | `ROBSE` | `XBSE` | | Budapest Stock Exchange | Hungary | `HUBSE` | `XBUD` | | Buenos Aires Stock Exchange | Argentina | `ARMVB` | `XBUE` | | Bulgarian Stock Exchange | Bulgaria | `BGBSE` | `XBUL` | | Cambodia Securities Exchange | Cambodia | `KHCSX` | `XCSX` | | Canadian Securities Exchange | Canada | `CACNQ` | `XCNQ` | | Cape Town Stock Exchange | South Africa | `ZA4AX` | `4AXE` | | Cape Verde Stock Exchange | Cape Verde | `CVCVSX` | `XBVC` | | Caracas Stock Exchange | Venezuela | `VECSE` | `BVCA` | | Casablanca Stock Exchange | Morocco | `MABVC` | `XCAS` | | Cayman Islands Stock Exchange | Cayman Islands | `KYSX` | `XCAY` | | Cboe Australia | Australia | `AUCBOE` | `CHIA` | | Cboe BZX | United States | `USBATS` | `BATS` | | Cboe Canada | Canada | `CANEO` | `NEOE` | | Central American Securities Exchange | Honduras | `HNCASE` | `XBCV` | | China Interbank Bond Market | China | `CNIBBM` | `N/A` | | Chittagong Stock Exchange | Bangladesh | `BDCSE` | `XCHG` | | Colombian Stock Exchange | Colombia | `COCSX` | `XBOG` | | Colombo Stock Exchange | Sri Lanka | `LKCSE` | `XCOL` | | Cyprus Stock Exchange | Cyprus | `CYCSE` | `XCYS` | | Damascus Securities Exchange | Syria | `SYDSE` | `XDSE` | | Dar es Salaam Stock Exchange | Tanzania | `TZTSE` | `XDAR` | | Dhaka Stock Exchange | Bangladesh | `BDDSE` | `XDHA` | | Dominican Republic Stock Exchange | Dominican Republic | `DODRSX` | `XBVR` | | Douala Stock Exchange | Cameroon | `CMDSX` | `XDSX` | | Dubai Financial Market | United Arab Emirates | `AEDFM` | `XDFM` | | Dusseldorf Stock Exchange | Germany | `DEDSE` | `XDUS` | | Dutch Caribbean Securities Exchange | Curacao | `CWDCSE` | `DCSX` | | Eastern Caribbean Securities Exchange | St. Kitts & Nevis | `KNECSE` | `XECS` | | Egyptian Exchange | Egypt | `EGCASE` | `XCAI` | | El Salvador Stock Exchange | El Salvador | `SVMVSV` | `XSVA` | | Eswatini Stock Exchange | Eswatini | `SZSS` | `XSWA` | | Euronext Amsterdam | Netherlands | `NLENA` | `XAMS` | | Euronext Brussels | Belgium | `BEENB` | `XBRU` | | Euronext Dublin | Ireland | `IEISE` | `XDUB` | | Euronext Lisbon | Portugal | `PTBVL` | `XLIS` | | Euronext London | United Kingdom | `GBENLN` | `XLDN` | | Euronext NOTC | Norway | `NOOTC` | `NOTC` | | Euronext Paris | France | `FRPEN` | `XPAR` | | FINRA OTCBB | United States | `USFNBB` | `XOTC` | | FINRA TRACE | United States | `USTRCE` | `FINR` | | Faroese Securities Market | Faroe Islands | `FOFSM` | `VMFX` | | Frankfurt Stock Exchange | Germany | `DEFSX` | `XFRA` | | Fukuoka Stock Exchange | Japan | `JPFSE` | `XFKA` | | Georgian Stock Exchange | Georgia | `GEGSE` | `XGSE` | | Ghana Stock Exchange | Ghana | `GHGSE` | `XGHA` | | Gibraltar Stock Exchange | Gibraltar | `GIGSX` | `GSXL` | | Guatemala National Stock Exchange | Guatemala | `GTGSE` | `XGTG` | | Guayaquil Stock Exchange | Ecuador | `ECGSE` | `XGUA` | | Guyana Stock Exchange | Guyana | `GYGSX` | `GSCI` | | Hamburg Stock Exchange | Germany | `DEHSE` | `XHAM` | | Hanoi Stock Exchange | Vietnam | `VNHSTC` | `HSTC` | | Hanover Stock Exchange | Germany | `DEHNSE` | `XHAN` | | Ho Chi Minh Stock Exchange | Vietnam | `VNSTC` | `XSTC` | | Hong Kong Shanghai Stock Connect - Southbound | Hong Kong | `HKHKSG` | `SHSC` | | Hong Kong Shenzhen Stock Connect - Southbound | Hong Kong | `HKHKSZ` | `SZSC` | | Hong Kong Stock Exchange | Hong Kong | `HKSEHK` | `XHKG` | | Indonesian Stock Exchange | Indonesia | `IDJSE` | `XIDX` | | Innex Stock Exchange | Ukraine | `UAINX` | `XDFB` | | Institutional Stock Exchange | Mexico | `MXBIVA` | `BIVA` | | Investors Exchange | United States | `USIEX` | `IEXG` | | Iran Fara Exchange | Iran | `IRIFX` | `IFBX` | | Iraq Stock Exchange | Iraq | `IQISX` | `XIQS` | | Istanbul Stock Exchange | Turkey | `TRIMKB` | `XIST` | | Jamaica Stock Exchange | Jamaica | `JMJSE` | `XJAM` | | Johannesburg Stock Exchange | South Africa | `ZAJSE` | `XJSE` | | Kazakhstan Stock Exchange | Kazakhstan | `KZKASE` | `XKAZ` | | Kiev International Stock Exchange | Ukraine | `UAKISE` | `XKIS` | | Korea New Exchange | South Korea | `KRKNX` | `XKON` | | Korea Stock Exchange | South Korea | `KRKSE` | `XKRX` | | Kuwait Stock Exchange | Kuwait | `KWKSE` | `XKUW` | | Kyrgyz Stock Exchange | Kyrgyz Republic | `KGKSE` | `XKSE` | | Labuan International Financial Exchange | Malaysia | `MYLFX` | `XLFX` | | Lao Securities Exchange | Laos | `LALSX` | `XLAO` | | Latin American Stock Exchange | Panama | `PABVP` | `XPTY` | | Libyan Stock Market | Libya | `LYLSM` | `XLSM` | | Ljubljana Stock Exchange | Slovenia | `SILSE` | `XLJU` | | London Stock Exchange | United Kingdom | `GBLSE` | `XLON` | | Lusaka Stock Exchange | Zambia | `ZMZSE` | `XLUS` | | Luxembourg Stock Exchange | Luxembourg | `LULSE` | `XLUX` | | MERJ Exchange | Seychelles | `SCSSE` | `TRPX` | | Macedonian Stock Exchange | North Macedonia | `MKMSE` | `XMAE` | | Madrid Stock Exchange | Spain | `ESMSE` | `XMAD` | | Malawi Stock Exchange | Malawi | `MWMSE` | `XMSW` | | Malaysian Exchange | Malaysia | `MYKLSE` | `XKLS` | | Maldives Stock Exchange | Maldives | `MVMAL` | `MALX` | | Malta Stock Exchange | Malta | `MTMSE` | `XMAL` | | Metropolitan Stock Exchange of India | India | `INMCX` | `MCXX` | | Moldova Stock Exchange | Moldova | `MDMSE` | `XMOL` | | Mongolian Stock Exchange | Mongolia | `MNSE` | `XULA` | | Montenegro Stock Exchange | Montenegro | `MEMSE` | `XMNX` | | Montevideo Stock Exchange | Uruguay | `UYBVM` | `XMNT` | | Moscow Exchange | Russia | `RUMICX` | `MISX` | | Mozambique Stock Exchange | Mozambique | `MZMSX` | `XBVM` | | Munich Stock Exchange | Germany | `DEMSE` | `XMUN` | | Muscat Securities Market | Oman | `OMMSM` | `XMUS` | | NYSE American | United States | `USAMEX` | `XASE` | | NYSE Arca | United States | `USPAC` | `ARCX` | | NYSE National | United States | `USCIN` | `XCIS` | | NZX | New Zealand | `NZSE` | `XNZE` | | Nagoya Stock Exchange | Japan | `JPNSE` | `XNGO` | | Nairobi Securities Exchange | Kenya | `KENSE` | `XNAI` | | Namibian Stock Exchange | Namibia | `NANSE` | `XNAM` | | Nasdaq | United States | `USNASD` | `XNAS` | | Nasdaq Copenhagen | Denmark | `DKCSE` | `XCSE` | | Nasdaq Dubai | United Arab Emirates | `AEDIFX` | `DIFX` | | Nasdaq Helsinki | Finland | `FIHSE` | `XHEL` | | Nasdaq Iceland | Iceland | `ISISE` | `XICE` | | Nasdaq Nordic Stockholm | Sweden | `SESSE` | `XSTO` | | Nasdaq Riga | Latvia | `LVRSE` | `XRIS` | | Nasdaq Tallinn | Estonia | `EETSE` | `XTAL` | | Nasdaq Vilnius | Lithuania | `LTNSE` | `XLIT` | | National Stock Exchange of Australia | Australia | `AUNSE` | `XNEC` | | National Stock Exchange of India | India | `INNSE` | `XNSE` | | Nepal Stock Exchange | Nepal | `NPNSE` | `XNEP` | | New York Stock Exchange | United States | `USNYSE` | `XNYS` | | Nicaragua Stock Exchange | Nicaragua | `NINSX` | `XMAN` | | Nigerian Stock Exchange | Nigeria | `NGLSE` | `XNSA` | | Nordic Derivatives Exchange | Sweden | `SENDX` | `XNDX` | | Nordic Growth Market | Sweden | `SENGM` | `XNGM` | | OTC Markets Group | United States | `USOTC` | `OTCM` | | Oslo Bors | Norway | `NOOB` | `XOSL` | | PFTS Stock Exchange | Ukraine | `UAPFTS` | `PFTS` | | Pakistan Stock Exchange | Pakistan | `PKKSE` | `XKAR` | | Palestine Exchange | Palestine | `PSAFM` | `XPAE` | | Papua New Guinea National Stock Exchange | Papua New Guinea | `PGNSE` | `XPOM` | | Perspektiva Stock Exchange | Ukraine | `UAPSX` | `SEPE` | | Philippine Stock Exchange | Philippines | `PHPSE` | `XPHS` | | Prague Stock Exchange | Czech Republic | `CZPSE` | `XPRA` | | Qatar Exchange | Qatar | `QADSM` | `DSMD` | | Quito Stock Exchange | Ecuador | `ECQSE` | `XQUI` | | Regional Stock Exchange of West Africa | Ivory Coast | `CIBVA` | `XBRV` | | Rosario Commerce Exchange | Argentina | `ARRCX` | `XROS` | | Russian Trading System | Russia | `RURTS` | `RTSX` | | Rwanda OTC | Rwanda | `RWOTC` | `ROTC` | | Rwanda Stock Exchange | Rwanda | `RWRSE` | `RSEX` | | SIM Venture Securities Exchange | Australia | `AUSIMV` | `SIMV` | | SIX Structured Products Exchange | Switzerland | `CHSCH` | `XQMH` | | SIX Swiss Exchange | Switzerland | `CHSSX` | `XSWX` | | Santiago Stock Exchange | Chile | `CLBCS` | `XSGO` | | Sapporo Securities Exchange | Japan | `JPSSE` | `XSAP` | | Sarajevo Stock Exchange | Bosnia and Herzegovina | `BASASE` | `XSSE` | | Saudi Stock Exchange | Saudi Arabia | `SARSE` | `XSAU` | | Shanghai Hong Kong Stock Connect - Northbound | China | `CNSGHK` | `XSSC` | | Shanghai Stock Exchange | China | `CNSGSE` | `XSHG` | | Shenzhen Hong Kong Stock Connect - Northbound | China | `CNSZHK` | `XSEC` | | Shenzhen Stock Exchange | China | `CNSSE` | `XSHE` | | Singapore Exchange | Singapore | `SGSSE` | `XSES` | | South Pacific Stock Exchange | Fiji | `FJSPSX` | `XSPS` | | Spanish Stock Exchange Interconnection System | Spain | `ESSIBE` | `XMCE` | | Spotlight Stock Market | Sweden | `SESSM` | `XSAT` | | Stock Exchange of Mauritius | Mauritius | `MUSEM` | `XMAU` | | Stock Exchange of Thailand | Thailand | `THSET` | `XBKK` | | Stuttgart Stock Exchange | Germany | `DESSE` | `XSTU` | | Sydney Stock Exchange | Australia | `AUSSX` | `APXL` | | TSX Venture Exchange | Canada | `CACVE` | `XTSX` | | Taipei Exchange | Taiwan | `TWOTC` | `ROCO` | | Taiwan Stock Exchange | Taiwan | `TWTSE` | `XTAI` | | Tashkent Republican Stock Exchange | Uzbekistan | `UZTRSX` | `XSTE` | | Tehran Stock Exchange | Iran | `IRTSE` | `XTEH` | | Tel-Aviv Stock Exchange | Israel | `ILTSE` | `XTAE` | | The International Stock Exchange | Guernsey | `GGCISX` | `XCIE` | | Tirana Stock Exchange | Albania | `ALTSE` | `XTIR` | | Tokyo Stock Exchange | Japan | `JPTSE` | `XJPX` | | Toronto Stock Exchange | Canada | `CATSE` | `XTSE` | | Trinidad & Tobago Stock Exchange | Trinidad and Tobago | `TTTTSE` | `XTRN` | | Tunis Stock Exchange | Tunisia | `TNTSE` | `XTUN` | | Uganda Securities Exchange | Uganda | `UGKSE` | `XUGA` | | Ukrainian Exchange | Ukraine | `UAUX` | `UKEX` | | Uruguay Electronic Exchange | Uruguay | `UYUEX` | `BVUR` | | Valencia Stock Exchange | Spain | `ESVSE` | `XVAL` | | Victoria Falls Stock Exchange | Zimbabwe | `ZWVFSE` | `VFEX` | | Vienna Stock Exchange | Austria | `ATVSE` | `XWBO` | | Warsaw Stock Exchange | Poland | `PLWSE` | `XWAR` | | Xetra | Germany | `DEXETR` | `XETR` | | Zagreb Stock Exchange | Croatia | `HRZSE` | `XZAG` | | Zimbabwe Stock Exchange | Zimbabwe | `ZWZSE` | `XZIM` | ### FRACTIONS | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `C` | Cash | | `D` | Round Down | | `F` | Fractions | | `U` | Round Up | ### FREQ | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `ANL` | Annual | | `BIM` | BiMonthly | | `DLY` | Daily | | `FNL` | Final | | `INT` | Interim | | `INTONMAT` | Interest on Maturity | | `IRG` | Irregular | | `ITM` | Interest on Maturity | | `MNT` | Monthly | | `QTR` | Quarterly | | `SMA` | Semi-Annual | | `TRM` | Trimesterly | | `UN` | Unspecified | | `WKL` | Weekly | ### GLOBSTATUS | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `A` | Active | | `D` | In default | | `I` | Inactive | ### LISTSOURCE | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `M` | Main WCA supported Listing | | `S` | Secondary Listing | ### LISTSTAT | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `D` | Delisted | | `G` | Reporting purposes only - Listed | | `H` | Reporting purposes only - Delisted | | `I` | Reporting purposes only - Suspended | | `L` | Listed | | `N` | New Listing | | `P` | Listing Pending | | `R` | Resumed | | `S` | Suspended | | `T` | Trading permitted - Listed | | `U` | Trading permitted - Delisted | | `V` | Trading permitted - Suspended | ### MANDVOLU | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `M` | Mandatory | | `V` | Voluntary | | `W` | Mandatory and/or Voluntary | ### MARKER | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `ANL` | Annual | | `ARR` | Arrears | | `CG` | Capital Gain | | `CGL` | Capital Gain Long Term | | `CGS` | Capital Gain Short Term | | `FNL` | Final | | `INS` | Installment | | `INT` | Interim | | `ISC` | Interest on Capital | | `MEM` | Memorial | | `QTR` | Quarterly | | `REG` | Regular | | `SMA` | Semi-annual | | `SPL` | Special | | `SUP` | Supplementary | | `TEI` | Tax Exempt Interest | | `TR` | Tax Return | | `UN` | Unspecified | | `VAR` | Variable | ### MICSEG | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `AIMX` | AIM | | `ALTX` | AltX | | `ALXA` | Euronext - Alternext Amsterdam | | `ALXB` | Euronext Growth | | `ALXL` | Euronext Growth | | `ALXP` | Euronext Growth | | `AQSL` | Main Board (Equity) | | `AQST` | Trading (Equity) | | `BERA` | Regulated Market | | `BERB` | Open Market | | `BERC` | Berlin Second Regulated Market | | `BETA` | BETa Market | | `BLBF` | Free Market | | `BSME` | SME | | `CAVE` | Caveat Emptor | | `DSME` | First North Growth Market | | `DUSA` | Regulated Market | | `DUSB` | Free Market | | `EBRA` | Multilateral Trading Facility | | `EMTF` | Euro MTF | | `ENAX` | Alternative Market | | `ENXL` | Euronext Access | | `EXGM` | Euronext Growth Milan | | `EXPM` | Expert Market | | `FNEE` | First North | | `FNIS` | First North | | `FNLT` | First North | | `FNLV` | First North | | `FRAA` | Regulated Market | | `FRAB` | Free Market | | `FRAS` | Free Market - Scale | | `FSME` | First North Growth Market | | `GBUL` | Beam | | `GROW` | BME MTF Equity - Growth | | `HAMA` | Regulated Market | | `HAMB` | Free Market | | `HANA` | Regulated Market | | `HANB` | Free Market | | `IFSM` | Institutional Financial Securities Market | | `JBUL` | MTF BSE International | | `MERK` | Euronext Growth | | `MESQ` | ACE Market | | `MIVX` | Euronext MIV Milan | | `MLXB` | Euronext Access | | `MOTX` | MOT | | `MTAA` | EURONEXT MILAN | | `MUNA` | Regulated Market | | `MUNB` | Free Market | | `NSME` | SME | | `OTCB` | OTCQB | | `OTCD` | OTCID Basic Market | | `OTCQ` | OTCQX | | `PINC` | Pink Current Information | | `PINI` | Pink No Information | | `PINL` | Pink Limited Information | | `PROS` | Prospects MTF | | `PSGM` | Grey Market | | `SCLE` | BME MTF Equity - Scaleup | | `SSME` | First North Growth Market | | `STUA` | Regulated Market | | `STUB` | Open Market | | `WBAH` | Official Market | | `WBDM` | Vienna MTF | | `XBIS` | International Securities Market | | `XCAN` | AeRO | | `XECM` | Emerging Companies Market | | `XESM` | Euronext Growth | | `XETA` | Regulated Market | | `XETB` | Free Market | | `XETS` | Free Market - Scale | | `XGEM` | Growth Enterprises Market | | `XGLO` | GlobalConnect | | `XHNX` | Unlisted Public Company Market | | `XJAS` | TOKYO STOCK EXCHANGE JASDAQ | | `XLJM` | SI Enter | | `XMAI` | Market for Alternative Investment | | `XMLI` | Euronext Access | | `XMOT` | ExtraMOT | | `XMSM` | Euronext Dublin | | `XNCM` | Capital Market | | `XNCO` | NewConnect | | `XNGS` | Global Select | | `XNMS` | Global Market | | `XOAM` | Nordic Alternative Bond Market | | `XOAS` | Euronext Expand | | `XSCA` | Catalist | | `XTAM` | Tokyo Pro Market | | `XTND` | Xtend | | `XTNX` | NEX | ### MKTSG | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `10` | OTCQX | | `100` | Alternative Securities Market | | `101` | First North | | `102` | Main List | | `103` | Secondary List | | `104` | Free Market | | `105` | Listed Main Market | | `106` | Listed Parallel Market | | `107` | Regulated Free Market | | `108` | Euronext Growth | | `109` | First Market | | `11` | OTCQB | | `110` | Second Market | | `111` | Third Market | | `112` | NXT Market | | `113` | Official Market | | `114` | Second Regulated Market | | `115` | Vienna MTF | | `116` | Free Market | | `117` | Prime Market | | `118` | Standard Market | | `119` | Start Market | | `12` | Pink Current Information | | `120` | Channel Islands | | `121` | Isle of Man | | `122` | Euronext Access | | `123` | Growth - Apex (Equity) | | `124` | Trading (Equity) | | `125` | Main Board (Equity) | | `126` | Prime Market | | `127` | Standard Market | | `128` | SI Enter | | `129` | LEAP Market | | `13` | Pink Limited Information | | `130` | Main Market | | `131` | Nomu - Parallel Market | | `132` | Tokyo Pro Market | | `133` | Premier Market | | `134` | Main Market | | `135` | Venture Capital Market | | `136` | Development Capital Market | | `137` | Black Economic Empowerment Segment | | `138` | Auction Market | | `139` | Regulated Market | | `14` | Pink No Information | | `140` | AeRO | | `141` | MTS International | | `142` | NEX | | `143` | Regulated Market | | `144` | Free Market | | `145` | Regulated Market | | `146` | Free Market | | `147` | Free Market | | `148` | Regulated Market | | `149` | Regulated Market - Prime Standard | | `150` | Regulated Market - General Standard | | `151` | Free Market - Basic Board | | `152` | Free Market - Scale | | `153` | Free Market - Basic Board | | `154` | Free Market - Scale | | `155` | Regulated Market - General Standard | | `156` | Regulated Market - Prime Standard | | `157` | Euronext Dublin | | `158` | Euronext Growth | | `159` | Atlantic Securities Market | | `16` | Grey Market | | `160` | Berlin Second Regulated Market | | `161` | Bonds Segment | | `162` | Exchange Traded Products Segment | | `163` | Government Securities Segment | | `164` | BaSE - Special Purpose Vehicles Segment | | `165` | Prime | | `166` | Standard | | `167` | T | | `168` | 1st Market | | `169` | 1st Secondary | | `17` | Caveat Emptor | | `170` | 2nd Market | | `171` | Main Board | | `172` | SME Board | | `173` | ChiNext | | `174` | Main Board | | `175` | Star Market | | `176` | Official Market | | `177` | Free Market | | `178` | Free Market - Issuers in Bankruptcy | | `179` | Official Market | | `18` | SME | | `180` | Official Market - Prime Market | | `181` | Official Market - Standard Market | | `182` | Official Market - Entry Market | | `183` | Free Market | | `184` | Official Listing - Super Listing | | `185` | Official Listing - Stock Exchange Listing | | `186` | Official Listing - Mandatory Listing | | `187` | Small Joint Stock Companies | | `188` | Free Market | | `189` | Euronext Milan | | `19` | SME | | `190` | Euronext STAR Milan | | `191` | Euronext MIV Milan | | `192` | Euronext Growth Milan | | `193` | Expert Market | | `194` | Main Market | | `195` | Standard Market | | `196` | Alternative Market | | `197` | MTS - Bond | | `198` | Regulated Market | | `199` | Free Market | | `20` | Euronext A | | `200` | Free Market - High Risk | | `201` | Free Market - SME Market | | `202` | Regulated Market | | `203` | Free Market | | `204` | Free Market - SME Market | | `205` | Tokyo Pro Bond Market | | `206` | First Market | | `207` | Second Market | | `208` | Third Market | | `209` | Yellow UTP Market | | `21` | Euronext B | | `210` | Orange UTP Market | | `211` | Red UTP Market | | `212` | SME Market | | `213` | Family & Friends | | `214` | Unlisted Public Company Market | | `215` | Euronext Growth Milan Professional | | `216` | Growth - Access (Equity) | | `217` | Regular Market | | `218` | Parallel Market | | `219` | Under Monitoring Market | | `22` | Euronext C | | `220` | Prime Market | | `221` | Standard Market | | `222` | Growth Market | | `223` | Domestic Main Board | | `224` | Foreign Main Board | | `225` | Domestic Venture Capital Board | | `226` | Foreign Venture Capital Board | | `227` | Tshipidi SME Board | | `228` | Serala OTC Board | | `229` | Prime Market | | `23` | Euronext Growth | | `230` | Official Market | | `231` | Regular Market | | `232` | Main List | | `233` | Secondary List | | `234` | First North | | `235` | Main List | | `236` | Secondary List | | `237` | First North | | `238` | Main List | | `239` | Secondary List | | `24` | Euronext Access | | `240` | First North | | `241` | BIST Star | | `242` | BIST Main | | `243` | BIST Submarket | | `244` | Watchlist Market | | `245` | Pre-market Trading Platform | | `246` | Main Market | | `247` | Innovation & Growth Market | | `248` | International Securities Market | | `249` | Main Market | | `25` | Euronext A | | `250` | Alternative Market | | `251` | Regulated Main Market | | `252` | Prospects MTF | | `253` | BdL Market | | `254` | Euro MTF | | `255` | LuxSE Securities Official List | | `256` | Small Capital Platform | | `257` | Main Market | | `258` | NewConnect | | `259` | Main Regulated | | `26` | Euronext B | | `260` | SME | | `261` | AIF | | `262` | Pre Market | | `263` | SME Main Market | | `264` | Worldcaps | | `265` | New Market | | `266` | Level 1 | | `267` | Level 2 | | `268` | Bovespa Plus | | `269` | Bovespa Plus Level 2 | | `27` | Euronext C | | `270` | OTC | | `271` | First North Premier Growth Market | | `272` | First North Premier Growth Market | | `274` | Institutional Financial Securities Market | | `275` | MTF BSE International | | `276` | Beam | | `277` | Regulated Market | | `278` | Open Market | | `279` | PepMarket | | `28` | Euronext Growth | | `280` | Premier Market | | `281` | Main Market | | `282` | Next Market | | `283` | Main Board | | `284` | Q Board | | `285` | Main Market | | `286` | Ambitious Market | | `287` | Main Board Premium | | `288` | Main Board Standard | | `289` | Alternative Board | | `29` | Euronext Access | | `290` | KASE Global | | `291` | Emerging Companies Market | | `292` | Growth Market | | `293` | Main Board | | `294` | Development Board | | `295` | Acceleration Board | | `296` | Main Board | | `297` | Small and Medium Enterprises Board | | `298` | Venture Capital Board | | `299` | Small Capital Platform | | `30` | Euronext A | | `300` | A Listing | | `301` | B Listing | | `302` | Main Market | | `303` | Venture Market | | `304` | New Economy Board | | `305` | Official Market | | `306` | Development & Enterprise Market (DEM) | | `307` | Main Market | | `308` | Growth Market | | `309` | First Market | | `31` | Euronext B | | `310` | Second Market | | `311` | First Tier Market | | `312` | Second Tier Market | | `313` | SME Market | | `314` | Main Board | | `315` | Small Cap | | `316` | Mezzanine Market | | `317` | Tier I | | `318` | Tier II | | `319` | Tier III | | `32` | Euronext C | | `320` | Main Board | | `321` | Diri Savi Board | | `322` | Empower Board | | `323` | Second Board | | `324` | Watch List | | `325` | SME Nile | | `326` | SME Tamayuz | | `327` | MOT | | `328` | ExtraMOT | | `329` | ExtraMOT Pro | | `330` | ExtraMOT Pro3 | | `331` | GlobalConnect | | `332` | Main Market | | `333` | Junior Market | | `334` | USD Market | | `335` | Official Prime Market | | `336` | Official Standard Market | | `337` | Free Market | | `338` | Main | | `339` | Belt and Road Market | | `34` | Euronext A | | `340` | Pre-IPO | | `341` | Regional Equity Market Segment | | `342` | Main Board | | `343` | SME Board | | `344` | Main Investment Market | | `345` | Enterprise Growth Market | | `346` | Main Market | | `347` | SME Market | | `348` | Main Market | | `349` | Alternate Market | | `35` | Euronext B | | `350` | Main Investment Market | | `351` | Alternative Investment Market | | `352` | Growth Enterprise Market | | `353` | Main Board | | `354` | GEM Board | | `355` | Pioneer Stock Board | | `356` | Main Market | | `357` | Alternative Market | | `358` | Taiwan Innovation Board | | `359` | Social Stock Exchange | | `36` | Euronext C | | `360` | Social Stock Exchange | | `361` | Xtend | | `362` | Official Market | | `363` | Second Market | | `364` | Third Market | | `365` | Spotlight Select | | `366` | Main Market | | `367` | Fukuoka Pro Market | | `368` | Multilateral Trading Facility | | `369` | OTCID Basic Market | | `37` | Euronext Growth | | `370` | BME MTF Equity - Growth | | `371` | BME MTF Equity - Scaleup | | `38` | Alternative Market | | `4` | First Section | | `40` | Main Market | | `45` | Main Board | | `46` | Growth Enterprises Market | | `47` | Premium Equities Segment | | `48` | Standard Equities Segment | | `49` | Special Purpose Vehicles Segment | | `5` | Second Section | | `50` | UCITs Segment | | `51` | Compensatory Instruments Segment | | `52` | Structured Products Segment | | `53` | BaSE - Equities Segment | | `54` | Category A | | `55` | Category B | | `56` | BETa Market | | `57` | Main Market | | `58` | ACE Market | | `59` | Euronext Expand | | `6` | Mothers | | `60` | Main Board | | `61` | Alternative Market | | `62` | Main Board | | `63` | Catalist | | `64` | Global Quote | | `65` | Market for Alternative Investment | | `66` | Main Board | | `67` | AltX | | `68` | Africa Board | | `69` | Main Market | | `7` | Global Select | | `70` | AIM | | `71` | Professional Securities Market | | `72` | Specialist Fund Market | | `73` | First North Growth Market | | `74` | JASDAQ | | `75` | First North Premier Growth Market | | `76` | SME-ITP | | `77` | SME-ITP | | `78` | Nordic Alternative Bond Market | | `79` | Surveillance Market | | `8` | Global Market | | `80` | Regulated Market | | `81` | Free Market | | `82` | Regulated Market | | `83` | Open Market | | `84` | First North Growth Market | | `85` | First North Growth Market | | `86` | Free Market - Subsegment 1 | | `87` | Free Market - Subsegment 2 | | `88` | Free Market - Subsegment 3 | | `89` | Emerging Stock Board | | `9` | Capital Market | | `90` | Prime Listing | | `91` | Standard Listing | | `92` | Open Market | | `93` | MTP | | `94` | A-List | | `95` | B-List | | `96` | Official List | | `97` | Secondary List | | `98` | Main Board | | `99` | Premium Board | ### MRGRSTAT | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `C` | Complete | | `O` | On-going | | `X` | No value | ### ONOFFMKT | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `F` | Off | | `N` | On | ### OUTTURNSTYLE | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `ADEX` | Additional for Existing Securities | | `NEWO` | New for Old Securities | ### PARTFINAL | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `F` | Final | | `P` | Part | ### PAYTYPE | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `B` | Cash&Stock | | `C` | Cash | | `D` | Dissenters Rights | | `S` | Stock | | `T` | ToBeAnnounced | ### REGS144A | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `144A` | 144A | | `REGD` | Regulation D | | `REGS` | Regulation S | ### SECTYPE | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `BND` | Bond | | `BSW` | Basket Warrant | | `CDA` | Share Depository Certificate | | `CDI` | Chess Depository Interest | | `CN` | Convertible Notes | | `CVR` | Contingent Value Rights | | `DR` | Depository Receipts | | `DRT` | Distribution Rights | | `DST` | Deferred Settlement Trading | | `EQS` | Equity Shares | | `ETC` | Exchange Traded Commodities | | `ETF` | Exchange Traded Fund | | `LOA` | Letter of Allotment | | `MF` | Mutual Fund | | `PFS` | Preferred Security | | `PRF` | Preference Share | | `PRL` | Parallel Line | | `RCP` | Receipt | | `RDR` | Redemption Rights | | `RDS` | Redeemable Shares | | `SP` | Structured Product | | `SRT` | Subscription Receipts | | `STL` | Second Trading Line | | `STP` | Stapled Security | | `TRT` | Tradeable Rights | | `TSS` | Tendered Shares Security | | `UNT` | Units | | `WAR` | Warrants | | `WD` | When Distributed | | `WIS` | When Issued | ### TKOVRSTAT | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `C` | Complete | | `L` | Lapsed | | `O` | On-going | ### UTTYP | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `O` | Open Ended Investment Companies (OEIC) | | `T` | Unit Investment Trust | | `U` | Unit Trust | ### VOTING | Value [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `L` | Limited Voting | | `M` | Multiple Voting | | `N` | No Voting | | `V` | Voting | --- Source: https://databento.com/docs/conventions/corporate-actions-events # Corporate actions events > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ### Company Meeting (`AGM`) Annual General meeting of shareholders is held once a year, usually after the end of the financial year. Also includes other meeting types such as Extraordinary General Meeting, Special General Meeting, Court Ordered Meeting, and Postal Ballot. - Event level: **Issuer** - Event category: **Proposals** - Event participation: **Voluntary** - Values for `event_subtype`: - `AGM`: Annual General Meeting - `BHM`: Bond Holder Meeting - `CGM`: Court Ordered General Meeting - `EGM`: Extraordinary General Meeting - `GM`: General Meeting - `SGM`: Special General Meeting #### **Calendar Dates** - `event_date` (equal to `meeting_date`) - `effective_date` (equal to `meeting_date`) - `record_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `meeting_number` | Company Meeting Number | | `event_info` | `meeting_time` | Local time for Start of Meeting | | `event_info` | `address_line1` | Address line 1 | | `event_info` | `address_line2` | Address line 2 | | `event_info` | `address_line3` | Address line 3 | | `event_info` | `address_line4` | Address line 4 | | `event_info` | `address_line5` | Address line 5 | | `event_info` | `address_line6` | Address line 6 | | `event_info` | `city` | City | | `event_info` | `country` | Country Code - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | ### Announcement (`ANN`) Intent of the company to have Corporate Action which may affect the shareholder's value. The intent may or may not materialise into an event. - Event level: **Issuer** - Event category: **Proposals** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `notification_date`) - `notification_date` ### Arrangement (`ARR`) A general term referring to complex reorganisation of a company or a group of companies and its capital. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory / Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` - `exp_completion_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | ### Assimilation (`ASSM`) A general term referring to complex reorganisation of a company or a group of companies and its capital. - Event level: **Listing** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `assimilation_date`) - `effective_date` (equal to `assimilation_date`) - `payment_date` ### Buy Back (`BB`) Buyback events can also be used to transmit Dutch Auction events. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Voluntary** - Values for `event_subtype`: - `DUTCHAUCT`: Dutch Auction - `null`: Generic event, no subtype provided #### **Calendar Dates** - `event_date` (equal to `end_date`) - `effective_date` (equal to `start_date`) - `record_date` - `payment_date` - `to_date` - `registration_date` - `start_date` - `end_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `event_info` | `buy_back_min_percentage` | Minimum percentage of the shares outstanding the company will buy back from shareholders. | | `event_info` | `buy_back_max_percentage` | Maximum percentage of the shares outstanding the company will buy back from shareholders. | | `event_info` | `on_off_flag` | The mechanism adopted by the company for buying back the shares. Default blank. N - On Market i.e. via the exchange. F - Off Market i.e. some other mechanism other than the exchange. (See **[ONOFFMKT](https://databento.com/docs/conventions/reference-data-enums.md#onoffmkt)** enum) | ### BBG Composite Change (`BBCC`) To allow change in Bloomberg's ticker or the mapping of BBG ID to a Security ID. - Event level: **Country** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_bbg_comp_id` | Old Bloomberg Composite ID | | `event_info` | `new_bbg_comp_id` | New Bloomberg Composite ID | | `event_info` | `old_bbg_comp_ticker` | Old Bloomberg Composite Ticker | | `event_info` | `new_bbg_comp_ticker` | New Bloomberg Composite Ticker | | `event_info` | `old_currency` | Old Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `new_currency` | New Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `old_country` | Old Country Code - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | | `event_info` | `new_country` | New Country Code - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | ### BBG Figi Change (`BBEC`) To allow change in Bloomberg's ticker or the mapping of BBG ID to a ISO 4217 exchange code. - Event level: **Listing** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_figi` | Old Bloomberg Exchange ID - Old Figi | | `event_info` | `new_figi` | New Bloomberg Exchange ID - New Figi | | `event_info` | `old_figi_ticker` | Old Bloomberg Exchange Ticker - Old Figi Ticker | | `event_info` | `new_figi_ticker` | New Bloomberg Exchange Ticker - New Figi Ticker | | `event_info` | `old_currency` | Old Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `new_currency` | New Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `old_exchange` | Old Exchange Code - ISO 4217 (See **[EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange)** enum) | | `event_info` | `new_exchange` | New Exchange Code - ISO 4217 (See **[EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange)** enum) | ### Bankruptcy (`BKRP`) When a company's net worth goes below the country's stipulated guideline. There could be several stages in the Bankruptcy process. Hence, one sees several announcements over a period of time. The end result of Bankruptcy could be revival, sale, re-organisation, liquidation etc. - Event level: **Issuer** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `filing_date`) - `effective_date` (equal to `filing_date`) - `notification_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `date_info` | `emergence_date` | Emergence from Bankruptcy Date currently used in US only. | ### Bonus (`BON`) Issue of FREE securities to the shareholders of the Company by capitalising the reserves of the company. Security being issued can be an instrument other than Equity Share. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory** - Values for `event_subtype`: - `DPRCPDIV`: Depository Receipt Dividend #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `lapsed_premium` | If the underlying security has a Bonus issue and company decides ADR shares will not have bonus. The company will sell the share on behalf of ADR holders in the market and return the money to ADR holders. So there is no ratio. | ### Bonus Rights (`BR`) A Bonus issue with tradable rights attached. Effectively, a mixture of Bonus and Rights. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory / Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `start_subscription_date` - `end_subscription_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `issue_price` | Issue Price | | `date_info` | `starttrade_date` | Start Trade Date | | `date_info` | `endtrade_date` | End Trade Date | | `date_info` | `split_date` | Split Date | | `event_info` | `sub_exp_time` | For a specified date, the time at which the event is set to expire. | | `event_info` | `sub_exp_time_zone` | Time Zone of the Expiry Time. | | `event_info` | `over_subscription` | A flag to indicate if the shareholders have the option of oversubscribing for the issue. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `trading_security_id` | Security ID of the security that may be created for Trading purposes. (Join with the `security_id` field) | | `event_info` | `trading_isin` | Trading ISIN if applicable/available | ### Call (`CALL`) Where the issue price of a security in an offer is split in such a way that the shareholders have to pay the issue price in installments. Until the full issue price is received by the company the security remains as Partly Paid. A call is a demand to pay an installment. - Event level: **Security** - Event category: **Distribution Debit** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `due_date`) - `effective_date` (equal to `due_date`) - `record_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `to_face_value` | To Face Value | | `rate_info` | `to_premium` | To Premium | | `event_info` | `call_number` | Number of the call which the company has made on the security. For example: 1st Call, 2nd Call, 3rd Call | ### Capital Reduction (`CAPRD`) Write off of existing Equity Capital of a company by either reducing the par value and/or by cancelling a certain number of shares or capital. If a cash payment is involved then it is described as Return of Capital. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` - `record_date` - `payment_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `par_value_old` | Old Par Value | | `rate_info` | `par_value_new` | New Par Value | ### Class Action (`CLSAC`) Holders of security(ies) of a company combine to file a suit for damages from the company. - Event level: **Security** - Event category: **Legal Action** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` - `effective_date` - `start_date` - `end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `settlement_amount` | Settlement Amount | | `date_info` | `filing_date` | Filing Date | | `date_info` | `lead_plaintiff_deadline_date` | Lead Plaintiff Deadline Date | | `date_info` | `proof_filing_deadline_date` | Proof Filing Deadline Date | | `date_info` | `claim_filing_deadline_date` | Claims Filing Deadline Date | | `date_info` | `objection_deadline_date` | Objection Deadline Date | | `date_info` | `exclusion_deadline_date` | Exclusion Deadline Date | | `event_info` | `court_id` | Unique ID of the Court with which the complaint has been filed | | `event_info` | `case_number` | Court assigned number of the case | | `event_info` | `judge` | Judge's name | | `event_info` | `case_name` | Name of the Case | | `event_info` | `allegations` | Reasons for which the claimants are initiating a class action. For example: Fraud, Misrepresentation, non-disclosure, insider trading etc. (See **[ALLEGATION](https://databento.com/docs/conventions/reference-data-enums.md#allegation)** enum) | | `event_info` | `eligible_securities` | Securities eligible to participate in the action | | `event_info` | `investor_class` | Investor class that can participate in the action (See **[INVESTCLS](https://databento.com/docs/conventions/reference-data-enums.md#investcls)** enum) | | `event_info` | `on_behalf_of` | The Class Action is on behalf of buyer or seller (See **[ONBEHALF](https://databento.com/docs/conventions/reference-data-enums.md#onbehalf)** enum) | | `event_info` | `lead_plaintiff` | Name of the Lead Plaintiff | | `event_info` | `case_status` | Current Status of the case (See **[ONBEHALF](https://databento.com/docs/conventions/reference-data-enums.md#onbehalf)** enum) | ### Consolidation (`CONSD`) Increase in the Face/Nominal Value of the security. Company's Equity Capital remains the same but the shares in issue reduces. Opposite of Sub-Division. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - Values for `event_subtype`: - `CONSD`: Reverse Split, currently only used in US events #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `par_value_old` | Old Par Value | | `rate_info` | `par_value_new` | New Par Value | ### Conversion (`CONV`) Details of individual Conversions of a convertible security (For example: Preference Share, Convertible Bond). Also sometimes used to transmit Voluntary Offers. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory / Voluntary** - Values for `event_subtype`: - `BB`: Buyback - `CALL`: Call Option Exercised - `CLAIMSET`: Claim Settled - `DEFPY`: Default Payment - `DR`: Drawings - `DRL`: Drawings by lottery - `ECONV`: Early Conversion - `MAT`: Maturity - `ORD`: Ordinary - `PUT`: Put Option Exercised - `REDEMCLAIM`: Redemption Claim - `WRTDN`: Write Down - `null`: Generic event, no subtype provided #### **Calendar Dates** - `event_date` (equal to `from_date`) - `event_date` (equal to `to_date`) - `effective_date` - `record_date` - `from_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `price` | Price | | `rate_info` | `price_as_percent` | Price as Percent | | `date_info` | `settlement_date` | Settlement Date | | `event_info` | `part_final` | Indicates if the conversion is Part or Final. Part conversions may be mandatory. (See **[PARTFINAL](https://databento.com/docs/conventions/reference-data-enums.md#partfinal)** enum) | | `event_info` | `amount_converted` | Total value of the Bonds that were converted. | | `event_info` | `cur_pair` | Identifies the different currencies involved for the FX rate to be applied - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `fxrate` | Foreign Exchange Rate | ### Certificate Exchange (`CTX`) Physical exchange of existing certificates for another set of certificates. This may be done in case of sub-division, lot change, name change, exchange of one type of security for another type of security. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `end_date`) - `start_date` - `end_date` ### Currency Redenomination (`CURRD`) Change in Security's Capital Currency For example: French Franc to Euro. This may also result in a Par Value change. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `par_value_old` | Old Par Value | | `rate_info` | `par_value_new` | New Par Value | | `event_info` | `old_currency` | Old Par Value Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `new_currency` | New Par Value Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | ### Distribution (`DIST`) Issue of **free** shares of Company B to the shareholders of Company A. There is no transfer of operations/assets of Company A to Company B. The shareholder continues to hold shares in both Company A and Company B. Hence, the event is distinct from Demerger or Bonus. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory** - Values for `event_subtype`: - `DIST`: Spin-Off, currently only used in US events #### **Calendar Dates** - `event_date` - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | ### Dividend (`DIV`) Distribution of a company's profits for a given period to the shareholders in Cash and/or Stock. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory** - Values for `event_subtype`: - `CAPDIST`: Capital Distribution - `CAPGAIN`: Capital Gain - `DIV`: Forward Split, currently only used in US events - `INTDIV`: Derived from Interest Payment - `null`: Generic event, no subtype provided #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `gross_dividend` | Gross Dividend | | `rate_info` | `net_dividend` | Net Dividend | | `date_info` | `declaration_date` | Date Dividend announced by Issuer | | `date_info` | `periodend_date` | Period End Date | | `date_info` | `foreign_ex_date` | The Date on which the USD Rate to Currency value was taken. | | `date_info` | `ex_date2` | This is the Ex Date value for Stock component of a Dividend with a Paytype of (B)oth Stock and Cash. The main Ex Date field will be for the cash payment portion of the same record. | | `date_info` | `pay_date2` | This is the Payment Date value for Stock component of a Dividend with a Paytype of (B)oth Stock and Cash. The main Payment Date field will be for the cash payment portion of the same record. | | `event_info` | `marker` | Dividend Marker (See **[MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker)** enum) | | `event_info` | `frequency` | Dividend Frequency (See **[FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq)** enum) | | `event_info` | `tbaflag` | The company intends to pay a dividend but the details are yet to be announced. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `nil_dividend` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `div_rescind` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `declared_currency` | Declared Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `declared_gross_amount` | Declared Gross Amount | | `event_info` | `unused` | Unused | | `event_info` | `approx_flag` | If dividend is temporarily an approximate value. Expect an update. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `tax_rate` | Domestic Withholding Tax Rate | | `event_info` | `coupon` | In countries where coupons are to be submitted to obtain a benefit this field is useful. | | `event_info` | `coupon_id` | Coupon ID | | `event_info` | `div_rate` | Rate of Dividends as a percentage of Par Value. | | `event_info` | `cur_pair` | Identifies the different currencies involved for the FX rate to be applied - ISO 4217 | | `event_info` | `dep_fees` | Depositary Fees - Applies only to DR dividends | | `event_info` | `dep_fees_currency` | Depositary Fees Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `recind_cash_div` | Cash Dividend Cancelled. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `recind_stock_div` | Stock Dividend Cancelled. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `usd_rate_to_currency` | USD Exchange Rate for Dividend Currency | | `event_info` | `div_postponed` | Indicating Postponement of Dividend Payment (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | ### Dividend Investment Fund (`DIVIF`) Distribution of a company's profits for a given period to the shareholders in Cash and/or Stock, Interest from interest bearing securities and cash and, where applicable, rental income from property funds. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory** - Values for `event_subtype`: - `DIVACC`: Dividend Accumulation - `DIVINC`: Dividend Income - `INT`: Interest Basis Unknown - `INTACC`: Interest Accumulation - `INTINC`: Interest Income - `PRO`: Property Basis Unknown - `PROACC`: Property Accumulation - `PROINC`: Property Income #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `group1_gross_div` | Group 1 Gross Dividend | | `rate_info` | `group1_net_div` | Group 1 Net Dividend | | `date_info` | `declaration_date` | Date Dividend announced by Issuer | | `date_info` | `periodend_date` | Period End Date | | `event_info` | `marker` | Dividend Marker (See **[MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker)** enum) | | `event_info` | `frequency` | Dividend Frequency (See **[FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq)** enum) | | `event_info` | `tbaflag` | The company intends to pay a dividend but the details are yet to be announced. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `nil_dividend` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `div_rescind` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `div_postponed` | Indicating Postponement of Dividend Payment (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `tax_rate` | | | `event_info` | `group2_gross_div` | Gross up of Group 2 Net distribution. Group 2 is Non pari passu dividend. | | `event_info` | `group2_net_div` | | | `event_info` | `equalisation` | Amount attributable to net realised income that is included in the price of units created or units cancelled by the unit trust fund during the reporting period. | | `event_info` | `franked_percentage` | Indicates the portion of the distribution that is franked expressed as a percentage. | | `event_info` | `franked_value` | Indicates the portion of the distribution that is franked expressed as a value. | | `event_info` | `un_fr_non_foreign_p_c_t` | Indicates the portion of the the non-foreign distribution that is unfranked expressed as a percentage. | | `event_info` | `un_fr_non_foreign_value` | Indicates the portion of the non-foreign distribution that is unfranked expressed as a value. | | `event_info` | `un_fr_foreign_p_c_t` | Indicates the portion of the the foreign distribution that is unfranked expressed as a percentage. | | `event_info` | `un_fr_foreign_value` | Indicates the portion of the foreign distribution that is unfranked expressed as a value. | ### Div Reclassification (`DIVRC`) - Event level: **Country** - Event category: **Tax Related** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `gross_dividend` | Gross Dividend | | `rate_info` | `net_dividend` | Net Dividend | | `date_info` | `declaration_date` | Date Dividend announce. by Issuer | | `date_info` | `periodend_date` | Period End Date | | `date_info` | `tax_yearend_date` | Tax Year End Date | | `event_info` | `div_alloc_tax_year` | Dividend Allocable to Tax Year | | `event_info` | `income_divs` | Income Dividends | | `event_info` | `qual_income_divs` | Qualified Income Dividends | | `event_info` | `short_term_cap_gains` | Short Term Capital Gains | | `event_info` | `long_term_cap_gains` | Long Term Capital Gains | | `event_info` | `total_cap_gains` | Total Capital Gains | | `event_info` | `qual_short_term_cap_gains` | Qualified Short Term Capital Gains | | `event_info` | `qual_long_term_cap_gains` | Qualified Long Term Capital Gains | | `event_info` | `qual_total_cap_gains` | Qualified Total Capital Gains | | `event_info` | `foreign_tax_paid` | Foreign Tax Paid | | `event_info` | `qual_foreign_tax_paid` | Qualified Foreign Tax Paid | | `event_info` | `section1202_gain` | Section 1202 Gain | | `event_info` | `collectables_28_percent_gain` | Collectibles(28%) Gain | | `event_info` | `cash_liq_dist` | Cash Liquidation Distribution | | `event_info` | `noncash_liq_dist` | Noncash Liquidation Distribution | | `event_info` | `exempt_interest_divs` | Exempt Interest Dividends | | `event_info` | `percentage_of_amount` | Percentage of AMT | | `event_info` | `return_of_capital` | For US. Portion of Dividend that is a Return of Capital distribution | | `event_info` | `under1250_section` | For US. Portion of Dividend that is a 1250 section distribution | | `event_info` | `section199a_inc_div` | Section 199A Income Dividend | | `event_info` | `section199a_short_term_cap_gain` | Section 199A STCG | | `event_info` | `section199a_foreign_tax_paid` | Section 199A Foreign Tax Paid | | `event_info` | `nra_exempt_inc_div` | NRA Exempt Income Dividend | ### Demerger (`DMRGR`) Separation of a portion of company A's operations/assets into Company B. Shareholders of Company A receive free shares of Company B. (AKA Spin-off) - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - Values for `event_subtype`: - `DMRGR`: Spin-Off, currently only used in US events #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `effective_date` - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` - `exp_completion_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | ### DR Change (`DRCHG`) Changes in the ratio or underlying security of the Depository Receipt. - Event level: **Security** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_deposit_receipt_ratio` | Number of old DR's held in DR - UN Ratio | | `event_info` | `new_deposit_receipt_ratio` | Number of new DR's held in DR - UN Ratio | | `event_info` | `old_unit_ratio` | Number of old Underlying held in DR - UN Ratio | | `event_info` | `new_unit_ratio` | Number of new Underlying held in DR - UN Ratio | | `event_info` | `old_unit_security_id` | Internal Security ID for the Old Underlying Security (Join with the `security_id` field) | | `event_info` | `new_unit_security_id` | Internal Security ID for the New Underlying Security (Join with the `security_id` field) | | `event_info` | `old_depbank` | Old Depositary Bank | | `event_info` | `new_depbank` | New Depositary Bank | | `event_info` | `old_deposit_receipttype` | Old DR Security Type (See **[DRTYPE](https://databento.com/docs/conventions/reference-data-enums.md#drtype)** enum) | | `event_info` | `new_deposit_receipttype` | New DR Security Type (See **[DRTYPE](https://databento.com/docs/conventions/reference-data-enums.md#drtype)** enum) | | `event_info` | `old_level` | Old Level | | `event_info` | `new_level` | New Level | ### Dividend Reinvestment Plan (`DRIP`) - Event level: **Country** - Event category: **Distribution** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `drip_reinv_price` | Reinvestment Price | | `date_info` | `declaration_date` | Date Dividend announced by Issuer | | `date_info` | `periodend_date` | Period End Date | | `date_info` | `drip_last_date` | Last Date to opt for DRIP | | `date_info` | `drip_pay_date` | DRIP Payment Date | | `date_info` | `crest_date` | Crest Settlement Date | | `event_info` | `marker` | Dividend Marker (See **[MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker)** enum) | | `event_info` | `frequency` | Dividend Frequency (See **[FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq)** enum) | | `event_info` | `tbaflag` | The company intends to pay a dividend but the details are yet to be announced. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `nil_dividend` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `div_rescind` | Same as a Cancelled Dividend (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `declared_currency` | Declared Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `declared_gross_amount` | Declared Gross Amount | | `event_info` | `stamp_duty` | Stamp Duty | | `event_info` | `stamp_as_percent` | Stamp Duty as Percent | | `event_info` | `com_fee` | Commission Fee | | `event_info` | `com_as_percent` | Commission Fee as Percent | | `event_info` | `div_postponed` | Indicating Postponement of Dividend Payment (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | ### Divestment (`DVST`) One or more Shareholders of a company offering whole or part of its holding to the existing shareholders of company at a price. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `start_subscription_date` - `end_subscription_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `event_info` | `trading_security_id` | Security ID of the security that may be created for trading purposes. (Join with the `security_id` field) | | `event_info` | `trading_isin` | Trading ISIN if applicable/available | ### Entitlement (`ENT`) Non-renounceable & non-tradable issue of shares. Is the right to purchase new shares at the specific price. The allotment of shares will usually be in the same company. - Event level: **Security** - Event category: **Distribution** - Event participation: **Voluntary** - Values for `event_subtype`: - `NRENRTS`: Non Renounceable Rights - `OPOFF`: Open Offer - `POFF`: Priority Offer - `SPP`: Share Purchase Plan #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `event_date` (equal to `end_subscription_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `start_subscription_date` - `end_subscription_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `ent_issue_price` | Entitlement Issue Price | | `event_info` | `over_subscription` | A flag to indicate if the shareholders have the option of oversubscribing for the issue. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | ### Franking (`FRANK`) - Event level: **Country** - Event category: **Tax Related** - Event participation: **Mandatory** - Values for `event_subtype`: - `F`: Fully franked - `N`: Not known - `P`: Partially franked - `U`: Unfranked #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `to_date` - `registration_date` - `option_election_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `franked_div` | Franked Dividend Amount | | `rate_info` | `unfranked_div` | Unfranked Dividend Amount | | `date_info` | `declaration_date` | Date Dividend announced by Issuer | | `date_info` | `periodend_date` | Period End Date | | `date_info` | `foreign_ex_date` | Date USD Rate to Currency value was recorded | | `event_info` | `marker` | Dividend Marker (See **[MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker)** enum) | | `event_info` | `frequency` | Dividend Frequency (See **[FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq)** enum) | | `event_info` | `tbaflag` | The company intends to pay a dividend but the details are yet to be announced. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `nil_dividend` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `div_rescind` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `declared_currency` | Declared Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `declared_gross_amount` | Declared Gross Amount | | `event_info` | `div_postponed` | Indicating Postponement of Dividend Payment (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `conduit_foreign_income` | Conduit Foreign Income | | `event_info` | `franking_credit` | Corporate Tax Rate for Franking Credit | ### Forward split (`FSPLT`) A bonus, dividend, or subdivision event. Currently only used for US securities. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory** - Values for `event_subtype`: - `BON`: Bonus - `DIV`: Dividend - `SD`: Subdivision #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `lapsed_premium` | If the underlying security has a Bonus issue and company decides ADR shares will not have bonus. The company will sell the share on behalf of ADR holders in the market and return the money to ADR holders. So there is no ratio. | ### Financial Transaction Tax (`FTT`) As per EU directive, list of securities in each country on which FTT would be leviable. - Event level: **Country** - Event category: **Tax Related** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `end_date`) - `start_date` - `end_date` ### Fin. Year Change (`FYCHG`) Change in accounting year-end of the company - Event level: **Issuer** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `notification_date`) - `notification_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `date_info` | `old_fin_year_start_date` | Old Financial Year Start Date | | `date_info` | `old_fin_year_end_date` | Old Financial Year End Date | | `date_info` | `new_fin_year_start_date` | New Financial Year Start Date | | `date_info` | `new_fin_year_end_date` | New Financial Year End Date | ### Internat. Code Change (`ICC`) A change in any of the international codes in the system. For a security the International codes will be common across countries. - Event level: **Security** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_i_s_i_n` | Old Isin | | `event_info` | `new_i_s_i_n` | New Isin | | `event_info` | `old_us_code` | Old US code | | `event_info` | `new_us_code` | New US code | ### Incorporation Change (`INCHG`) Change in the Country of Incorporation of the Issuer. - Event level: **Issuer** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `incorporation_change_date`) - `effective_date` (equal to `incorporation_change_date`) #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_country` | Old ISO Country - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | | `event_info` | `new_country` | New ISO Country - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | ### Issuer Name Change (`ISCHG`) Change in the name of the Issuer. - Event level: **Issuer** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `name_change_date`) - `effective_date` (equal to `name_change_date`) #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `issuer_old_name` | Old Issuer Name | | `event_info` | `issuer_new_name` | New Issuer Name | | `event_info` | `legal_name` | Flagged if New Issuer Name is the legal name (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `meeting_date_flag` | Meeting Date Flag (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | ### Local Code Change (`LCC`) A change to the Localcode/Symbol/Ticker of the Security. - Event level: **Listing** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_localcode` | Old Local Code | | `event_info` | `new_localcode` | New Local Code | ### Liquidation (`LIQ`) A process by which a bankrupt company is sold off as per the Liquidators' guidelines. The shareholders of the company may receive payment from the liquidation proceeds. - Event level: **Issuer** - Event category: **Distribution** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `record_date`) - `record_date` - `payment_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `event_info` | `liquidator` | Name of the agency appointed for liquidating the assets of the company. | | `event_info` | `liq_add1` | Address Line 1 | | `event_info` | `liq_add2` | Address Line 2 | | `event_info` | `liq_add3` | Address Line 3 | | `event_info` | `liq_add4` | Address Line 4 | | `event_info` | `liq_add5` | Address Line 5 | | `event_info` | `liq_add6` | Address Line 6 | | `event_info` | `liq_city` | City | | `event_info` | `liq_country` | Country code - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | | `event_info` | `liq_tel` | Phone | | `event_info` | `liq_fax` | Fax | | `event_info` | `liq_email` | Email | ### List Status Change (`LSTAT`) Change in the Listing Status of a security, namely Suspension, Delisting or Resumption of trading. All securities are assumed to be traded unless the listing status is changed. - Event level: **Listing** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_lotsize` | Old Lot Size | | `event_info` | `new_lotsize` | New Lot Size | | `event_info` | `old_min_trading_qty` | Old Minimum Trading Quantity | | `event_info` | `new_min_trading_qty` | New Minimum Trading Quantity | ### Mutual Fund Conv. (`MFCON`) - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_u_t_type` | Old Unit Trust Type (See **[UTTYP](https://databento.com/docs/conventions/reference-data-enums.md#uttyp)** enum) | | `event_info` | `new_u_t_type` | New Unit Trust Type (See **[UTTYP](https://databento.com/docs/conventions/reference-data-enums.md#uttyp)** enum) | ### Market Seg Change (`MKCHG`) Captures the movement of security from one segment to another within an Exchange. - Event level: **Listing** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_market_segment_id` | Old Market Segment ID (See **[MKTSG](https://databento.com/docs/conventions/reference-data-enums.md#mktsg)** enum) | | `event_info` | `new_market_segment_id` | New Market Segment ID (See **[MKTSG](https://databento.com/docs/conventions/reference-data-enums.md#mktsg)** enum) | ### Merger (`MRGR`) Coming together of two or more companies to form a new company. Shareholders of all the old companies have to exchange their holding for shares of the new company. The original companies cease to exist. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` - `exp_completion_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `date_info` | `appointed_date` | Appointed Date | | `event_info` | `mrgr_status` | Progress of the Merger (See **[MRGRSTAT](https://databento.com/docs/conventions/reference-data-enums.md#mrgrstat)** enum) | ### New Listing (`NLIST`) A new listing is when a security lists on an exchange for the first time. - Event level: **Listing** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` - `effective_date` ### Odd Lot Offer (`ODDLT`) Offer by the Issuer to the shareholder to change their odd lot holdings into round lots. It may entail sale of odd lot shares to the company or a Buy-in of balance amount by the shareholder. - Event level: **Listing** - Event category: **Reorganisation** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `end_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `start_date` - `end_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `event_info` | `buy_in_currency` | Buy in Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `buy_in_price` | The price at which the Shareholder can opt to buy shares from the company to change his odd lot shareholding into round lot. | | `event_info` | `buy_in_flag` | Indicates if a Buy-in is available. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | ### Property Income Dividend (`PID`) - Event level: **Country** - Event category: **Tax Related** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `pid` | Property Income Dividend Portion | | `rate_info` | `non_pid` | Non Property Income Dividend Portion | | `date_info` | `declaration_date` | Date Dividend Announced by Issuer | | `date_info` | `periodend_date` | Period End Date | | `event_info` | `marker` | Dividend Marker (See **[MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker)** enum) | | `event_info` | `frequency` | Dividend Frequency (See **[FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq)** enum) | | `event_info` | `tbaflag` | The company intends to pay a dividend but the details are yet to be announced. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `nil_dividend` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `div_rescind` | Real Dividend event but of zero payrate. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `declared_currency` | Declared Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `declared_gross_amount` | Declared Gross Amount | | `event_info` | `div_postponed` | Indicating Postponement of Dividend Payment. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | ### Purchase Offer (`PO`) Company A has managerial control of Company B. Company A decides to increase its stake in Company B by offering to buy shares from the shareholders of the Company B. Company B will continue to exist. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `close_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `open_date` - `close_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `event_info` | `negotiated_price` | Price at which the purchasing company struck a deal for a large chunk of the free holding, | | `event_info` | `sealed_bid` | True if Sealed Bid (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `min_percent` | TOTAL Percentage of the shares outstanding that the purchaser will buy from the shareholders. | | `event_info` | `max_percent` | TOTAL Percentage of the shares outstanding that the purchaser will buy from the shareholders. | ### Primary Exchange Chg (`PRCHG`) Change in the Primary Exchange of a security. - Event level: **Security** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_exchange` | Old Primary Exchange Code (See **[EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange)** enum) | | `event_info` | `new_exchange` | New Primary Exchange Code (See **[EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange)** enum) | ### Preferential Offer (`PRF`) Offer to the shareholders of Company A to buy securities of Company B on a preferential basis. This offer is a reserved portion of a public offer of Company B. Both companies are normally part of the same group. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `end_subscription_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `start_subscription_date` - `end_subscription_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `event_info` | `offeree_issuer_id` | Offeree Issuer ID (Join with the `issuer_id` field) | | `event_info` | `offeree_name` | Name of the issuer making the offer but the issuer is not available in the Corporate Actions dataset. | ### Par Value Redenomination (`PVRD`) Write up or write down of Par Value where shares outstanding remains the same BUT the Equity Capital changes. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `par_value_old` | Old Par Value | | `rate_info` | `par_value_new` | New Par Value | ### Return of Capital (`RCAP`) The company decides to return a portion of the equity capital in form of cash. It is often a result of Capital Reduction. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory** - Values for `event_subtype`: - `CAPRD`: Capital Reduction - `LIQ`: Liquidation - `RES`: Reserves - `SOA`: Sale of Assets - `SPA`: Share Premium Account - `null`: Blank - `null`: Generic event, no subtype provided #### **Calendar Dates** - `event_date` (equal to `payment_date`) - `effective_date` - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `financial_year_end_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `cash_back` | Cash Back | ### Record Date (`RD`) Depository Receipt Record Date Event - Event level: **Security** - Event category: **Other** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `record_date`) - `effective_date` (equal to `record_date`) - `record_date` - `to_date` ### Redemption (`REDEM`) Details of individual redemption of a redeemable security eg. Preference Share, Bond. Also sometimes used to transmit Voluntary Offers. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory / Voluntary** - Values for `event_subtype`: - `AMT`: Amortisation - `BBED`: Buyback Early Deadline - `BBRD`: Buyback Regular Deadline - `CORR`: Correction - `CU`: Clean Up - `ER`: Early Redemption - `MWC`: Make Whole Call - `PF`: Purchase Fund - `RM`: Residual Maturity - `null`: Generic event, no subtype provided #### **Calendar Dates** - `event_date` (equal to `payment_date`) - `event_date` (equal to `close_date`) - `effective_date` - `record_date` - `payment_date` - `to_date` - `registration_date` - `open_date` - `close_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `redemption_price` | Redemption Price | | `rate_info` | `redemption_premium` | Redemption Premium | | `event_info` | `part_final` | Is the Redemption for this date Part or Final. Part Redemptions may be Mandatory. (See **[PARTFINAL](https://databento.com/docs/conventions/reference-data-enums.md#partfinal)** enum) | | `event_info` | `amount_redeemed` | The total value of all the bonds redeemed i.e. in Millions. Mutually exclusive from Redemption in %. | | `event_info` | `redemption_percentage` | Redemption in %. To allow for Redemption of security which is only on % basis. | | `event_info` | `pool_factor` | The outstanding principal balance divided by the original principal balance with the result expressed as a Float. Useful in calculating redemption and outstanding par value. Introduced to capture this figure whenever its available. | | `event_info` | `price_as_percentage` | Flag to indicate that the Issue Price has been quoted only in percentages. | | `event_info` | `tender_offeror` | Name of the Offeror if it is different from the Issuer. | ### Reverse split (`RSPLT`) A consolidation event, currently only used for US securities. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - Values for `event_subtype`: - `CONSD`: Consolidation #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `par_value_old` | Old Par Value | | `rate_info` | `par_value_new` | New Par Value | ### Rights (`RTS`) Rights Issue is an offer to existing shareholders to subscribe to further securities in proportion to their holding, at a fixed price, made by means of the issue of a renounceable letter which may be traded for a period before payment for securities is due. The allotment of shares will usually be in the same company. - Event level: **Security** - Event category: **Distribution** - Event participation: **Voluntary** - Values for `event_subtype`: - `DPRCPDIV`: Depository Receipt Dividend - `null`: Generic event, no subtype provided #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `start_subscription_date` - `end_subscription_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `issue_price` | Issue Price | | `date_info` | `start_trade_date` | Start Trade Date | | `date_info` | `end_trade_date` | End Trade Date | | `date_info` | `split_date` | Split Date | | `event_info` | `sub_exp_time` | Subscription expiry time | | `event_info` | `sub_exp_time_zone` | Subscription expiry time zone | | `event_info` | `over_subscription` | A flag to indicate if the shareholders have the option of oversubscribing for the issue. (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `lapsed_premium` | If the shareholder does not subscribe to the rights issue the company would pay him this sum. | | `event_info` | `ratio_old_dist` | Distribution of Rights - Number of old shares held in old - new ratio | | `event_info` | `ratio_new_dist` | Distribution of Rights - Number of new Rights tradable shares received in old - new ratio | | `event_info` | `ratio_old_exer` | Exercise of Rights - Number of Rights exercised in old - new ratio | | `event_info` | `ratio_new_exer` | Exercise of Rights - Number of resultant securities received per Rights exercised in old - new ratio | | `event_info` | `trading_security_id` | Security ID of the security that may be created for Trading purposes. (Join with the `security_id` field) | | `event_info` | `trading_isin` | Trading ISIN if applicable/available | | `event_info` | `pp_security_id` | Poison Pill Rights: This rights behave same as normal rights but they do not trade at all. This rights are normally announced by the company to avoid hostile takeover. (Join with the `security_id` field) | | `event_info` | `pp_isin` | PP ISIN if applicable/available | ### Description Change (`SCCHG`) Change in the Security Description. - Event level: **Security** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `security_old_name` | Previous Name | | `event_info` | `security_new_name` | New Name | | `event_info` | `old_security_type` | Old Security Type Code (See **[SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype)** enum) | | `event_info` | `new_security_type` | New Security Type Code (See **[SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype)** enum) | | `event_info` | `old_reg_s144a` | Old RegS144A Value (See **[REGS144A](https://databento.com/docs/conventions/reference-data-enums.md#regs144a)** enum) | | `event_info` | `new_reg_s144a` | New RegS144A Value (See **[REGS144A](https://databento.com/docs/conventions/reference-data-enums.md#regs144a)** enum) | ### Security Swap (`SCSWP`) An exchange of securities of Company A with those of Company B. For example: Hong Kong companies changing to Bermudan companies prior to British handover to China. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `issue_price` | Issue Price | ### Subdivision (`SD`) Reduction in the Face/Nominal Value of the security. Company's Equity Capital remains the same but the shares in issue increases. Opposite of Consolidation. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - Values for `event_subtype`: - `SD`: Forward Split, currently only used in US events #### **Calendar Dates** - `event_date` (equal to `ex_date`) - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `par_value_old` | Old Par Value | | `rate_info` | `par_value_new` | New Par Value | | `event_info` | `old_par_value_currency` | ISO Old Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `new_par_value_currency` | ISO New Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | ### Sedol Change (`SDCHG`) Change in the SEDOL code. - Event level: **Country** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `event_info` | `old_sedol` | Old Sedol Code | | `event_info` | `new_sedol` | New Sedol Code | | `event_info` | `old_country` | Old Country Code - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | | `event_info` | `new_country` | New Country Code - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | | `event_info` | `old_reg_country` | Old Country of Register - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | | `event_info` | `new_reg_country` | New Country of Register - ISO 3166-1 (See **[CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry)** enum) | | `event_info` | `old_trading_currency` | Old Trading Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | | `event_info` | `new_trading_currency` | New Trading Currency Code - ISO 4217 (See **[CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen)** enum) | ### Security Reclassif. (`SECRC`) Company reclassifying Security A as Security B or exchanging one security for another. Hence, it happens within the company. For example: Bearer Shares to Registered Shares. This event is distinct from Security Swap. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` ### SOS Change (`SHOCH`) Changes in number of Shares Outstanding of a company. - Event level: **Security** - Event category: **Static Reference** - Event participation: **Mandatory** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `effective_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `date_info` | `old_outstanding_date` | Effective Date for previous SOS change | | `date_info` | `new_outstanding_date` | Effective Date for current SOS change | | `event_info` | `old_shares_outstanding` | Old shares outstanding amount | | `event_info` | `new_shares_outstanding` | New shares outstanding amount | ### Spin-off (`SOFF`) A distribution or demerger event, currently only used for US securities. - Event level: **Security** - Event category: **Distribution** - Event participation: **Mandatory** - Values for `event_subtype`: - `DIST`: Distribution - `DMRGR`: Demerger #### **Calendar Dates** - `event_date` - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | ### Takeover (`TKOVR`) Company A takes over the managerial control of Company B. Both companies may continue to exist. A Takeover may result in Compulsory Acquisition if the country's trigger limit of minimum public shareholding is breached. Compulsory Acquisition turns the Takeover event from Voluntary to Mandatory. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Mandatory / Voluntary** - Values for `event_subtype`: - `MRGR`: Merger - `TEND`: Tender Offer - `TENDMRGR`: Tender resulting in Merger - `TKOVRMINI`: Mini-Takeover - `UKWNSUBTYP`: Insufficient data to assign a TKOVR event subtype #### **Calendar Dates** - `event_date` (equal to `effective_date`) - `event_date` (equal to `close_date`) - `effective_date` - `ex_date` - `record_date` - `payment_date` - `duebills_redemption_date` - `to_date` - `registration_date` - `open_date` - `close_date` - `option_election_date` - `withdrawal_rights_from_date` - `withdrawal_rights_to_date` - `exp_completion_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `min_price` | Minimum Price | | `rate_info` | `max_price` | Maximum Price | | `date_info` | `unconditional_date` | Unconditional Date | | `date_info` | `compulsory_acq_date` | Compulsory Acquisition Date | | `event_info` | `pre_offer_qty` | Number of shares held by the Offeror prior to making the Takeover offer. | | `event_info` | `pre_offer_percentage` | Percentage of the Company's shares outstanding that the Offeror held prior to making the Takeover offer | | `event_info` | `target_qty` | Total Quantity of shares targeted for the Takeover by the Offeror. | | `event_info` | `target_percentage` | Percentage of the Company's shares outstanding targeted by the Offeror. | | `event_info` | `offeror_issuer_id` | Internal `issuer_id` of the Offeror (Join with the `issuer_id` field) | | `event_info` | `offeror_name` | Name of the issuer making the offer but the issuer is not available in the Corporate Actions dataset. | | `event_info` | `hostile` | (T)rue if Hostile Takeover (See **[BOOLEAN](https://databento.com/docs/conventions/reference-data-enums.md#boolean)** enum) | | `event_info` | `event_status` | Takeover Status Code (See **[TKOVRSTAT](https://databento.com/docs/conventions/reference-data-enums.md#tkovrstat)** enum) | ### Exercise of Warrants (`WAREX`) A Warrant normally has tenure, and this tenure could result in redemption or expire: this event describes the exercise terms. - Event level: **Security** - Event category: **Reorganisation** - Event participation: **Voluntary** - **Does not use** `event_subtype` #### **Calendar Dates** - `event_date` (equal to `to_date`) - `effective_date` (equal to `from_date`) - `from_date` - `to_date` #### **Dynamic Fields** | Parent [//]: # (column: sortable) | Field [//]: # (column: sortable) | Description | | `rate_info` | `strike_price` | Strike Price | | `rate_info` | `price_per_share` | Price Per Share | --- Source: https://databento.com/docs/datasets # Venues and datasets Differences between venue protocols and how they're normalized by Databento. | Title | Subtitle | Image | Supported languages | | [Blue Ocean ATS MEMOIR Depth](https://databento.com/docs/datasets/ocea-memoir.md) | | ![Blue Ocean icon](https://databento.com/docs/ocea-purple.svg.md) | Equities, OCEA.MEMOIR | | [Cboe BYX Depth](https://databento.com/docs/datasets/baty-pitch.md) | | ![Cboe icon](https://databento.com/docs/cboe-purple.svg.md) | Equities, BATY.PITCH | | [Cboe BZX Depth](https://databento.com/docs/datasets/bats-pitch.md) | | ![Cboe icon](https://databento.com/docs/cboe-purple.svg.md) | Equities, BATS.PITCH | | [Cboe EDGA Depth](https://databento.com/docs/datasets/edga-pitch.md) | | ![Cboe icon](https://databento.com/docs/cboe-purple.svg.md) | Equities, EDGA.PITCH | | [Cboe EDGX Depth](https://databento.com/docs/datasets/edgx-pitch.md) | | ![Cboe icon](https://databento.com/docs/cboe-purple.svg.md) | Equities, EDGX.PITCH | | [CFE](https://databento.com/docs/datasets/xcbf-pitch.md) | | ![CFE icon](https://databento.com/docs/cboe-purple.svg.md) | Futures, Options, XCBF.PITCH | | [CME Globex MDP 3.0](https://databento.com/docs/datasets/glbx-mdp3.md) | | ![CME icon](https://databento.com/docs/cme-purple.svg.md) | Futures, Options, GLBX.MDP3 | | [Databento Core Indices - CGIF](https://databento.com/docs/datasets/dbix-cgif.md) | | ![Databento icon](https://databento.com/docs/dbeq-purple.svg.md) | Indices, DBIX.CORE | | [Databento US Equities Mini](https://databento.com/docs/datasets/equs-mini.md) | | ![DBEQ icon](https://databento.com/docs/dbeq-purple.svg.md) | Equities, EQUS.MINI | | [Databento US Equities Summary](https://databento.com/docs/datasets/equs-summary.md) | | ![DBEQ icon](https://databento.com/docs/dbeq-purple.svg.md) | Equities, EQUS.SUMMARY | | [Eurex](https://databento.com/docs/datasets/xeur-eobi.md) | | ![XEUR icon](https://databento.com/docs/xeur-purple.svg.md) | Futures, Option, XEUR.EOBI | | [European Energy Exchange](https://databento.com/docs/datasets/xeee-eobi.md) | | ![XEEE icon](https://databento.com/docs/xeee-purple.svg.md) | Futures, Option, XEEE.EOBI | | [ICE Endex](https://databento.com/docs/datasets/ndex-impact.md) | | ![ICE icon](https://databento.com/docs/ice-purple.svg.md) | Futures, Options, NDEX.IMPACT | | [ICE Europe Commodities](https://databento.com/docs/datasets/ifeu-impact.md) | | ![ICE icon](https://databento.com/docs/ice-purple.svg.md) | Futures, Options, IFEU.IMPACT | | [ICE Europe Financials](https://databento.com/docs/datasets/ifll-impact.md) | | ![ICE icon](https://databento.com/docs/ice-purple.svg.md) | Futures, Options, IFLL.IMPACT | | [ICE Futures US](https://databento.com/docs/datasets/ifus-impact.md) | | ![ICE icon](https://databento.com/docs/ice-purple.svg.md) | Futures, Options, IFUS.IMPACT | | [IEX TOPS](https://databento.com/docs/datasets/iexg-tops.md) | | ![IEX icon](https://databento.com/docs/iex-purple.svg.md) | Equities, IEXG.TOPS | | [MEMX MEMOIR Depth](https://databento.com/docs/datasets/memx-memoir.md) | | ![MEMX icon](https://databento.com/docs/memx-purple.svg.md) | Equities, MEMX.MEMOIR | | [MIAX Pearl Depth of Market](https://databento.com/docs/datasets/eprl-dom.md) | | ![MIAX icon](https://databento.com/docs/miax-purple.svg.md) | Equities, EPRL.DOM | | [Nasdaq Basic with NLS Plus](https://databento.com/docs/datasets/xnas-basic.md) | | ![NASDAQ icon](https://databento.com/docs/nasdaq-purple.svg.md) | Equities, XNAS.BASIC | | [Nasdaq TotalView-ITCH](https://databento.com/docs/datasets/xnas-itch.md) | | ![NASDAQ icon](https://databento.com/docs/nasdaq-purple.svg.md) | Equities, XNAS.ITCH | | [Nasdaq PSX TotalView-ITCH](https://databento.com/docs/datasets/xpsx-itch.md) | | ![NASDAQ icon](https://databento.com/docs/nasdaq-purple.svg.md) | Equities, XPSX.ITCH | | [Nasdaq Texas TotalView-ITCH](https://databento.com/docs/datasets/xbos-itch.md) | | ![NASDAQ icon](https://databento.com/docs/nasdaq-purple.svg.md) | Equities, XBOS.ITCH | | [NYSE Integrated](https://databento.com/docs/datasets/xnys-pillar.md) | | ![NYSE icon](https://databento.com/docs/nyse-purple.svg.md) | Equities, XNYS.PILLAR | | [NYSE American Integrated](https://databento.com/docs/datasets/xase-pillar.md) | | ![NYSE icon](https://databento.com/docs/nyse-purple.svg.md) | Equities, XASE.PILLAR | | [NYSE Arca Integrated](https://databento.com/docs/datasets/arcx-pillar.md) | | ![NYSE icon](https://databento.com/docs/nyse-purple.svg.md) | Equities, ARCX.PILLAR | | [NYSE National Trades and BBO](https://databento.com/docs/datasets/xcis-tradesbbo.md) | | ![NYSE icon](https://databento.com/docs/nyse-purple.svg.md) | Equities, XCIS.TRADESBBO | | [NYSE Texas Integrated](https://databento.com/docs/datasets/xchi-pillar.md) | | ![NYSE icon](https://databento.com/docs/nyse-purple.svg.md) | Equities, XCHI.PILLAR | | [OPRA](https://databento.com/docs/datasets/opra-pillar.md) | | ![OPRA icon](https://databento.com/docs/opra-purple.svg.md) | Options, OPRA.PILLAR | | [Adjustment factors](https://databento.com/docs/datasets/adjustment-factors.md) | | ![Adjustment factors icon](https://databento.com/docs/adjustment-factors-dataset.svg.md) | Equities, Reference | | [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) | | ![Corporate actions icon](https://databento.com/docs/corporate-actions-dataset.svg.md) | Equities, Reference | | [Security master](https://databento.com/docs/datasets/security-master.md) | | ![Security master icon](https://databento.com/docs/security-master-dataset.svg.md) | Equities, Reference | --- Source: https://databento.com/docs/datasets/ocea-memoir # Blue Ocean ATS MEMOIR Depth **Dataset ID**: [OCEA.MEMOIR]($PLATFORM_HOSTNAME/catalog/us-equities#OCEA.MEMOIR) Blue Ocean ATS disseminates full depth-of-book data for its venue through its Blue Ocean ATS MEMOIR Depth feed, which Databento receives over UDP multicast in our NY4 data center. This dataset captures Blue Ocean ATS trading activity in US equities during the Blue Ocean Session, which runs from 8:00 PM to 4:00 AM ET, Sunday through Thursday. ### Symbol convention Raw symbols in this dataset use the **CMS** convention (e.g. `BRK B`, `AAC WS`); the source feed splits root and suffix across the `Symbol` and `SymbolSfx` fields, which Databento joins with a space (see [Definition normalization](https://databento.com/docs/datasets/ocea-memoir.md#definition-normalization)). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Blue Ocean messages received by Databento include one exchange timestamp with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. Blue Ocean does not include an exchange send timestamp, so `ts_in_delta` is set to `0` for normalized MBO and trades records. | Databento Field | MEMOIR Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | | `ts_in_delta` | N/A | Set to `0` because Blue Ocean does not include an exchange send timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### MBO normalization There are six MEMOIR message types that directly affect displayed book state or trade prints. Databento normalizes these messages to MBO records as follows: | Databento Message(s) | MEMOIR Message | Description | | **A**dd | *Order Added* | Adds a new displayed order with `order_id`, `side`, `price`, and `size`. | | **C**ancel | *Order Deleted* | Removes a displayed order. `size` is the displayed quantity removed from the book. | | **C**ancel | *Order Reduced* | Reduces a displayed order without changing time priority. `size` is the quantity removed from the order. | | **T**rade, **F**ill, **C**ancel | *Order Executed* | Executes a displayed order and updates displayed quantity by the executed amount. | | **T**rade | *Trade* | Non-displayed execution. Does not modify displayed book state. | | Clea**R** | *Clear Book* | Clears all displayed orders for the specified instrument. | For *Order Executed* messages, the aggressing order is not included in MEMOIR, so normalized **T**rade records have `order_id` set to `0`. Normalized records from a single native message share the same `sequence`. For *Order Executed* and *Trade*, Databento also publishes records in the [trades schema](https://databento.com/docs/schemas/trades.md). ### Trade side normalization For trade-derived records across the MBO, MBP-1, MBP-10, and trades schemas, the `side` field indicates the side that initiates the event. For `Order Executed`, Databento sets `side` to the aggressive order side, which is the opposite of the resting order side in the book. For `Trade`, the feed does not provide enough information to infer the aggressor, so the `side` is **N**one. ### Messages currently not normalized Databento does not emit MBO, trades, status, or definition output from these native message types: - *Broken Trade* - *Corrected Trade* ### Status normalization Status records are normalized from three MEMOIR messages: - *Security Trading Status* - *Trading Session Status* - *Reg SHO Restriction* **Security Trading Status** | Databento Action | Is trading | Is quoting | MEMOIR SecurityTradingStatus | | `8` (Halt) | No | No | `H` (Halted) | | `9` (Pause) | No | No | `P` (Paused) | | `3` (Quoting) | No | Yes | `Q` (Quoting) | | `7` (Trading) | Yes | Yes | `T` (Trading) | MEMOIR `SecurityTradingStatusReason` is normalized to status `reason`: | Databento Reason | MEMOIR SecurityTradingStatusReason | | None | `X` (None) | | Regulatory | `R` (Regulatory) | | Administrative | `A` (Administrative) | **Trading Session Status** | Databento Action | Is trading | Is quoting | MEMOIR TradingSession | | `1` (Pre-open) | No | Yes | `1` (Opening) | | `7` (Trading) | Yes | Yes | `2` (Trading) | | `13` (Post-close) | Yes | Yes | `3` (PostTrading) | | `12` (Close) | No | No | `4` (Closed) | **Reg SHO Restriction** `RegSHORestriction` is normalized to a status record for a short sell restriction change and updates `is_short_sell_restricted` from the source `ShortSaleRestriction` flag. Status publication is gated until both system-level state (`Trading Session Status`) and per-instrument state (`Security Trading Status`) are known. ### Definition normalization Definition records are sourced from *Instrument Directory* messages. | Databento Field | MEMOIR Field | Description | | `raw_instrument_id` | `SecurityID` | Native session-scoped identifier. | | `raw_symbol` | `Symbol` + `SymbolSfx` | CMS root and suffix combined with a space into Databento raw symbol. | | `min_lot_size_round_lot` | `RoundLot` | Round lot size. | | `min_price_increment_amount` | `MPV` | Minimum price variation converted to Databento's 1e-9 price units. | If a security is updated intraday, the corresponding definition is republished with updated values. Databento also sets `exchange` to `OCEA` and `match_algorithm` to `F` (FIFO). ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/baty-pitch # Cboe BYX Depth **Dataset ID**: [BATY.PITCH]($PLATFORM_HOSTNAME/catalog/us-equities#BATY.PITCH) Cboe BYX disseminates full depth-of-book data for their exchange through their Pitch Feed, which Databento receives over UDP multicast in our NY4 data center. The [Multicast Depth of Book (Pitch) Specification](https://cdn.cboe.com/resources/membership/US_EQUITIES_OPTIONS_MULTICAST_PITCH_SPECIFICATION.pdf) is available as a PDF from Cboe's website. > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`), even though the source venue is Cboe rather than Nasdaq. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Cboe messages received by Databento will generally include one timestamp with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pitch Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Time + TimeOffset | The matching-engine-received timestamp. | | `ts_in_delta` | None | The difference between `ts_recv` and `ts_event`. Redundant and can be safely ignored. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Statistics Databento provides Opening Price and Closing Price statistics, normalized from the *Auction Summary* message. These statistics have the auction uncrossing price and quantity for the respective event. ### Imbalance normalization Cboe BYX provides information about opening and closing cross imbalances in their *Imbalance* message. Databento normalizes each message into a record in the [imbalance schema](https://databento.com/docs/schemas/imbalance.md). | Databento Field | Pitch Field | Description | | `ref_price` | ReferencePrice | Collared price at which the periodic auction would trade. | | `paired_qty` | min(BuyShares, SellShares) | The number of shares that are eligible to match at the `ref_price`. | | `total_imbalance_qty` | abs(SellShares - BuyShares) | The number of shares not paired at the `ref_price`. | | `auction_type` | AuctionType | **P**: Periodic auction. | | `side` | BuyShares, SellShares | The market side of the order imbalance: **B**id (buy) imbalance, **A**sk (sell) imbalance, **N**o imbalance. | ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/bats-pitch # Cboe BZX Depth **Dataset ID**: [BATS.PITCH]($PLATFORM_HOSTNAME/catalog/us-equities#BATS.PITCH) Cboe BZX disseminates full depth-of-book data for their exchange through their Pitch Feed, which Databento receives over UDP multicast in our NY4 data center. The [Multicast Depth of Book (Pitch) Specification](https://cdn.cboe.com/resources/membership/US_EQUITIES_OPTIONS_MULTICAST_PITCH_SPECIFICATION.pdf) is available as a PDF from Cboe's website. > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`), even though the source venue is Cboe rather than Nasdaq. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Cboe messages received by Databento will generally include one timestamp with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pitch Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Time + TimeOffset | The matching-engine-received timestamp. | | `ts_in_delta` | None | The difference between `ts_recv` and `ts_event`. Redundant and can be safely ignored. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Statistics normalization Cboe BZX provides information about the execution price and shares matched from an auction in their *Auction Summary* message. Databento normalizes each message into a record in the [statistics schema](https://databento.com/docs/schemas/statistics.md). Statistics are categorized by the following auction types: - Opening price: Opening - Closing price: Closing ### Imbalance normalization Cboe BZX provides information about opening and closing cross imbalances in their *Imbalance* message. Databento normalizes each message into a record in the [imbalance schema](https://databento.com/docs/schemas/imbalance.md). | Databento Field | Pitch Field | Description | | `ref_price` | ReferencePrice | The BBO collared auction price. | | `cont_book_clr_price` | IndicativePrice | Price at which the auction book and the continuous book would match. | | `auct_interest_clr_price` | AuctionOnlyPrice | Price at which the auction book would match using only Eligible Auction Orders. | | `paired_qty` | min(BuyShares, SellShares) | The number of shares that are eligible to match at the `ref_price`. | | `total_imbalance_qty` | abs(SellShares - BuyShares) | The number of shares not paired at the `ref_price`. | | `auction_type` | AuctionType | **O**: Opening auction. **C**: Closing auction. **H**: Halt auction. **I**: IPO auction. **M**: Cboe Market Close. | | `side` | BuyShares, SellShares | The market side of the order imbalance: **B**id (buy) imbalance, **A**sk (sell) imbalance, **N**o imbalance. | ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/edga-pitch # Cboe EDGA Depth **Dataset ID**: [EDGA.PITCH]($PLATFORM_HOSTNAME/catalog/us-equities#EDGA.PITCH) Cboe EDGA disseminates full depth-of-book data for their exchange through their Pitch Feed, which Databento receives over UDP multicast in our NY4 data center. The [Multicast Depth of Book (Pitch) Specification](https://cdn.cboe.com/resources/membership/US_EQUITIES_OPTIONS_MULTICAST_PITCH_SPECIFICATION.pdf) is available as a PDF from Cboe's website. > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`), even though the source venue is Cboe rather than Nasdaq. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Cboe messages received by Databento will generally include one timestamp with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pitch Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Time + TimeOffset | The matching-engine-received timestamp. | | `ts_in_delta` | None | The difference between `ts_recv` and `ts_event`. Redundant and can be safely ignored. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/edgx-pitch # Cboe EDGX Depth **Dataset ID**: [EDGX.PITCH]($PLATFORM_HOSTNAME/catalog/us-equities#EDGX.PITCH) Cboe EDGX disseminates full depth-of-book data for their exchange through their Pitch Feed, which Databento receives over UDP multicast in our NY4 data center. The [Multicast Depth of Book (Pitch) Specification](https://cdn.cboe.com/resources/membership/US_EQUITIES_OPTIONS_MULTICAST_PITCH_SPECIFICATION.pdf) is available as a PDF from Cboe's website. > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`), even though the source venue is Cboe rather than Nasdaq. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Cboe messages received by Databento will generally include one timestamp with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pitch Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Time + TimeOffset | The matching-engine-received timestamp. | | `ts_in_delta` | None | The difference between `ts_recv` and `ts_event`. Redundant and can be safely ignored. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Flags normalization For Cboe EDGX Depth, the `F_PUBLISHER_SPECIFIC` flag is used in the [MBO](https://databento.com/docs/schemas/mbo.md) schema to indicate Retail Priority Orders (RPOs). MBO messages with action `A` will have the `F_PUBLISHER_SPECIFIC` flag set if the order is an RPO. This flag will not repeat on subsequent modifications or cancellations. Retail Priority changes the order allocation priority model on EDGX to price-retail priority-time priority. Thus, Retail Priority Orders will have priority ahead of all other available interest at a price level. --- Source: https://databento.com/docs/datasets/xcbf-pitch # CFE **Dataset ID**: [XCBF.PITCH]($PLATFORM_HOSTNAME/catalog/xcbf/XCBF.PITCH) Cboe disseminates full depth-of-book data for their Cboe Futures Exchange (CFE) through their Multicast Depth of Book (PITCH) feed, which Databento receives over UDP multicast in our NY4 data center. PDF specifications are available on [Cboe's website](https://www.cboe.com/us/futures/support/technical) for [Multicast PITCH](https://cdn.cboe.com/resources/membership/CBOE_FUTURES_EXCHANGE_MULTICAST_PITCH_SPECIFICATION.pdf) and [Options on Futures Multicast PITCH](https://cdn.cboe.com/resources/membership/CFE_OOF_Multicast_PITCH_Specification.pdf). ### Timestamps Cboe messages received by Databento will generally include one timestamp with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. As this protocol does not have a `Sending Time` field, the field `ts_in_delta` is set to 0 and should be ignored. | Databento Field | Pitch Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | EpochTime + TimeOffset | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Trading hours CFE operates from Sunday through Friday on a weekly schedule, with daily sessions. Trading sessions vary by product, refer to the Cboe [trading hours](https://www.cboe.com/about/hours/us-futures) for up to date information. * Regular trading runs each day from Monday through Friday * There is extended trading both before and after each regular trading session. * At the end of the post-session extended trading all orders are canceled * There is a pause between 4:00 PM and 5:00 PM CT | Description | Hours | | Regular Trading Hours | 8:30 AM to 3:00 PM (CT) | | Extended Trading Hours (pre-session) | 5:00 PM (previous day) to 08:30 AM (CT) | | Extended Trading Hours (post-session) | 3:00 PM to 4:00 PM (CT) | | Daily pause | 4:00 PM to 5:00 PM (CT) | Note: Not all products have extended trading. The [trading hours](https://www.cboe.com/about/hours/us-futures) for some commonly traded products are: * **[VX](https://www.cboe.com/tradable-products/vix/vix-futures/specifications) and [VXM](https://cdn.cboe.com/resources/futures/VXM_Contract_Specifications.pdf) futures**: Sunday 5:00 PM - Friday 4:00 PM CT (*extended trading hours* + *regular trading hours*), with a daily trading halt from 4:00 PM to 5:00 PM CT. * [**UX (Options on VX futures)**](https://cdn.cboe.com/resources/membership/VIX_OOF_Contract_Specification.pdf): Monday-Friday, 8:30 AM - 3:00 PM CT (*regular trading hours*). ### MBO normalization **Order lifecycle messages** There are seven message types in CFE PITCH that change the state of the order book. Databento normalizes these messages to MBO records as follows: | CFE Message | Databento Message(s) | Description | | Add Order (long) | **A**dd | | | Add Order (short) | **A**dd | | | Order Executed | **T**rade, **F**ill, **C**ancel | | | Reduce Size (long) | **C**ancel | The `size` field indicates the number of contracts to remove from the display size | | Reduce Size (short) | **C**ancel | The `size` field indicates the number of contracts to remove from the display size | | Modify (long) | **M**odify | The `price` and `size` fields indicate the new order price and quantity | | Modify (short) | **M**odify | The `price` and `size` fields indicate the new order price and quantity | | Delete | **C**ancel | The `size` field indicates the remaining display size of the deleted order | **T**rade records normalized from *Order Executed* will have `order_id` set to `0`, as Cboe does not provide information about the aggressor's order. **T**rade records normalized from *Order Executed* will have `side` set to the opposite side of the matched resting order. **Trade messages** CFE Multicast PITCH also has a `Trade` message for providing execution details for match events that occur off the CFE book, such as [ECRP or Block trades](https://cdn.cboe.com/resources/membership/cfe-block-ecrp-trades.pdf). Databento normalizes this *Trade* message as a single MBO **T**rade record with side **N**one and publisher set to `XCBF.PITCH.XOFF`. Cboe sends `Trade` messages with an obfuscated Order Id, this is passed through in the normalized **T**rade record. ### Statistics normalization For the `XCBF.PITCH` dataset, statistics records are normalized from the following CFE messages. | CFE Message | StatType(s) | | Price Limits | Upper price limit, lower price limit | | Settlement | Settlement price | | Open Interest | Open interest | | End of Day Summary | Highest bid, trading session high price, lowest offer, trading session low price, opening price, close price, cleared volume, block volume, venue-specific volume 1 | **Note**: Fields `Accrued Day Variance`, `Num Final Returns`, and `Num Elapsed Returns` from the Futures Variance Symbol Mapping Message are not currently normalized. Fields from these messages are normalized into records in the [statistics schema](https://databento.com/docs/schemas/statistics.md) with the appropriate `StatType`. **Price Limits** CFE publishes `Upper Price Limit` and `Lower Price Limit` in a `Price Limits` Message. These fields are normalized into [statistics](https://databento.com/docs/schemas/statistics.md) records with a `StatType` of upper price limit and lower price limit respectively, as prices. **Settlement** CFE publishes `Settlement Price` in a `Settlement` Message. This field is normalized into a [statistics](https://databento.com/docs/schemas/statistics.md) record with a `StatType` of settlement price, as a price. **Open Interest** CFE publishes `Open Interest` in an `Open Interest` Message. This field is normalized into a [statistics](https://databento.com/docs/schemas/statistics.md) record with a `StatType` of open interest, as a quantity. **End of Day Summary** The `End of Day Summary` message is sent after trading ends for a symbol. ***Quantity Fields*** `Total Volume` represents the total number of contracts traded for the day including [Block and ECRP trades](https://cdn.cboe.com/resources/membership/cfe-block-ecrp-trades.pdf). `Block Volume` represents the total number of Block and Derived contracts traded for the day. `ECRP Volume` represents the total number of ECRP contracts traded for the day. These fields are normalized as [statistics](https://databento.com/docs/schemas/statistics.md) records as quantities, with the following `StatType`s: | CFE Field | StatType | | Total Volume | Cleared volume | | Block Volume | Block volume | | ECRP Volume | Venue-specific volume 1 | ***Price Fields*** `High Price` is the higher of highest bid and highest trade price for the day. `Low Price` is the lower of lowest offer and lowest trade price for the day. `Open Price` is set by the first trade of the day (in any session). `Close Price` is set by the last trade of the day (in any session). [Block and ECRP trades](https://cdn.cboe.com/resources/membership/cfe-block-ecrp-trades.pdf) do not affect these price fields. | CFE Field | StatType | | High Price (if from bid) | Highest bid | | High Price (if from trade) | Trading session high price | | Low Price (if from offer) | Lowest offer | | Low Price (if from trade) | Trading session low price | | Open Price | Opening price | | Close Price | Close price | ### Status normalization `Trading Status` messages are normalized into records in the [status schema](https://databento.com/docs/schemas/status.md). | CFE Trading Status | `action` | `is_trading` | `is_quoting` | | Suspended | `SUSPEND` | No | No | | Queuing | `PRE_OPEN` | No | Yes | | Trading | `TRADING` | Yes | Yes | | Halted | `HALT` | No | No | ### Definition normalization Definition records are sourced from *Futures Instrument Definition*, *OOF Symbol Mapping*, and *Complex Instrument Definition Expanded* messages. The current lists of CFE products can be downloaded from the [CFE Contract Listings and Product Level System Parameters](https://www.cboe.com/markets/us/futures/product-level-system-parameters) on the Cboe page. * [Simple Futures](https://cdn.cboe.com/data/us/futures/market_statistics/symbol_reference/latest-simple.csv) * [Variance Futures](https://cdn.cboe.com/data/us/futures/market_statistics/symbol_reference/va-latest-simple.csv) * [Complex Futures](https://cdn.cboe.com/data/us/futures/market_statistics/symbol_reference/latest-complex.csv) * [Options on Futures](https://cdn.cboe.com/data/us/futures/market_statistics/symbol_reference/latest-simple-options.csv) * [Complex Options on Futures](https://cdn.cboe.com/data/us/futures/market_statistics/symbol_reference/latest-complex-options.csv) **Instrument IDs** CFE publishes instrument IDs as a six character base-62 ASCII value. Databento normalizes this to instrument ID by converting from base62 to decimal base 10. For example: | CFE Symbol | `instrument_id` | | "000000"b62 | 0 | | "000001"b62 | 1 | | "00000z"b62 | 61 | | "000100"b62 | 3844 (62*62) | Complex Options on Futures symbols start at values "z00000"b62, which is 55884102752. The maximum base-10 value that can be represented by a six-char base62 string is "zzzzzz" which is 56800235583. The Databento `instrument_id` field is 32-bit unsigned, which has a maximum value of 4294967295. Because of this, values above 4000000000 are used for normalization of Complex Option on Futures symbols. Values of "z00000"b62 and above are mapped to offset 4000000000. | CFE Symbol | `instrument_id` | | "z00000"b62 | 4000000000 | | "z00001"b62 | 4000000001 | | "z0000z"b62 | 4000000061 | **Futures Instrument Definition Message** | Databento Field | CFE Field | Description | | `instrument_id` | Symbol | The mapped value from base62 | | `raw_instrument_id` | Symbol | The original decoded base62 value | | `ts_event` | Unit Timestamp | Used only if non-zero, otherwise populated from TimeOffset | | `asset` | Report Symbol | | | `expiration` | Expiration Date | | | `original_contract_size` | Contract Size | | | `min_price_increment` | Price Increment | | | `leg_count` | Leg Count | | | `maturity_year` | Contract Date | Zero for spread instruments | | `maturity_month` | Contract Date | Zero for spread instruments | | `maturity_day` | Contract Date | Zero for spread instruments | | `leg_ratio_qty_numerator` | Leg Ratio | Absolute value of the leg ratio | | `leg_ratio_qty_denominator` | Leg Ratio | Always 1 | | `leg_side` | Leg Ratio | Positive ratio=`BID`, negative ratio=`ASK` | | `leg_instrument_id` | Leg Symbol | Converted from base62 | The following fields are not normalized: `Futures Flags`, `Listing State`, `Leg Offset`. The `raw_symbol` for Simple Futures attempts to match the Cboe "Product". This field is not published in the **Futures Instrument Definition** message. Instead it is constructed from the **Report Symbol** and **Contract Date** fields. The format is `[asset]/[maturity_month_code][maturity_year_code]`. Where these fields are derived from: Month Codes: | Month | Code | | January | F | | February | G | | March | H | | April | J | | May | K | | June | M | | July | N | | August | Q | | September | U | | October | V | | November | X | | December | Z | Maturity year uses the last digit of the year for maturities before 2030. Two digits are used for maturities from 2030 onward. Examples: | Asset | Maturity month | Maturity year | Symbol | | VX | January | 2026 | `VX/F6` | | PBT | December | 2035 | `PBT/Z35` | The raw symbol for future spreads is determined by the legs. The format is: `LegSymbol:Ratio:Side - LegSymbol:Ratio:Side`. For example, for a spread with ratio 1:1, with legs `VX/J6` and `VX/H6` the resulting spread `raw_symbol` will be `VX/H6:1:S - VX/J6:1:B` **VX Weekly Expirations** VX weekly futures use a different **Report Symbol** from the VX futures product. For these weekly contracts, CFE uses VX01 through VX53. The embedded suffix denotes the specific week of a calendar year during which a contract is settled. Each VX weekly expiration has a corresponding [parent contract](https://databento.com/docs/conventions/symbology.md#parent). Examples: | Asset | Maturity week | Maturity month | Maturity year | Symbol | | VX | Week 1 | January | 2026 | `VX01/F6` | | VX | Week 12 | March | 2052 | `VX12/Z52` | **OOF Symbol Mapping Message** | Databento Field | CFE Field | Description | | `ts_event` | Time Offset | | | `instrument_id` | Feed Symbol | The mapped value from base62 | | `raw_instrument_id` | Feed Symbol | The original decoded base62 value | | `group` | Futures Product | The underlying futures product root (e.g. `VX`) | | `underlying_id` | Futures Symbol | Converted from base62 | | `strike_price` | Strike Price | | | `instrument_class` | Call Put Indicator | | | `expiration` | Options Expiration | | | `raw_symbol` | Options on Futures Name | | The following fields are not normalized: `Futures Expiration`, `Symbol Condition`. The `asset` field is derived from the `raw_symbol` by extracting the root up to the first `/` character. The `security_type` field is set to `OOF`. The `raw_symbol` comes from the `Options on Futures Name` field, which follows the Cboe US Symbology definition. See section *CFE Options on Futures Symbology* of the [Cboe US Symbology Reference](https://cdn.cboe.com/resources/membership/US_Symbology_Reference.pdf). **Complex Instrument Definition Expanded Message** | Databento Field | CFE Field | Description | | `ts_event` | Time Offset | | | `instrument_id` | Complex Instrument Id | The mapped value from base62 | | `raw_instrument_id` | Complex Instrument Id | The original decoded base62 value | | `group` | Complex Instrument Underlying | The underlying futures product root (e.g. `VX`) | | `leg_count` | Leg Count | | | `leg_instrument_id` | Leg Symbol | Converted from base62 | | `leg_ratio_qty_numerator` | Leg Ratio | Absolute value of the leg ratio | | `leg_ratio_qty_denominator` | Leg Ratio | Always 1 | | `leg_side` | Leg Ratio | Positive ratio=`BID`, negative ratio=`ASK` | The following fields are not normalized: `Complex Instrument Type`. The `instrument_class` is set to `OPTION_SPREAD` and `security_type` is set to `OOF`. The `asset` field is determined from the leg definitions. The `raw_symbol` for complex options is determined by the legs, using the same format as complex futures: `LegSymbol:Ratio:Side - LegSymbol:Ratio:Side`. For example, for an option spread with ratio 1:1 and legs `UX4E/J6 C3000` and `UX4E/J6 C4000`, where both sides are `BID`, the resulting `raw_symbol` would be `UX4E/J6 C3000:1:B - UX4E/J6 C4000:1:B` ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/dbix-cgif # Databento Core Indices - CGIF **Dataset ID**: [DBIX.CORE]($PLATFORM_HOSTNAME/catalog/dbix/DBIX.CORE) `DBIX.CORE` consolidates index data from multiple upstream feeds into a single dataset, organized by `venue`. It's an aggregate of index feeds rather than an order-driven market, so Databento does not offer an [MBO schema](https://databento.com/docs/schemas/mbo.md) for `DBIX.CORE`. This page covers the channels sourced from Cboe's Titanium Cboe Global Indices Feed (CGIF). ### Channels Cboe disseminates index values, contributor values, and index summaries from several index providers through CGIF, which Databento receives over UDP multicast in our NY4 data center and normalizes into `DBIX.CORE`. CGIF carries values from seven index providers, each its own channel, mapped to a distinct `Publisher` within `DBIX.CORE`: | Channel | Publisher (ID) | Provider | Source messages | | `MAIN` | `DBIX.CORE.MAIN` (146) | Cboe Global Indices Feed | `IndexValue` | | `MSCI` | `DBIX.CORE.MSCI` (147) | MSCI | `IndexValue` | | `FTSE` | `DBIX.CORE.FTSE` (148) | FTSE Russell | `IndexValue` | | `INAV` | `DBIX.CORE.INAV` (149) | Intraday Net Asset Values | `IndexValue` | | `MSTAR` | `DBIX.CORE.MSTAR` (150) | Morningstar | `ContributorValue`, `IndexSummary` | | `CCCY` | `DBIX.CORE.CCCY` (151) | Crypto Currency Indices | `IndexValue` | | `CGI` | `DBIX.CORE.CGI` (152) | Cboe Global Indices | `IndexValueWithStatus` | Index status is normalized to the [status](https://databento.com/docs/schemas/status.md) schema, and index definitions to [instrument definitions](https://databento.com/docs/schemas/instrument-definitions.md), the same way across all seven channels. ### Timestamps CGIF messages include a `SendingTime` in the packet header, shared by every message in that packet. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | CGIF Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | `SendingTime` (packet header) | The gateway-sending timestamp, shared by every message in the same packet. | | `ts_in_delta` | N/A | The delta between `ts_recv` and `ts_event`, or between `ts_recv` and `TransactTime` for `MSTAR`. | Note that `SendingTime` only carries millisecond resolution, and no additional precision is added when normalizing it to a nanosecond `ts_event`. For `MSTAR`, messages also include a `TransactTime`, a millisecond-resolution epoch timestamp present on both `ContributorValue` and `IndexSummary` messages. `TransactTime` is the source for `ts_in_delta` on `MSTAR`'s [MBP-1](https://databento.com/docs/schemas/mbp-1.md), [trades](https://databento.com/docs/schemas/trades.md), and [statistics](https://databento.com/docs/schemas/statistics.md) records; it's more accurate than the packet-level `SendingTime` used for the other channels' `ts_in_delta`. For `IndexSummary` messages, the [statistics schema](https://databento.com/docs/schemas/statistics.md)'s `ts_ref` field is set to the summary's effective `Date` (a `YYYYMMDD` value) decoded to UTC midnight, rather than a precise event time. More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### MBP-1 and Trades normalization CGIF does not publish order book depth, so each source message is normalized to a single-level [MBP-1](https://databento.com/docs/schemas/mbp-1.md) record: the `Bid` and `Ask` values become `bid_px` and `ask_px` at the top book level, the index value becomes `price`, and `action` is fixed to Trade. Every MBP-1 record also produces a matching [trades](https://databento.com/docs/schemas/trades.md) record, copying over `ts_recv`, `ts_event`, `ts_in_delta`, `price`, `flags`, and `action`, with `depth` fixed to `0`. **MAIN, MSCI, FTSE, INAV, CCCY** The source message is `IndexValue`, and `flags` is fixed to `F_LAST`. **CGI** The source message is `IndexValueWithStatus`, but only entries without a `Close` or `Settlement` value are normalized this way; see Statistics normalization below for the rest. `F_LAST` is always set. `F_PUBLISHER_SPECIFIC` indicates an indicative index value: set when `IndexStatus` is `Indicative`, and unset when `Normal`. **MSTAR** The source message is `ContributorValue`, and `flags` is fixed to `F_LAST`. ### Statistics normalization Only `CGI` and `MSTAR` produce [statistics schema](https://databento.com/docs/schemas/statistics.md) records; the other CGIF channels have no source messages that normalize to statistics. **CGI** `CGI`'s `IndexValueWithStatus` entries that carry a `Close` or `Settlement` value are normalized to statistics instead of MBP-1/trades: | CGIF MDEntryType | `stat_type` | `stat_flags` | | `Close` | `CLOSE_PRICE`, or `INDICATIVE_CLOSE_PRICE` when `IndexStatus` is `Indicative` | Unset | | `Settlement` | `SETTLEMENT_PRICE` | `1` when `IndexStatus` is `Normal`, or `0` when `Indicative` | `CGI` only carries `IndexValueWithStatus` messages - it doesn't receive `IndexSummary` messages, so it never produces the start-of-day and end-of-day statistics that `MSTAR` does. **MSTAR** `MSTAR`'s `IndexSummary` messages are always normalized to statistics, one record per `MDEntryType` entry. `stat_type` is always `VENUE_SPECIFIC_PRICE_1`. `stat_flags` is a bit field that depends on the entry's `MDEntryType` and whether `SummaryType` is start-of-day (`SOD`) or end-of-day (`EOD`): | Value | Decimal | Description | | `1 << 0` | 1 | `IndexValue` entry, `SummaryType` is `SOD`. | | `1 << 1` | 2 | `Bid` entry, `SummaryType` is `SOD`. | | `1 << 2` | 4 | `Ask` entry, `SummaryType` is `SOD`. | | `1 << 3` | 8 | `IndexValue` entry, `SummaryType` is `EOD`. | | `1 << 4` | 16 | `Bid` entry, `SummaryType` is `EOD`. | | `1 << 5` | 32 | `Ask` entry, `SummaryType` is `EOD`. | `MSTAR` doesn't receive `IndexValueWithStatus` messages, so it never produces the Close and Settlement statistics that `CGI` does. ### Definition normalization Index definitions are normalized to [instrument definitions](https://databento.com/docs/schemas/instrument-definitions.md) the same way across all seven channels: | Databento Field | CGIF Field | Description | | `raw_symbol` | `Symbol` | Normalized verbatim. | | `group` | `Agent` | A 2-character classification code. | | `secsubtype` | `Channel` | The CGIF channel that index data is received on. | | `exchange` | N/A | Always set to the literal `CGIF`. | | `instrument_class` | N/A | Always `INDEX`. | | `match_algorithm` | N/A | Always `UNDEFINED`, since CGIF is an index feed with no matching engine. | ### Status normalization The corresponding [status schema](https://databento.com/docs/schemas/status.md) record is derived from the definition's `Status` field: | CGIF Trading Status | `action` | `is_trading` | `is_quoting` | | Active | `TRADING` | Yes | Yes | | Inactive | `CLOSE` | No | No | | Deleted | `SUSPEND` | No | No | --- Source: https://databento.com/docs/datasets/glbx-mdp3 # CME Globex MDP 3.0 **Dataset ID**: [GLBX.MDP3]($PLATFORM_HOSTNAME/catalog/cme/GLBX.MDP3) The CME Group disseminates full depth-of-book data for CME, CBOT, NYMEX, and COMEX through their MDP 3.0 feed, which Databento receives over UDP multicast in our DC3 datacenter. Databento provides full coverage of all futures and options on futures beginning in June 2010. In May 2017, CME introduced MDP 3.0, which provides full granularity for every order event (MBOFD) in addition to providing aggregated depth at a limited number of price levels (MBP). Before May 2017, data was based on the legacy FIX/FAST protocol. CME makes the specifications of their MDP 3.0 feed available [on their website here](https://www.cmegroup.com/confluence/display/EPICSANDBOX/CME+MDP+3.0+Market+Data). ### Timestamps CME MDP 3.0 messages received by Databento will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | MDP 3.0 Tag | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | 60-TransactTime | The matching-engine-received timestamp. | | `ts_in_delta` | 52-SendingTime | The matching-engine-sending timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### MBO normalization **`F_LAST` flag in MBO** When interpreting MBO data, the `F_LAST` [flag](https://databento.com/docs/conventions/common-fields-enums-types.md#flags) (`0x80`, 128) is used to mark the last record in a single event for each `instrument_id`. This flag is based on CME's "End of event" flag in tag 5799-MatchEventIndicator. However, CME only sets this flag on one single message, even if the event spanned multiple instruments. Instead, we set the `F_LAST` flag on the last record for _each_ instrument, so that the data can be interpreted consistently with any subset of instruments. Records with `action` **N**one may carry `F_LAST`. Across all of Databento's feeds, it is important to use the `F_LAST` flag when calculating the best bid and offer, like our MBP-1 schema. Between records *without* the `F_LAST` flag, the book is in the process of updating, and the apparent best bid and offer may have already been traded. This is accounted for when we construct our other schemas, such as MBP-1 and Trades. Outside of MBO, this flag can be ignored. **Order events** Records in the [MBO schema](https://databento.com/docs/schemas/mbo.md) with **A**dd, **M**odify, and **C**ancel actions are normalized from [Market Data Incremental Refresh - MBOFD](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Incremental+Refresh+-+MBOFD), and records with **T**rade and **F**ill actions are normalized from [Market Data Incremental Refresh - Trade Summary](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Incremental+Refresh+-+Trade+Summary) messages, and records with the clea**R** action are normalized from [Market Data Incremental Refresh - Channel Reset](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Incremental+Refresh+-+Channel+Reset) messages. **Trade events** The MBO schema includes detailed information about both sides of the trade by normalizing [Market Data Incremental Refresh - Trade Summary](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Incremental+Refresh+-+Trade+Summary) messages from CME into **T**rade and **F**ill actions. The [MBP-n and trades schemas](https://databento.com/docs/schemas/mbp-1.md) include **T**rade records, but do not include **F**ill detail records. Each CME MDP 3.0 [Market Data Incremental Refresh - Trade Summary](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Incremental+Refresh+-+Trade+Summary) message from CME contains two repeating groups: a set of *trade summaries*, followed by a set of *order ID entries*. Each trade summary in the message is normalized into one **T**rade record. If there is a defined aggressor for the trade, its corresponding order ID entry from the second set is used to populate the `order_id` field. This commonly happens when an incoming order partially executes, leaving the remaining quantity to rest on the book. Following each **T**rade record, the remaining (passive) order ID entries are normalized into **F**ill records. These records correspond to resting orders in the order book, but **do not** modify the orders. If the aggressing order existed on the book, then a **F**ill record will be emitted for it. This can happen if a resting order was modified to cross the book. If the aggressing order did not already exist on the book then its **F**ill record is suppressed, however its order ID is still reported in the **T**rade record as described above. **Side** [Implied trades](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Implied+Book) may not have an aggressing side set. These trades will be normalized with [side](https://databento.com/docs/conventions/common-fields-enums-types.md#side) set to **N**one. **Order Priority** Although Databento does not expose tag 37707-MDOrderPriority in the MBO schema, first in, first out priority (FIFO) can be determined from message order. Messages for a single `instrument_id` are never reordered, even if they have the same timestamps. Our daily MBO snapshots also preserve FIFO order priority. **CME MBO snapshot** At the beginning of the weekly trading session, CME publishes a MBO order snapshot. This snapshot contains any orders that persisted from the previous trading session, such as Good 'Till Cancelled orders. The event timestamp (`ts_event`) for these orders corresponds to the time the snapshot is generated by CME. It does not reflect the original time the order was entered or last modified. These orders are not published by CME in priority order. As Databento does not expose tag 37707-MDOrderPriority, if these MBO records were published by Databento in the same order they were sent from CME, it would not be possible to determine the correct order priority. In order to publish the records in priority order, Databento buffers the orders from the first CME event of the trading session, per instrument. Once the End of Event is reached, Databento sorts the records in priority order and then publishes them. Therefore, the order priority is correctly reflected by the order of publishing. The `ts_recv` for these records will be set to the `ts_recv` of the final record of the event. As the records have been slightly delayed and the `ts_recv` of the initial records may not match the original `ts_recv`, the flag `F_BAD_TS_RECV` is set on these records. The `F_SNAPSHOT` flag is also set on these records to indicate that they are from the CME snapshot. **Weekly Sessions** Although CME pauses trading daily, the market follows a weekly session structure with most instruments accepting orders from Sunday night to Friday night (local time). To make it easier to work with historical MBO data, Databento includes an [order book snapshot](https://databento.com/docs/conventions/mbo-snapshot.md) at 00:00:00 UTC each weekday (Monday-Friday). Before the trading session begins, or during the daily pause, it is normal for order books to be locked or crossed: CME will accept orders but won't execute any trades until the opening uncrossing. Details about the trading hours and sessions for each product group are available on the [Trading hours](https://www.cmegroup.com/trading-hours.html) page of CME's website. ### Implied book In futures and options markets, implied depth allows for extra liquidity beyond what's visible in the order book for a specific instrument. It is derived from the relationships between different instruments, such as between outrights, spreads, or multi-leg options. CME's matching engine uses implied depth to construct liquidity, ensuring that even if a direct order isn't placed, the system can infer potential liquidity from other related contracts. CME's MDP3 feed has support for implied orders over their MBP feed. The MBP feed provides up to 10 levels of real liquidity and 2 levels of implied liquidity. Databento combines the real and implied books, publishing the merged top-of-book on the [CMBP-1](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1) schema. Databento also provides subsampled views of the merged top-of-book on the [CBBO-1s and CBBO-1m](https://databento.com/docs/schemas/bbo.md#consolidated-bbo-on-interval-cbbo) schemas. CME provides [documentation and examples](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457095346/Implied+Orders) for the implied book. ### Statistics normalization Databento normalizes the following daily statistics from [Market Data Incremental Refresh - Daily Statistics](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Incremental+Refresh+-+Daily+Statistics): - Settlement price - Cleared volume - Open interest - Fixing price The exact times these statistics are published will vary. Cleared volume and open interest statistics will normally publish on the following UTC date, unless they are for Friday's session, in which case they will be published on the following Sunday. Settlement price statistics will first publish shortly after the [settlement window](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457085528/Daily+Settlement+Time+Details) for an instrument. The CME does not publish settlement prices on their MDP feed for instruments without open interest or volume. Databento normalizes the following session statistics from [Market Data Incremental Refresh - Session Statistics](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Incremental+Refresh+-+Session+Statistics): - Opening and indicative opening price - Trading session high and low price - Trading session highest bid and lowest offer These statistics are published throughout the trading session as the values change. **Price limits** Databento normalizes the `HighLimitPrice` and `LowLimitPrice` fields of the Limits Banding message into upper price limit and lower price limit statistics. Both statistics are published on every limits banding update, using tag 60-TransactTime as `ts_event`. **Reference timestamp (`ts_ref`)** CME includes tag 5796-TradingReferenceDate, the trading session date, with daily statistics. This field is normalized to `ts_ref` as a nanosecond-precision UNIX timestamp for consistency with other timestamps. Because the source only has date precision, users should avoid localizing `ts_ref` to their time zone. **Multiple records for the same date** CME often publishes multiple messages for the same trading session date (indicated by `ts_ref`) and daily statistic. The initial messages indicate preliminary values, and the final message for the trading session date indicates the final value. The final message is always the most accurate and the one you should use, when available. **Stat flags** Settlement price statistics normalize the [bit field](https://en.wikipedia.org/wiki/Bit_field) [tag 731-SettlPriceType](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Settlement+Price) to the `stat_flags` field. | Value | Decimal | Description | | `1 << 0` | 1 | Whether the settlement price is final, as opposed to preliminary. | | `1 << 1` | 2 | Whether the settlement price is actual, as opposed to theoretical. | | `1 << 2` | 4 | Some products have a different trading tick size than their clearing tick size. 1 if settling at the trading tick and 0 if at the clearing tick. | | `1 << 3` | 8 | Whether the price is an intraday settlement price and disseminated before the official end-of-day settlement calculation. | [The CME documentation on settlement prices](https://www.cmegroup.com/confluence/display/EPICSANDBOX/Settlement+Prices#SettlementPrices-SettlementatTradingTick/SettlementatClearingTick) expands on the meaning of each flag value. Settlement prices before 2015-11-20 follow different rules, described under [MDP 2 data](https://databento.com/docs/datasets/glbx-mdp3.md#mdp-2-data). ### Symbology Databento normalizes the `asset` field from tag 6937-Asset of the [Market Data Security Definition](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457672532/MDP+3.0+-+Security+Definition) message. The `asset` field is used when requesting data with [parent symbology](https://databento.com/docs/conventions/symbology.md#parent). Databento normalizes the `raw_symbol` field from tag 55-Symbol of the Security Definition message. This symbol contains the product code, followed by the [month code](https://www.cmegroup.com/month-codes.html), followed by the year. Previously, this field would only contain a 1-digit year, e.g. `ESZ3` for the E-mini S&P 500 December 2023 contract. In some recently listed contracts, CME has begun using a 2-digit year, e.g. `NGN25` for the Henry Hub Natural Gas July 2025 contract. ### Definition normalization **FX spots** CME lists foreign exchange spot instruments alongside futures. Databento sets `instrument_class` to `X` (FX spot) for instruments whose CFI code begins with `IF`. The `cfi` and `security_type` fields are passed through from CME unchanged. **Instrument attributes** CME publishes a [bit field](https://en.wikipedia.org/wiki/Bit_field) of instrument attributes in tag 872-InstAttribValue, normalized to the `inst_attrib_value` field. | Value | Description | | `1 << 0` | Whether the instrument is eligible for electronic matching. | | `1 << 1` | Whether the instrument is eligible for order cross. | | `1 << 2` | Whether the instrument is eligible for block trades. | | `1 << 3` | Whether the instrument is EFP eligible. | | `1 << 4` | Whether the instrument is EBF eligible. | | `1 << 5` | Whether the instrument is EFS eligible. | | `1 << 6` | Whether the instrument is EFR eligible. | | `1 << 7` | Whether the instrument is OTC eligible. | | `1 << 8` | Whether the instrument is eligible for iLink indicative mass quoting. | | `1 << 9` | Whether the instrument may have a negative strike price. | | `1 << 10` | Whether the outright instrument may have a negative price. | | `1 << 11` | Whether prices for the instrument are displayed as fractions. | | `1 << 12` | Whether the option is quoted in volatility. | | `1 << 13` | Whether the instrument is eligible for RFQ cross. | | `1 << 14` | Whether the outright instrument may have a price of zero. | | `1 << 15` | Whether the product is a decaying product. | | `1 << 16` | Whether the product is a variable product. | | `1 << 17` | Whether the product is a daily product. | | `1 << 18` | Whether the instrument accepts good-til orders. | | `1 << 19` | Whether the instrument is eligible for implied matching. | | `1 << 20` | Whether the instrument is eligible for triangulation. | | `1 << 21` | Whether the instrument is eligible for a variable cabinet price. | | `1 << 22` | Whether the instrument's book is inverted. | | `1 << 23` | Whether the instrument is all-or-none. | | `1 << 24` | Whether the instrument is SEF regulated. | | `1 << 25` | Whether the instrument is MTF regulated. | | `1 << 26` | Whether the instrument is an EFIX instrument. | | `1 << 27` | Whether the instrument is a hedge instrument. | [The CME MDP 3.0 documentation](https://www.cmegroup.com/confluence/display/EPICSANDBOX/CME+MDP+3.0+Market+Data) expands on the meaning of each attribute. `1 << 11` pertains to how a price is displayed and marks instruments CME quotes in the exchange's [fractional convention](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457092591/Fractional+Pricing), and pairs with the `main_fraction`, `sub_fraction`, and `price_display_format` fields. The [Displaying fractional prices](https://databento.com/docs/examples/instrument-definitions/fractional-prices.md) example renders prices using these fields. ### Spreads and combos In addition to futures and options on futures, this dataset also contains data for both exchange-listed spreads and [user-defined spreads (UDS)](https://www.cmegroup.com/confluence/display/EPICSANDBOX/User+Defined+Spread+-+UDS). Spreads and combinations consisting only of futures (such as exchange-listed calendar spreads) are considered part of the futures [parent symbols](https://databento.com/docs/conventions/symbology.md#parent). For example, `ESH4-ESZ4` is a calendar spread contained in `ES.FUT`. These spreads can have negative prices. Spreads and combinations containing options (such as butterfly or covered spreads) are considered part of the option [parent symbols](https://databento.com/docs/conventions/symbology.md#parent). For example, `UD:1V: VT 2533938` is a vertical option spread contained in `ES.OPT`. [The CME documentation on spreads and combinations](https://www.cmegroup.com/confluence/display/EPICSANDBOX/Spreads+and+Combinations+Available+on+CME+Globex) contains an exhaustive list of the different spread types. The `instrument_class` field in the definition schema can be used to identify spread instruments generically. The `SecuritySubType` field in the exchange documentation is normalized to the `secsubtype` field in the `definition` schema and can be used to identify the exchange-specific spread type. To identify the parent symbol of strategies and user-defined spreads, the `asset` field in the definition schema should be used. **Leg definitions** Databento publishes one definition record per leg of a spread or combination. The records for an instrument are identical apart from the fields beginning with `leg_`, which describe a single leg. `leg_count` is the number of legs and `leg_index` is the 0-based index of the leg the record describes. | Databento field | MDP 3.0 tag | Description | | `leg_count` | 555-NoLegs | The number of legs in the spread. | | `leg_index` | N/A | The 0-based index of the leg within the spread. | | `leg_instrument_id` | 602-LegSecurityID | The instrument ID of the leg instrument. | | `leg_side` | 624-LegSide | The side taken for the leg when buying the spread. | | `leg_ratio_qty_numerator` | 623-LegRatioQty | The quantity of the leg instrument per unit of the spread. | | `leg_ratio_qty_denominator` | N/A | Always 1 for this dataset. | | `leg_price` | 566-LegPrice | The tied price of the leg, used for hedged strategies. | | `leg_delta` | 1017-LegOptionDelta | The delta of the leg, used for hedged strategies. | | `leg_raw_symbol` | N/A | Taken from the leg instrument's own definition. | | `leg_instrument_class` | N/A | Taken from the leg instrument's own definition. | | `leg_underlying_id` | N/A | Taken from the leg instrument's own definition. | The `leg_ratio_price_numerator` and `leg_ratio_price_denominator` fields are not populated for this dataset. CME can publish the definition of a spread before the definitions of its legs. Until the leg instrument's own definition arrives, the fields taken from it are unset. Databento then republishes the affected leg records with `security_update_action` set to **M**odify. ### Status normalization Status records are normalized from [Market Data Security Status](https://cmegroupclientsite.atlassian.net/wiki/x/JbtAGw) messages. A single status message from CME may update the status of all instruments in a `group`, all instruments in a `group` and `asset`, or an individual instrument. To support requesting data for individual instruments, Databento normalizes all these messages into instrument-level status messages. When requesting status schema data for multiple instruments in the same `group` or `asset`, there will be status records that are identical minus the `instrument_id`. The state fields of the status schema: `is_trading`, `is_quoting`, and `is_short_sell_restricted` are based solely on the status updates from CME. Group-level status updates generate a status record for each instrument with a definition, regardless of activation or expiration. As a result, the status state fields (`is_trading` and `is_quoting`) may be incorrect before activation or after expiration. ### Matching Algorithms CME implements various [matching algorithms](https://databento.com/docs/schemas/instrument-definitions.md#matching-algorithm) across its universe of products, with **F**IFO being the most common. Specific details on the different algorithms can be found on the [Supported Matching Algorithms page](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457218479/Supported+Matching+Algorithms) on the CME website. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. ### MDP 2 data On 2017-05-21, CME introduced full granularity for every order event (MBOFD). Data prior to this date is based on the MDP 2, level-aggregated FIX protocol. This leads to some differences in the data. **Timestamps** MDP 2 data is sourced from FIX flat files. Because of this, capture timestamps (commonly available as `ts_recv`) are not available. The `ts_recv` field on all MDP 2 data is set to the same value as `ts_event`. Because of this, all records on MDP 2 data have the `F_BAD_TS_RECV` flag set. This is normal and expected. Nanosecond-resolution timestamps have been introduced by CME on 2015-11-20. Timestamps prior to that date are limited to millisecond resolution. | Databento Field | CME Tag | Description | | `ts_recv` | 52-SendingTime | The matching-engine-sending timestamp. | | `ts_event` | 52-SendingTime | The matching-engine-sending timestamp. | **MBO format** MDP 2 data predates the introduction of full-depth MBO by CME. Because of this, the MBO schema is not available. The highest granularity schema available for MDP 2 data is MBP-10. **CMBP-1 format** The CMBP-1, CBBO-1s, and CBBO-1m schemas are not available for MDP 2 data. **Settlement prices** Before 2015-11-20, CME sent the settlement price type in tag 286-OpenCloseSettlFlag instead of tag 731-SettlPriceType. For products not subject to settlement price rounding, a `stat_flags` value of 4 marks an unrounded preliminary price rather than settlement at the trading tick. Final settlement prices before 2015-11-20 come from the weekly security definition replay CME sent each Sunday, which covers the last trading session of the week. From 2015-11-20, CME sent tag 731-SettlPriceType on daily updates, which include final settlement prices. **Status schema** There's significantly less status data prior to 2015; this is expected. Before November 2015, status changes from the normal trading schedule did not result in a status message. --- Source: https://databento.com/docs/datasets/equs-mini # Databento US Equities Mini **Dataset ID**: [EQUS.MINI]($PLATFORM_HOSTNAME/catalog/us-equities#EQUS.MINI) Databento US Equities Mini is a derived top-of-book dataset, sourced from the proprietary market data feeds of both alternative trading systems (ATS) and RegNMS venues. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`); in this dataset, `raw_symbol` and the `nasdaq` stype are equivalent. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Schemas The following schemas are available for this dataset: [MBP-1](https://databento.com/docs/schemas/mbp-1.md), [BBO](https://databento.com/docs/schemas/bbo.md), [TBBO](https://databento.com/docs/schemas/tbbo.md), [Trades](https://databento.com/docs/schemas/trades.md), [OHLCV-1s](https://databento.com/docs/schemas/ohlcv.md), [OHLCV-1m](https://databento.com/docs/schemas/ohlcv.md), [OHLCV-1h](https://databento.com/docs/schemas/ohlcv.md), [OHLCV-1d](https://databento.com/docs/schemas/ohlcv.md), and [Definitions](https://databento.com/docs/schemas/instrument-definitions.md). ### Aggregated BBO Databento US Equities Mini is an aggregated dataset, quotes are not provided for the individual component venues. A single aggregated BBO is printed for each instrument. To provide this aggregated BBO, Databento calculates the best bid and best ask prices across the component venues of the feed, adding together all quantity (including odd lots) at the price of the BBO regardless of venue. We do not aggregate the order counts across the component venues, so `bid_ct_00` and `ask_ct_00` will always be 0. ### Venues To qualify as a derived dataset, trade and BBO data are anonymized from the original venues' data. Due to this, the `publisher_id` of a trade does not identify the original venue of the trade, and will always have the value `EQUS.MINI.EQUS`. The `sequence` field is set to 0 on all messages. ### Aggregated OHLCV The OHLCV schemas for Databento US Equities Mini aggregate prices and volume across all the component venues. A single aggregated OHLCV message is printed per instrument, per interval. ### Timestamps Messages received by Databento from component venues will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The `ts_in_delta` field is not populated for this dataset and will always be set to 0. | Databento Field | Description | | `ts_recv` | The capture-server-received timestamp. | | `ts_event` | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/equs-summary # Databento US Equities Summary **Dataset ID**: [EQUS.SUMMARY]($PLATFORM_HOSTNAME/catalog/us-equities#EQUS.SUMMARY) Nasdaq disseminates consolidated volume and end-of-day statistics via their Nasdaq NLS+ feed. Databento receives these over UDP multicast in our NY4 data center. PDF specifications are available on Nasdaq's website for the [NLS+](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NLSPlus-cloud.pdf) feed. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`); in this dataset, `raw_symbol` and the `nasdaq` stype are equivalent. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Schemas Only the `ohlcv-1d`, `statistics`, and `definition` schemas are available. Databento normalizes the consolidated volume provided by the Nasdaq NLS+ feed on every trade into the `statistics` schema. This volume is consolidated across all US exchanges. Databento normalizes the end-of-day summary provided by the Nasdaq NLS+ feed into the `ohlcv-1d` schema. Databento normalizes *Stock Directory* messages into the `definition` schema. ### Definition normalization Definition records are sourced from NLS+ *Stock Directory* messages. | Databento Field | NLS+ Field | Description | | `exchange` | Market Category | Normalized to the MIC code of the listing venue. All Nasdaq-listed instruments will normalize to `XNAS`. | | `security_type` | Issue Classification | Identifies the security class for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix E [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NLSPlusSpecification3.0.pdf). | | `secsubtype` | Issue Sub-Type | Identifies the security sub-type for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix F [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NLSPlusSpecification3.0.pdf). | ### Timestamps Nasdaq NLS+ messages received by Databento have at most one timestamp, which has nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The extra field `ts_in_delta` does not convey any additional information for this dataset, and is equivalent to `ts_recv - ts_event`. | Databento Field | TotalView Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### End-of-day Summary Nasdaq NLS+ disseminates an end-of-day summary via their API at three different times: 1. Once at approximately 16:15 ET 2. Once at approximately 17:00 ET 3. Once at approximately 20:15 ET Each of those summaries reflects the summary at the publishing time. Databento only provides the last summary (from 20:15 ET) on the `ohlcv-1d` schema, which includes the total post-market volume, published as a single OHLCV-1d message per date. This can lead to discrepancies when comparing against other data sources that use the other earlier summaries. The first two preliminary summaries are disseminated by Databento under the `statistics` schema. The `stat_flags` field is set to 1 for the first summary and 2 for the second summary. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xeur-eobi # Eurex **Dataset ID**: [XEUR.EOBI]($PLATFORM_HOSTNAME/catalog/xeur/XEUR.EOBI) Eurex Exchange disseminates full depth-of-book data for their exchange through their EOBI (Enhanced Order Book Interface) Feed, which Databento receives over UDP multicast in our FR2 data center. The latest [EOBI Specification](https://www.eurex.com/resource/blob/4417682/52249af896465e10295f4203d25fef91/data/T7_R.13.1_%20EOBI_Manual_Version_3.pdf) is available from Eurex website. ### Trading hours The [regular trading hours](https://www.eurex.com/ex-en/trade/trading-hours) for Eurex are: * 08:00 CET to 17:30 CET (European hours) * 17:30 CET to 22:00 CET (U.S. hours) The [extended trading hours](https://www.eurex.com/ex-en/trade/trading-hours) for Eurex are: * 01:00 CET to 08:00 CET (Asian hours) Extended (Asian) trading hours only include a [subset of instruments](https://www.eurex.com/ex-en/trade/trading-hours/thx), including European indices, interest rate futures, and cryptocurrency futures. ### Timestamps Eurex EOBI messages received by Databento include multiple timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). | Databento Field | Eurex EOBI Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | RequestTime, TransactTime | RequestTime (Gateway request in timestamp) when available. TransactTime otherwise. | | `ts_in_delta` | TransactTime from PacketHeader | The exchange matching engine processing time. | EOBI `RequestTime` field is not available for the off-market trades (EOBI message `TES Trade Report`), manual and auction trades (EOBI message `Trade Report`), and cross request (EOBI message `Cross Request`). Thus `ts_event` is populated with `TransactTime` field, which corresponds to the time when the transaction was processed by the exchange's matching engine. The `ts_event` for passive trade messages (EOBI messages `Partial Order Execution` and `Full Order Execution`) is populated with the `RequestTime` from the aggressive order. ### Definition normalization Definition records are sourced daily from [Reference Data Interface](https://www.eurex.com/resource/blob/4417678/b4ce595ef44d7959f2aca306eaab85a8/data/T7_R.13.1_%20EMDI_MDI_RDI_Manual_Version_3.pdf) snapshot. Definition records for intra-day instruments are sourced from EOBI incremental feed. | Databento Field | Eurex RDI Field | Description | | `appl_id` | PartitionId | Identifies the instrument's partition. | | `asset` | MarketSegment | Identifies the instrument's market segment. This field is used by parent and continuous [symbology](https://databento.com/docs/conventions/symbology.md). | | `group` | MarketId | Market identifier code specified in ISO 10383. | | `raw_symbol` | SecurityDesc | Instrument identifier (See **Symbology** section below). | | `secsubtype` | ProductComplex | Identifies the type of instrument (See **Symbology** section below). | | `underlying` | MarketSegment | The market segment of the underlying (if there is an underlying). | ### Symbology Raw symbols are sourced from `SecurityDesc` string field. The semantics varying according to the type of instrument (`secsubtype` in the normalized message). The table below describes each instrument type, based on the venue's [documentation](https://www.eurex.com/resource/blob/4417678/b4ce595ef44d7959f2aca306eaab85a8/data/T7_R.13.1_%20EMDI_MDI_RDI_Manual_Version_3.pdf). | Value | Instrument type | Format | | `1` | Simple Instrument | `PROD SI YYYYMMDD SM ES C/P StrikePrice VN` | | `2` | Standard option strategy | `PROD.O.YYMMDD.IST.SEQ-NO` | | `3` | Non-standard option strategy | `PROD.O.YYMMDD.IST.SEQ-NO` | | `4` | Volatility strategy | `PROD.V.YYMMDD.IST.SEQ-NO` | | `5` | Futures spread | `PROD.S.MONYY.MONYY.SPD` | | `6` | Inter-product spread | `PROD.I.YYMMDD.IST.SEQ-NO` | | `7` | Standard futures strategy | `PROD.F.YYMMDD.IST.SEQ-NO` | | `8` | Pack and bundle | `PROD.B.MONYY.IST` | | `9` | Strip | `PROD.T.YYMMDD.SEQ-NO` | | `10` | Flexible instrument | `PROD FI YYYYMMDD SM ES C/P StrikePrice VN` for options. `PROD FI YYYYMMDD SM` for futures. | | `11` | Commodity strip | `PROD.C.YYMMDD.IST.SEQ-NO` | | `12` | Scaled simple instrument | `PROD SI YYYYMMDD SM ES C/P StrikePrice VN QSF` | | `13` | Non-standard volatility strategy | `PROD.D.YYYYMMDD.SEQ-NO` | | `14` | Total return future strategy | `PROD.R.YYYYMMDD.IST.SEQ-NO` | The acronyms from the table above correspond to: * `SM`: Settlement method (`CS` for cash, `PS` for Physical). * `ES`: Exercise style (`EU` for European, `AM` for American). * `VN`: Version number of an option. The version changes as a result of corporate events. * `IST`: Standard strategy type for complex instrument, as detailed in the [Trading parameters document](https://www.eurex.com/ex-en/data/trading-files/product-information). * `SEQ-NO`: Unique sequence number of 6 characters per product. * `QSF`: Quantity scaling factor of scaled simple instruments. * `YYYYMMDD`: Contract date. * `MONYY`: Contract month-year. * `YYMMDD`: Creation date. * `O`, `N`, `V`, `D`, `F`: Identify different strategy types. * `I`: Identify inter-product spreads instruments. * `T`: Identify strip instruments. * `C`: Identify commodity strip instruments. ### Expirations Eurex only provides date-granularity expiration data in `MaturityDate` field from the reference data. As a result, the `expiration` field in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) will always be at UTC midnight, even when the instrument expires during or after the trading day. Eurex also provides the `MaturityMonthYear` field in the reference data, which corresponds to the instrument expiration month, and may be in a different month than the `MaturityDate` field. The `MaturityMonthYear` field is mapped to the fields `maturity_year` and `maturity_month` in the definition schema (`maturity_day` is not populated). ### Statistics normalization Databento provides the following statistics: * Opening price: First trade price of the trading session. * Trading session high price: Highest trade price of the trading session. * Trading session low price: Lowest trade price of the trading session. * Close price: Last trade price of the trading session. * Cleared volume: Total on-exchange trading volume. * Settlement price: Published during the day, as soon as they are available. Different products have different schedules. * [Adjusted Open Interest](https://www.eurex.com/resource/blob/3974454/ff664ad26b898adf33e78a310bcce01e/data/C7%20Derivatives%20Clearing%20Functional%20Reference%20Guide.pdf): Published daily around 13:30 CE(S)T, it considers the previous day's open interest and adjusts it by subtracting the current day's closed contracts. Trading session high price, low price, close price, and cleared volume are published daily around 22:00:00 UTC, and only account for on-exchange trades. On-exchange trades manually entered by [T7 Market Supervision](https://www.eurex.com/resource/blob/4417682/52249af896465e10295f4203d25fef91/data/T7_R.13.1_%20EOBI_Manual_Version_3.pdf) do not affect price statistics. **Stat flags** Settlement price statistics normalize the field [tag 731-SettlPriceType](https://www.eurex.com/resource/blob/4332354/8a3ed0147bbbb5e5f7452fad509289b5/data/T7_R.13.1_EMDS_-_Trade_Prices-_Settlement_Prices_and_Open_Interest_Data_Manual_Version_1.pdf) to the `stat_flags` field. | Value | Decimal | Description | | `1 << 0` | 1 | Final settlement | | `1 << 1` | 2 | Theoretical (daily) settlement | ### Off-market trades Trades reported in `TES Trade Report` EOBI message are considered off-market. Off-market trades with the price field set to `UNDEF_PRICE` contain the non-disclosed trade volume during the day. Off-market trades have their `publisher_id` set to `XEUR.EOBI.XOFF` to distinguish them from on-exchange trades, which have their `publisher_id` set to `XEUR.EOBI.XEUR`. This distinction is also reflected on the OHLCV schemas - if an interval contains both on-market and off-market trades for an instrument, two OHLCV messages will be published for that instrument on that interval, one with `XEUR.EOBI.XEUR` as the `publisher_id` and one with `XEUR.EOBI.XOFF`. ### Order Identifier EOBI does not provide an order identifier (`order_id`) field. Instead, it relies on two fields to uniquely identify an order of a given instrument: `TrdRegTSTimePriority` (priority timestamp) and `Side`. Databento normalizes the `order_id` field in [MBO schema](https://databento.com/docs/schemas/mbo.md) by combining both priority timestamp and side. To extract the priority timestamp from the `order_id`, the most-significant bit can be masked, as below: ```python priority_timestamp = order_id & 0x7FFFFFFFFFFFFFFF ``` ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xeee-eobi # European Energy Exchange **Dataset ID**: [XEEE.EOBI]($PLATFORM_HOSTNAME/catalog/xeee/XEEE.EOBI) European Energy Exchange (EEX) disseminates full depth-of-book data for their exchange through their EOBI (Enhanced Order Book Interface) feed, which Databento receives over UDP multicast in our FR2 data center. The `XEEE.EOBI` dataset does not disseminate data for EEX spot market, only for derivatives market. The latest [EOBI Specification](https://www.eurex.com/resource/blob/4417682/52249af896465e10295f4203d25fef91/data/T7_R.13.1_%20EOBI_Manual_Version_3.pdf) is available from Eurex website. ### Trading hours The base [trading hours](https://www.eex.com/fileadmin/EEX/Downloads/Trading/Trading_Hours/20250701_Trading_Hours_on_EEX_Derivatives_Markets_.pdf) on the EEX derivatives take place on weekdays from 08:00:00 CET to 17:00:00 CET. These trading hours can vary depending on the type of market. EEX [Japanese power futures](https://www.eex.com/en/markets/power/japanese-power-market) trading hours are from 16:00 JST to 18:00 JST (08:00 CET to 10:00 CET). [EEX](https://www.eex.com/en/trading-resources/trading-information/calendar) provides holiday calendars across different market types. ### Timestamps EEX EOBI messages received by Databento include multiple timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). | Databento Field | EEX EOBI Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | RequestTime, TransactTime | RequestTime (Gateway request in timestamp) when available. TransactTime otherwise. | | `ts_in_delta` | TransactTime from PacketHeader | The exchange matching-engine processing time. | EOBI `RequestTime` field is not available for the off-market trades (EOBI message `TES Trade Report`), manual and auction trades (EOBI message `Trade Report`), and cross request (EOBI message `Cross Request`). Thus `ts_event` is populated with `TransactTime` field, which corresponds to the time when the transaction was processed by the exchange's matching engine. The `ts_event` for passive trade messages (EOBI messages `Partial Order Execution` and `Full Order Execution`) is populated with the `RequestTime` from the aggressive order. ### Definition normalization Definition records are sourced daily from [Reference Data Interface](https://www.eurex.com/resource/blob/4417678/b4ce595ef44d7959f2aca306eaab85a8/data/T7_R.13.1_%20EMDI_MDI_RDI_Manual_Version_3.pdf) snapshot. Definition records for intra-day instruments are sourced from EOBI incremental feed. | Databento Field | EEX RDI Field | Description | | `appl_id` | PartitionId | Identifies the instrument's partition. | | `asset` | MarketSegment | Identifies the instrument's market segment. This field is used by parent and continuous [symbology](https://databento.com/docs/conventions/symbology.md). | | `group` | MarketId | Market identifier code specified in ISO 10383. | | `raw_symbol` | SecurityDesc | Instrument identifier (See **Symbology** section below). | | `secsubtype` | ProductComplex | Identifies the type of instrument (See **Symbology** section below). | | `underlying` | MarketSegment | The market segment of the underlying (if there is an underlying). | The `MarketSegment` field from EEX message is truncated to fit into `asset` field in `InstrumentDefMsg` (size of `ASSET_CSTR_LEN`). ### Symbology Raw symbols are sourced from `SecurityDesc` string field. The semantics varying according to the type of instrument (`secsubtype` in the normalized message). The table below describes each instrument type, based on the venue's [documentation](https://www.eurex.com/resource/blob/4417678/b4ce595ef44d7959f2aca306eaab85a8/data/T7_R.13.1_%20EMDI_MDI_RDI_Manual_Version_3.pdf). | Value | Instrument type | Format | | `1` | Simple Instrument | `PROD SI YYYYMMDD SM ES C/P StrikePrice VN` | | `2` | Standard option strategy | `PROD.O.YYMMDD.IST.SEQ-NO` | | `3` | Non-standard option strategy | `PROD.O.YYMMDD.IST.SEQ-NO` | | `4` | Volatility strategy | `PROD.V.YYMMDD.IST.SEQ-NO` | | `5` | Futures spread | `PROD.S.MONYY.MONYY.SPD` | | `6` | Inter-product spread | `PROD.I.YYMMDD.IST.SEQ-NO` | | `7` | Standard futures strategy | `PROD.F.YYMMDD.IST.SEQ-NO` | | `8` | Pack and bundle | `PROD.B.MONYY.IST` | | `9` | Strip | `PROD.T.YYMMDD.SEQ-NO` | | `10` | Flexible instrument | `PROD FI YYYYMMDD SM ES C/P StrikePrice VN` for options. `PROD FI YYYYMMDD SM` for futures. | | `11` | Commodity strip | `PROD.C.YYMMDD.IST.SEQ-NO` | | `12` | Scaled simple instrument | `PROD SI YYYYMMDD SM ES C/P StrikePrice VN QSF` | | `13` | Non-standard volatility strategy | `PROD.D.YYYYMMDD.SEQ-NO` | | `14` | Total return future strategy | `PROD.R.YYYYMMDD.IST.SEQ-NO` | The acronyms from the table above correspond to: * `SM`: Settlement method (`CS` for cash, `PS` for Physical). * `ES`: Exercise style (`EU` for European, `AM` for American). * `VN`: Version number of an option. The version changes as a result of corporate events. * `IST`: Standard strategy type for complex instrument, as detailed in the [Trading parameters document](https://www.eurex.com/ex-en/data/trading-files/product-information). * `SEQ-NO`: Unique sequence number of 6 characters per product. * `QSF`: Quantity scaling factor of scaled simple instruments. * `YYYYMMDD`: Contract date. * `MONYY`: Contract month-year. * `YYMMDD`: Creation date. * `O`, `N`, `V`, `D`, `F`: Identify different strategy types. * `I`: Identify inter-product spreads instruments. * `T`: Identify strip instruments. * `C`: Identify commodity strip instruments. ### Expirations EEX only provides date-granularity expiration data in `MaturityDate` field from the reference data. As a result, the `expiration` field in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) will always be at UTC midnight, even when the instrument expires during or after the trading day. EEX also provides the `MaturityMonthYear` field in the reference data, which corresponds to the instrument expiration month, and may be in a different month than the `MaturityDate` field. The `MaturityMonthYear` field is mapped to the fields `maturity_year` and `maturity_month` in the definition schema (`maturity_day` is not populated). ### Statistics normalization Databento provides the following statistics: * Opening price: First trade price of the trading session. * Trading session high price: Highest trade price of the trading session. * Trading session low price: Lowest trade price of the trading session. * Close price: Last trade price of the trading session. * Cleared volume: Total on-exchange trading volume. * Final Settlement price: Published daily around 19:25:00 CET. * Open Interest: Published daily around 21:10:00 CET. Settlement price and open interest are only provided for options and futures outrights that are currently active and tradable in EEX's trading system. Settlement price and open interest are not provided for Freight Futures and Options. Trading session high price, low price, close price, and cleared volume are published daily around 22:00:00 UTC, and only account for on-exchange trades. On-exchange trades manually entered by [T7 Market Supervision](https://www.eurex.com/resource/blob/4417682/52249af896465e10295f4203d25fef91/data/T7_R.13.1_%20EOBI_Manual_Version_3.pdf) do not affect price statistics. ### Off-market trades Trades reported in `TES Trade Report` EOBI message are considered off-market. Off-market trades with the price field set to `UNDEF_PRICE` contain the non-disclosed trade volume during the day. Off-market trades have their `publisher_id` set to `XEEE.EOBI.XOFF` to distinguish them from on-exchange trades, which have their `publisher_id` set to `XEEE.EOBI.XEEE`. This distinction is also reflected on the OHLCV schemas - if an interval contains both on-market and off-market trades for an instrument, two OHLCV messages will be published for that instrument on that interval, one with `XEEE.EOBI.XEEE` as the `publisher_id` and one with `XEEE.EOBI.XOFF`. ### Order Identifier EOBI does not provide an order identifier (`order_id`) field. Instead, it relies on two fields to uniquely identify an order of a given instrument: `TrdRegTSTimePriority` (priority timestamp) and `Side`. Databento normalizes the `order_id` field in [MBO schema](https://databento.com/docs/schemas/mbo.md) by combining both priority timestamp and side. To extract the priority timestamp from the `order_id`, the most-significant bit can be masked, as below: ```python priority_timestamp = order_id & 0x7FFFFFFFFFFFFFFF ``` ### Data prior to 2025-03-09 Data prior to 2025-03-09 was not normalized from EEX full depth-of-book captures, but from [EEX Market Data API](https://www.eex.com/fileadmin/EEX/Downloads/Market_Data/EEX_Group_DataSource/API/EEX_Market_Data_-_API_User_Guide_v021.pdf). The `definition`, `statistics`, `ohlcv-1d`, and `trades` schemas are available prior to this date for options and futures. The reported volume (in `statistics` and `ohlcv-1d`) and `size` (in `trades`) correspond to the number of traded contracts/lots reported by [EEX Market Data API](https://www.eex.com/fileadmin/EEX/Downloads/Market_Data/EEX_Group_DataSource/API/EEX_Market_Data_-_API_User_Guide_v021.pdf). The `raw_instrument_id` field from `definition` schema is not populated, as it is not provided by the venue. The `group` field is populated according to the [EEX instrument's commodity class](https://www.eex.com/fileadmin/EEX/Downloads/Market_Data/EEX_Group_DataSource/API/EEX_Market_Data_-_API_User_Guide_v020.pdf): `POWER`, `NATGAS`, `AGRI`, `EMISSIONS`, and `FREIGHT`. The following `statistics` are published: Opening price, trading session high price, trading session low price, close price, cleared volume, open interest, and settlement price. Cleared volume accounts for on and off-exchange traded volume. Implied trades have no price reported by the venue, thus price will be `UNDEF_PRICE` (`0x7fffffffffffffff`, `9223372036854775808`). Due to not being generated from packet captures, both `ts_event` and `ts_recv` fields are set to midnight UTC, except: * For `trades`, where both `ts_event` and `ts_recv` report the trade timestamp. * For `ohlcv-1d`, where `ts_event` marks the start of the bar. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/ndex-impact # ICE Endex **Dataset ID**: [NDEX.IMPACT]($PLATFORM_HOSTNAME/catalog/ndex/NDEX.IMPACT) ICE disseminates FOD (full order depth), MBP (market by price), TOB (top-of-book) and market statistics data for futures, spot contracts and options on futures via their iMpact feed, which Databento receives over UDP multicast in our Aurora DC3 data center. For futures and spot, ICE publishes both FOD and MBP-5. Databento only normalizes the FOD messages. For options, ICE publishes MBP-10, MBP-5 and top-of-book. Databento only normalizes the MBP-10 messages. This means that for options, only the top 10 levels of the book are available even in the MBO schema. For futures the full market is shown. ### Timestamps ICE iMpact messages received by Databento will generally include two timestamps with microsecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | ICE Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | TransactDateTime and SequenceWithinMillis | The matching-engine-processed timestamp with sub-microsecond sequence number. | | `ts_in_delta` | SentDateTime | The matching-engine-sending timestamp. | ICE provides microsecond granularity timestamps along with sub-microsecond sequence numbers for ordering that are normalized as nanosecond timestamps for consistency in ordering with other datasets. More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Matching Engine Location ICE Endex has matching engines both in Chicago and Basildon. Databento captures the data for both matching engines in our Aurora DC3 location. ### MBO normalization For options, the ICE iMpact feed only includes market-by-price, which matches Databento's [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md). We do not offer these messages in our [MBO schema](https://databento.com/docs/schemas/mbo.md). They are only available on MBP-10 and other downstream schemas. Each ICE "Trade Message" is normalized into a single MBO record with the **T**rade action, and the message's "TradeID" field is used as the `order_id`. ICE does not disseminate information about the passive side of the trade. The corresponding **F**ill record will have `side` equal to `N` and the same `order_id` value as the **T**rade record. When ICE does not specify the aggressing side, **T**rade records will have `side` equal to `N`. ### System priced legs A system priced leg is the result of a trade in the outright spread. These trades, as published by the exchange, can have prices that are 10% or more off from the best bid and offer. ICE recommends that the price of these trades isn't counted towards market statistics. Databento normalizes system priced legs as a **T**rade record with `side` equal to `N` and an `UNDEF_PRICE` (`0x7fffffffffffffff`, 9223372036854775808) value for the price. These trades are counted towards OHLCV volume, but not towards the open, high, low or close values. This means that OHLCV messages can have some (or even all) price values as `UNDEF_PRICE`. ### Off-market trades Trades marked by the exchange as off-market have their `publisher_id` set to `NDEX.IMPACT.XOFF` to distinguish them from on-exchange trades. This distinction is also reflected on the OHLCV schemas - if an interval contains both on-market and off-market trades for an instrument, two OHLCV messages will be published for that instrument on that interval, one with `NDEX.IMPACT.NDEX` as the `publisher_id` and one with `NDEX.IMPACT.XOFF`. ### Statistics **Stat flags** The `IsOfficial` field in the ICE iMpact Settlement Price Message and `IsSettlePriceOfficial` in the Market Snapshot Message are normalized to the first (least-significant) bit of `stat_flags` where the bit is set if `IsOfficial` is `Y`. This is consistent with the behavior for [CME Globex MDP 3.0](https://databento.com/docs/datasets/glbx-mdp3.md#statistics). ### Symbology ICE uses separate symbology for outrights and strategies. **Outrights** For outrights -- individual futures and options -- raw symbols are sourced from `ContractSymbol` field. If `ContractSymbolExtra` is present, it's appended to the `ContractSymbol`. Outright contract symbols are composed of several parts indicating different aspects of the instrument. For example, the Dutch TTF Natural Gas August 2024 futures contract's symbol is `TFM FMQ0024!`. This can be broken down as follows: | Component | Name | Description | | `TFM ` | Contract code | The right space-padded four-character [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | `F` | Contract type | Indicates whether the instrument is a future (`F`), swap, or other type of contract. | | `M` | Term | The contract term, usually month (`M`). | | `Q` | Month | The expiration month, e.g. `Q` for August. | | `00` | Day | The effective date of the contract. `00` for monthly contracts. | | `24` | Year | The two-digit expiration year. | | `!` | Switch character | Indicates additional distinguishing information about the contract. `!` indicates no additional data. | Options begin with their underlying future's symbol followed by an underscore (`_`) and the option block. Using the Dutch TTF Natural Gas August 2024 87.00 call `TFO FMQ0024_OMCE0000045002072624` as an example, the option block breaks down as: | Component | Name | Description | | `O` | Option indicator | Will always be `O` for options. | | `M` | Option type | The option term, e.g. `M`onthly or `W`eekly. | | `C` | Put or call | Whether the option is a `C`all or `P`ut. | | `E` | Option style | The exercise style of the option, e.g. `E` for European. | | `000004500` | Strike | The zero-padded nine-digit strike price. | | `2` | Strike decimal | The number of decimal places in the strike price. Applying two decimal places to 4500 gives a strike price of 45.00. | | `072624` | Expiration date | The MMDDYY expiration date of the option. `TFO` options expire before the contract month, in this case the August options expire July 26, 2024. | Refer to ICE's [Instrument Naming Convention document](https://www.ice.com/publicdocs/technology/Instrument_Naming_Convention.pdf) for a full explanation of the format of contract symbols. Because each dataset has a separate symbology namespace, Databento omits the MIC code from the fully-qualified symbol in ICE's examples. **Spots** Endex spots have neither a `ContractSymbol` nor a `StrategySymbol`, instead raw symbols are sourced from `MarketDesc`. **Strategies** For spreads and strategies where ICE does not define a `ContractSymbol`, raw symbols are sourced from `StrategySymbol`. Strategy symbols are composed of several parts, however, due to the use of numeric identifiers, strategy raw symbols are harder to interpret than those for outrights. The Dutch TTF Natural Gas year calendar strip `TFO 800 108281469` can be broken down as follows: | Component | Name | Description | | `TFO ` | Contract code | The [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | `800` | Strategy code | A left space-padded three-digit number identifying the type of strategy, e.g. `900` for year calendar strip. Refer to the table in [this document](https://www.ice.com/publicdocs/technology/ICE_Strategy_Code_Reference_Manual.pdf) for a complete list of strategy codes. | | ` 108281469` | Market ID | A left space-padded 10-digit unique identifier. The same as `instrument_id`. | To inspect spread legs, including intercommodity futures spreads, use the leg fields in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) instead of parsing the raw strategy symbol. See [Finding ICE intercommodity spreads](https://databento.com/docs/examples/instrument-definitions/ice-crack-spreads.md). ### Definitions - `MarketTypeId` is normalized to `market_segment_id` - `MICCode` is normalized to `exchange` when it is provided: only for futures and future strategies **Expirations** ICE only provides date-granularity expiration data in `OptionsExpirationDate` and `ScreenLastTrade` fields. As a result, the `expiration` field in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) will always be at UTC midnight, even when the option expires during or after the trading day. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/ifeu-impact # ICE Europe Commodities **Dataset ID**: [IFEU.IMPACT]($PLATFORM_HOSTNAME/catalog/ifeu/IFEU.IMPACT) ICE disseminates FOD (full order depth), MBP (market by price), TOB (top-of-book) and market statistics data for futures and options on futures via their iMpact feed, which Databento receives over UDP multicast in our Aurora DC3 data center. For futures, ICE publishes both FOD and MBP-5. Databento only normalizes the FOD messages. For options, ICE publishes MBP-10, MBP-5 and top-of-book. Databento only normalizes the MBP-10 messages. This means that for options, only the top 10 levels of the book are available even in the MBO schema. For futures the full market is shown. ### Timestamps ICE iMpact messages received by Databento will generally include two timestamps with microsecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | ICE Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | TransactDateTime and SequenceWithinMillis | The matching-engine-processed timestamp with sub-microsecond sequence number. | | `ts_in_delta` | SentDateTime | The matching-engine-sending timestamp. | ICE provides microsecond granularity timestamps along with sub-microsecond sequence numbers for ordering that are normalized as nanosecond timestamps for consistency in ordering with other datasets. More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Matching Engine Location ICE Europe Commodities has matching engines both in Chicago and Basildon. Currently, the products offered by Databento all trade in the ICE silo, located in 350 E Cermak, Chicago. Databento captures the data in our Aurora DC3 location. ### MBO normalization For options, the ICE iMpact feed only includes market-by-price, which matches Databento's [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md). We do not offer these messages in our [MBO schema](https://databento.com/docs/schemas/mbo.md). They are only available on MBP-10 and other downstream schemas. Each ICE "Trade Message" is normalized into a single MBO record with the **T**rade action, and the message's "TradeID" field is used as the `order_id`. ICE does not disseminate information about the passive side of the trade. The corresponding **F**ill record will have `side` equal to `N` and the same `order_id` value as the **T**rade record. When ICE does not specify the aggressing side, **T**rade records will have `side` equal to `N`. ### System priced legs A system priced leg is the result of a trade in the outright spread. These trades, as published by the exchange, can have prices that are 10% or more off from the best bid and offer. ICE recommends that the price of these trades isn't counted towards market statistics. Databento normalizes system priced legs as a **T**rade record with `side` equal to `N` and an `UNDEF_PRICE` (`0x7fffffffffffffff`, 9223372036854775808) value for the price. These trades are counted towards OHLCV volume, but not towards the open, high, low or close values. This means that OHLCV messages can have some (or even all) price values as `UNDEF_PRICE`. ### Off-market trades Trades marked by the exchange as off-market have their `publisher_id` set to `IFEU.IMPACT.XOFF` to distinguish them from on-exchange trades. This distinction is also reflected on the OHLCV schemas - if an interval contains both on-market and off-market trades for an instrument, two OHLCV messages will be published for that instrument on that interval, one with `IFEU.IMPACT.IFEU` as the `publisher_id` and one with `IFEU.IMPACT.XOFF`. ### Statistics **Stat flags** The `IsOfficial` field in the ICE iMpact Settlement Price Message and `IsSettlePriceOfficial` in the Market Snapshot Message are normalized to the first (least-significant) bit of `stat_flags` where the bit is set if `IsOfficial` is `Y`. This is consistent with the behavior for [CME Globex MDP 3.0](https://databento.com/docs/datasets/glbx-mdp3.md#statistics). ### Symbology ICE uses separate symbology for outrights and strategies. **Outrights** For outrights -- individual futures and options -- raw symbols are sourced from `ContractSymbol` field. If `ContractSymbolExtra` is present, it's appended to the `ContractSymbol`. Outright contract symbols are composed of several parts indicating different aspects of the instrument. For example, the Brent August 2024 futures contract's symbol is `BRN FMQ0024!`. This can be broken down as follows: | Component | Name | Description | | `BRN ` | Contract code | The right space-padded four-character [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | `F` | Contract type | Indicates whether the instrument is a future (`F`), swap, or other type of contract. | | `M` | Term | The contract term, usually month (`M`). | | `Q` | Month | The expiration month, e.g. `Q` for August. | | `00` | Day | The effective date of the contract. `00` for monthly contracts. | | `24` | Year | The two-digit expiration year. | | `!` | Switch character | Indicates additional distinguishing information about the contract. `!` indicates no additional data. | Options begin with their underlying future's symbol followed by an underscore (`_`) and the option block. Using the Brent August 2024 87.00 call `BRN FMQ0024_OMCA0000087002062524` as an example, the option block breaks down as: | Component | Name | Description | | `O` | Option indicator | Will always be `O` for options. | | `M` | Option type | The option term, e.g. `M`onthly or `W`eekly. | | `C` | Put or call | Whether the option is a `C`all or `P`ut. | | `A` | Option style | The exercise style of the option, e.g. `A` for American. | | `000008700` | Strike | The zero-padded nine-digit strike price. | | `2` | Strike decimal | The number of decimal places in the strike price. Applying two decimal places to 8700 gives a strike price of 87.00. | | `062524` | Expiration date | The MMDDYY expiration date of the option. `BRN` options expire before the contract month, in this case the August options expire June 25, 2024. | Refer to ICE's [Instrument Naming Convention document](https://www.ice.com/publicdocs/technology/Instrument_Naming_Convention.pdf) for a full explanation of the format of contract symbols. Because each dataset has a separate symbology namespace, Databento omits the MIC code from the fully-qualified symbol in ICE's examples. **Strategies** For spreads and strategies where ICE does not define a `ContractSymbol`, raw symbols are sourced from `StrategySymbol`. Strategy symbols are composed of several parts, however, due to the use of numeric identifiers, strategy raw symbols are harder to interpret than those for outrights. The Brent options iron condor `BRN 40 30775617` can be broken down as follows: | Component | Name | Description | | `BRN ` | Contract code | The [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | ` 40` | Strategy code | A left space-padded three-digit number identifying the type of strategy, e.g. `40` for iron condor. Refer to the table in [this document](https://www.ice.com/publicdocs/technology/ICE_Strategy_Code_Reference_Manual.pdf) for a complete list of strategy codes. | | ` 30775617` | Market ID | A left space-padded 10-digit unique identifier. The same as `instrument_id`. | To inspect spread legs, including intercommodity futures spreads, use the leg fields in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) instead of parsing the raw strategy symbol. See [Finding ICE intercommodity spreads](https://databento.com/docs/examples/instrument-definitions/ice-crack-spreads.md). **Databento MBO snapshot** ICE Europe Commodities iMpact daily [trading hours](https://www.ice.com/trading-hours) cross 00:00:00 UTC. To make it easier to work with historical MBO data, Databento includes an [order book snapshot](https://databento.com/docs/conventions/mbo-snapshot.md) at 00:00:00 UTC each weekday (Monday-Friday). ### Definitions - `MarketTypeId` is normalized to `market_segment_id` - `MICCode` is normalized to `exchange` when it is provided: only for futures and future strategies **Expirations** ICE only provides date-granularity expiration data in `OptionsExpirationDate` and `ScreenLastTrade` fields. As a result, the `expiration` field in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) will always be at UTC midnight, even when the option expires during or after the trading day. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/ifll-impact # ICE Europe Financials **Dataset ID**: [IFLL.IMPACT]($PLATFORM_HOSTNAME/catalog/ifll/IFLL.IMPACT) ICE disseminates FOD (full order depth), MBP (market by price), TOB (top-of-book) and market statistics data for futures and options on futures via their iMpact feed, which Databento receives over UDP multicast in our Aurora DC3 data center. For futures, ICE publishes both FOD and MBP-5. Databento only normalizes the FOD messages. For options, ICE publishes MBP-10, MBP-5 and top-of-book. Databento only normalizes the MBP-10 messages. This means that for options, only the top 10 levels of the book are available even in the MBO schema. For futures the full market is shown. ### Timestamps ICE iMpact messages received by Databento will generally include two timestamps with microsecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | ICE Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | TransactDateTime and SequenceWithinMillis | The matching-engine-processed timestamp with sub-microsecond sequence number. | | `ts_in_delta` | SentDateTime | The matching-engine-sending timestamp. | ICE provides microsecond granularity timestamps along with sub-microsecond sequence numbers for ordering that are normalized as nanosecond timestamps for consistency in ordering with other datasets. More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Matching Engine Location ICE Europe Financials has matching engines both in Chicago and Basildon. Currently, the products offered by Databento all trade in the ICE silo, located in 350 E Cermak, Chicago. Databento captures the data in our Aurora DC3 location. ### MBO normalization For options, the ICE iMpact feed only includes market-by-price, which matches Databento's [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md). We do not offer these messages in our [MBO schema](https://databento.com/docs/schemas/mbo.md). They are only available on MBP-10 and other downstream schemas. Each ICE "Trade Message" is normalized into a single MBO record with the **T**rade action, and the message's "TradeID" field is used as the `order_id`. ICE does not disseminate information about the passive side of the trade. The corresponding **F**ill record will have `side` equal to `N` and the same `order_id` value as the **T**rade record. When ICE does not specify the aggressing side, **T**rade records will have `side` equal to `N`. ### System priced legs A system priced leg is the result of a trade in the outright spread. These trades, as published by the exchange, can have prices that are 10% or more off from the best bid and offer. ICE recommends that the price of these trades isn't counted towards market statistics. Databento normalizes system priced legs as a **T**rade record with `side` equal to `N` and an `UNDEF_PRICE` (`0x7fffffffffffffff`, 9223372036854775808) value for the price. These trades are counted towards OHLCV volume, but not towards the open, high, low or close values. This means that OHLCV messages can have some (or even all) price values as `UNDEF_PRICE`. ### Off-market trades Trades marked by the exchange as off-market have their `publisher_id` set to `IFLL.IMPACT.XOFF` to distinguish them from on-exchange trades. This distinction is also reflected on the OHLCV schemas - if an interval contains both on-market and off-market trades for an instrument, two OHLCV messages will be published for that instrument on that interval, one with `IFLL.IMPACT.IFLL` as the `publisher_id` and one with `IFLL.IMPACT.XOFF`. ### Statistics **Stat flags** The `IsOfficial` field in the ICE iMpact Settlement Price Message and `IsSettlePriceOfficial` in the Market Snapshot Message are normalized to the first (least-significant) bit of `stat_flags` where the bit is set if `IsOfficial` is `Y`. This is consistent with the behavior for [CME Globex MDP 3.0](https://databento.com/docs/datasets/glbx-mdp3.md#statistics). ### Symbology ICE uses separate symbology for outrights and strategies. **Outrights** For outrights -- individual futures and options -- raw symbols are sourced from `ContractSymbol` field. If `ContractSymbolExtra` is present, it's appended to the `ContractSymbol`. Outright contract symbols are composed of several parts indicating different aspects of the instrument. For example, the Three Month Euribor June 2025 futures contract's symbol is `I FMM0025!`. This can be broken down as follows: | Component | Name | Description | | `I ` | Contract code | The right space-padded four-character [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | `F` | Contract type | Indicates whether the instrument is a future (`F`), swap, or other type of contract. | | `M` | Term | The contract term, usually month (`M`). | | `M` | Month | The expiration month, e.g. `M` for June. | | `00` | Day | The effective date of the contract. `00` for monthly contracts. | | `25` | Year | The two-digit expiration year. | | `!` | Switch character | Indicates additional distinguishing information about the contract. `!` indicates no additional data. | Options begin with their underlying future's symbol followed by an underscore (`_`) and the option block. Using the Three Month Euribor June 2025 87.00 call `I FMM0025_OMCA0009793754061625` as an example, the option block breaks down as: | Component | Name | Description | | `O` | Option indicator | Will always be `O` for options. | | `M` | Option type | The option term, e.g. `M`onthly or `W`eekly. | | `C` | Put or call | Whether the option is a `C`all or `P`ut. | | `A` | Option style | The exercise style of the option, e.g. `A` for American. | | `000979375` | Strike | The zero-padded nine-digit strike price. | | `4` | Strike decimal | The number of decimal places in the strike price. Applying four decimal places to 979375 gives a strike price of 97.9375. | | `061625` | Expiration date | The MMDDYY expiration date of the option. | Refer to ICE's [Instrument Naming Convention document](https://www.ice.com/publicdocs/technology/Instrument_Naming_Convention.pdf) for a full explanation of the format of contract symbols. Because each dataset has a separate symbology namespace, Databento omits the MIC code from the fully-qualified symbol in ICE's examples. **Strategies** For spreads and strategies where ICE does not define a `ContractSymbol`, raw symbols are sourced from `StrategySymbol`. Strategy symbols are composed of several parts, however, due to the use of numeric identifiers, strategy raw symbols are harder to interpret than those for outrights. The Three Month Euribor options call spread `I 6 20284553` can be broken down as follows: | Component | Name | Description | | `I ` | Contract code | The [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | ` 6` | Strategy code | A left space-padded three-digit number identifying the type of strategy, e.g. `6` for call spread. Refer to the table in [this document](https://www.ice.com/publicdocs/technology/ICE_Strategy_Code_Reference_Manual.pdf) for a complete list of strategy codes. | | ` 20284553` | Market ID | A left space-padded 10-digit unique identifier. The same as `instrument_id`. | To inspect spread legs, including intercommodity futures spreads, use the leg fields in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) instead of parsing the raw strategy symbol. See [Finding ICE intercommodity spreads](https://databento.com/docs/examples/instrument-definitions/ice-crack-spreads.md). **Databento MBO snapshot** ICE Europe Financials iMpact daily [trading hours](https://www.ice.com/trading-hours) cross 00:00:00 UTC. To make it easier to work with historical MBO data, Databento includes an [order book snapshot](https://databento.com/docs/conventions/mbo-snapshot.md) at 00:00:00 UTC each weekday (Monday-Friday). ### Definitions - `MarketTypeId` is normalized to `market_segment_id` - `MICCode` is normalized to `exchange` when it is provided: only for futures and future strategies **Expirations** ICE only provides date-granularity expiration data in `OptionsExpirationDate` and `ScreenLastTrade` fields. As a result, the `expiration` field in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) will always be at UTC midnight, even when the option expires during or after the trading day. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/ifus-impact # ICE Futures US **Dataset ID**: [IFUS.IMPACT]($PLATFORM_HOSTNAME/catalog/ifus/IFUS.IMPACT) ICE disseminates FOD (full order depth), MBP (market by price), TOB (top-of-book) and market statistics data for futures and options on futures via their iMpact feed, which Databento receives over UDP multicast in our Aurora DC3 data center. For futures, ICE publishes both FOD and MBP-5. Databento only normalizes the FOD messages. For options, ICE publishes MBP-10, MBP-5 and top-of-book. Databento only normalizes the MBP-10 messages. This means that for options, only the top 10 levels of the book are available even in the MBO schema. For futures the full market is shown. ### Timestamps ICE iMpact messages received by Databento will generally include two timestamps with microsecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | ICE Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | TransactDateTime and SequenceWithinMillis | The matching-engine-processed timestamp with sub-microsecond sequence number. | | `ts_in_delta` | SentDateTime | The matching-engine-sending timestamp. | ICE provides microsecond granularity timestamps along with sub-microsecond sequence numbers for ordering that are normalized as nanosecond timestamps for consistency in ordering with other datasets. More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### MBO normalization For options, the ICE iMpact feed only includes market-by-price, which matches Databento's [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md). We do not offer these messages in our [MBO schema](https://databento.com/docs/schemas/mbo.md). They are only available on MBP-10 and other downstream schemas. Each ICE "Trade Message" is normalized into a single MBO record with the **T**rade action, and the message's "TradeID" field is used as the `order_id`. ICE does not disseminate information about the passive side of the trade. The corresponding **F**ill record will have `side` equal to `N` and the same `order_id` value as the **T**rade record. When ICE does not specify the aggressing side, **T**rade records will have `side` equal to `N`. ### System priced legs A system priced leg is the result of a trade in the outright spread. These trades, as published by the exchange, can have prices that are 10% or more off from the best bid and offer. ICE recommends that the price of these trades isn't counted towards market statistics. Databento normalizes system priced legs as a **T**rade record with `side` equal to `N` and an `UNDEF_PRICE` (`0x7fffffffffffffff`, 9223372036854775808) value for the price. These trades are counted towards OHLCV volume, but not towards the open, high, low or close values. This means that OHLCV messages can have some (or even all) price values as `UNDEF_PRICE`. ### Off-market trades Trades marked by the exchange as off-market have their `publisher_id` set to `IFUS.IMPACT.XOFF` to distinguish them from on-exchange trades. This distinction is also reflected on the OHLCV schemas - if an interval contains both on-market and off-market trades for an instrument, two OHLCV messages will be published for that instrument on that interval, one with `IFUS.IMPACT.IFUS` as the `publisher_id` and one with `IFUS.IMPACT.XOFF`. ### Statistics **Stat flags** The `IsOfficial` field in the ICE iMpact Settlement Price Message and `IsSettlePriceOfficial` in the Market Snapshot Message are normalized to the first (least-significant) bit of `stat_flags` where the bit is set if `IsOfficial` is `Y`. This is consistent with the behavior for [CME Globex MDP 3.0](https://databento.com/docs/datasets/glbx-mdp3.md#statistics). ### Symbology ICE uses separate symbology for outrights and strategies. **Outrights** For outrights -- individual futures and options -- raw symbols are sourced from `ContractSymbol` field. If `ContractSymbolExtra` is present, it's appended to the `ContractSymbol`. Outright contract symbols are composed of several parts indicating different aspects of the instrument. For example, the Sugar No. 11 July 2025 futures contract's symbol is `SB FMN0025!`. This can be broken down as follows: | Component | Name | Description | | `SB ` | Contract code | The right space-padded four-character [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | `F` | Contract type | Indicates whether the instrument is a future (`F`), swap, or other type of contract. | | `M` | Term | The contract term, usually month (`M`). | | `N` | Month | The expiration month, e.g. `N` for July. | | `00` | Day | The effective date of the contract. `00` for monthly contracts. | | `25` | Year | The two-digit expiration year. | | `!` | Switch character | Indicates additional distinguishing information about the contract. `!` indicates no additional data. | Options begin with their underlying future's symbol followed by an underscore (`_`) and the option block. Using the Sugar No. 11 October 2025 _____ put `SB FMV0025_OMPA0000011002091525` as an example, the option block breaks down as: | Component | Name | Description | | `O` | Option indicator | Will always be `O` for options. | | `M` | Option type | The option term, e.g. `M`onthly or `W`eekly. | | `P` | Put or call | Whether the option is a `C`all or `P`ut. | | `A` | Option style | The exercise style of the option, e.g. `A` for American. | | `000001100` | Strike | The zero-padded nine-digit strike price. | | `2` | Strike decimal | The number of decimal places in the strike price. Applying two decimal places to 1100 gives a strike price of 11.00. | | `091525` | Expiration date | The MMDDYY expiration date of the option. `SB` options expire before the contract month, in this case the October options expire September 15, 2025. | Refer to ICE's [Instrument Naming Convention document](https://www.ice.com/publicdocs/technology/Instrument_Naming_Convention.pdf) for a full explanation of the format of contract symbols. Because each dataset has a separate symbology namespace, Databento omits the MIC code from the fully-qualified symbol in ICE's examples. **Strategies** For spreads and strategies where ICE does not define a `ContractSymbol`, raw symbols are sourced from `StrategySymbol`. Strategy symbols are composed of several parts, however, due to the use of numeric identifiers, strategy raw symbols are harder to interpret than those for outrights. The Sugar No. 11 options diagonal put spread `SB 10 30951651` can be broken down as follows: | Component | Name | Description | | `SB ` | Contract code | The [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) or product this instrument belongs to. | | ` 10` | Strategy code | A left space-padded three-digit number identifying the type of strategy, e.g. `10` for diagonal put spread. Refer to the table in [this document](https://www.ice.com/publicdocs/technology/ICE_Strategy_Code_Reference_Manual.pdf) for a complete list of strategy codes. | | ` 30951651` | Market ID | A left space-padded 10-digit unique identifier. The same as `instrument_id`. | To inspect spread legs, including intercommodity futures spreads, use the leg fields in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) instead of parsing the raw strategy symbol. See [Finding ICE intercommodity spreads](https://databento.com/docs/examples/instrument-definitions/ice-crack-spreads.md). **Databento MBO snapshot** ICE Futures US iMpact daily [trading hours](https://www.ice.com/trading-hours) cross 00:00:00 UTC. To make it easier to work with historical MBO data, Databento includes an [order book snapshot](https://databento.com/docs/conventions/mbo-snapshot.md) at 00:00:00 UTC each weekday (Monday-Friday). ### Definitions - `MarketTypeId` is normalized to `market_segment_id` - `MICCode` is normalized to `exchange` when it is provided: only for futures and future strategies **Expirations** ICE only provides date-granularity expiration data in `OptionsExpirationDate` and `ScreenLastTrade` fields. As a result, the `expiration` field in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) will always be at UTC midnight, even when the option expires during or after the trading day. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/iexg-tops # IEX TOPS **Dataset ID**: [IEXG.TOPS]($PLATFORM_HOSTNAME/catalog/us-equities#IEXG.TOPS) IEX disseminates top-of-book and trade information for its US equities exchange through their IEX TOPS feed over the IEX-TP v1 protocol, which Databento receives over UDP multicast in our NY4 data center. PDF specifications are available through IEX's website for [IEX TOPS](https://www.iexexchange.io/documents/tops-v1-66) and [IEX-TP v1](https://www.iexexchange.io/documents/iex-tp-v1). > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps IEX TOPS messages received by Databento will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | IEX Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp (TOPS) | The matching-engine-received timestamp. | | `ts_in_delta` | Send Time (TP) | The matching-engine-sending timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/memx-memoir # MEMX MEMOIR Depth **Dataset ID**: [MEMX.MEMOIR]($PLATFORM_HOSTNAME/catalog/us-equities#MEMX.MEMOIR) MEMX disseminates full depth-of-book data for their exchange through their MEMOIR Depth Feed, which Databento receives over UDP multicast in our NY4 data center. The [MEMX Depth Feed Client Specifications](https://www.memxtrading.com/specs) are available from MEMX's website. > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **CMS** convention (e.g. `BRK B`, `AAC WS`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps MEMX messages received by Databento include one timestamp with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. MEMX does not include an exchange send timestamp, thus `ts_in_delta` field is set to `0` for all messages. | Databento Field | MEMX Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | | `ts_in_delta` | N/A | Set to 0. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/eprl-dom # MIAX Pearl Depth of Market **Dataset ID**: [EPRL.DOM]($PLATFORM_HOSTNAME/catalog/us-equities#EPRL.DOM) MIAX disseminates full depth-of-book information for MIAX Pearl Equities through the Miax Pearl Equities Depth of Market Feed (DoM) over the MACH protocol, which Databento receives over UDP multicast in our NY4 data center. PDF specifications are available through MIAX's website for [Depth of Market](https://www.miaxglobal.com/sites/default/files/2022-05/MIAX_PEARL_Equities_DoM_Feed_v1.3a.pdf) and [MACH](https://www.miaxglobal.com/sites/default/files/page-files/MIAX_PEARL_MIAX_MACH_Protocol_v1_2c.pdf). > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps MIAX DoM messages received by Databento have at most one timestamp, which has nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The extra field `ts_in_delta` does not convey any additional information for this dataset, and is equivalent to `ts_recv - ts_event`. | Databento Field | MIAX DoM Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xnas-basic # Nasdaq Basic with NLS Plus **Dataset ID**: [XNAS.BASIC]($PLATFORM_HOSTNAME/catalog/us-equities#XNAS.BASIC) Nasdaq disseminates top-of-book data via their QBBO feed, and last sale data via their NLS and NLS+ feeds. Databento receives these over UDP multicast in our NY4 data center. Because they come from separate feeds, BBO and trade events may be out-of-order, even within a single instrument. PDF specifications are available on Nasdaq's website for the [QBBO](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/QBBOSpecification2.1.pdf), [NLS](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NLSSpecification3.0.pdf) and [NLS+](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NLSPlus-cloud.pdf) feeds. ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Nasdaq Basic messages received by Databento have at most one timestamp, which has nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The extra field `ts_in_delta` does not convey any additional information for this dataset, and is equivalent to `ts_recv - ts_event`. | Databento Field | TotalView Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Source feeds Nasdaq Basic is composed of data from the Nasdaq QBBO and Nasdaq NLS+ feeds. ### Venues Nasdaq Basic contains trade data for the following venues: - Nasdaq (XNAS) - Nasdaq PSX (XPSX) - Nasdaq Texas (XBOS) - FINRA/Nasdaq TRF Carteret (FINN) - FINRA/Nasdaq TRF Chicago (FINC) However, only quotes for Nasdaq (XNAS) are provided. The `publisher_id` field on every record from Databento can be used to identify the venue associated with the event. Because Nasdaq PSX, Nasdaq Texas, and the TRF facilities do not have a BBO disseminated by Nasdaq QBBO, the [MBP-1](https://databento.com/docs/schemas/mbp-1.md), [BBO](https://databento.com/docs/schemas/bbo.md) and [TBBO](https://databento.com/docs/schemas/tbbo.md) schemas are not available. Instead, the equivalent consolidated schemas are provided: [CMBP-1](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1), [CBBO](https://databento.com/docs/schemas/bbo.md#consolidated-bbo-on-interval-cbbo) and [TCBBO](https://databento.com/docs/schemas/tbbo.md#consolidated-bbo-on-trade-tcbbo). ### Normalization There are only **A**dd and **T**rade actions for Nasdaq Basic. **T**rade records are normalized from NLS+ *Trade Report for Non-NextShares* or *Long form Trade Report for Non-NextShares* messages. The `publisher_id` field will identify the venue associated with the trade. There is no information about the side of the trade, so `side` will always be `N`. **A**dd records are normalized from Nasdaq QBBO *Quotation Message*. An **A** record will be published anytime the best bid or offer quote is updated. The `publisher_id` field will always be set to 93 (`XNAS_BASIC_EQUS`) ### Definition normalization Definition records are sourced from NLS+ *Stock Directory* messages. | Databento Field | NLS+ Field | Description | | `exchange` | Market Category | Normalized to the MIC code of the listing venue. All Nasdaq-listed instruments will normalize to `XNAS`. | | `security_type` | Issue Classification | Identifies the security class for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix E [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NLSPlusSpecification3.0.pdf). | | `secsubtype` | Issue Sub-Type | Identifies the security sub-type for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix F [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NLSPlusSpecification3.0.pdf). | ### Statistics normalization Statistics records are sourced from NLS+ *Trade Report for Non-NextShares* messages. Databento provides the opening price and closing price statistics based on the sale condition field of these trade messages. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xnas-itch # Nasdaq TotalView-ITCH **Dataset ID**: [XNAS.ITCH]($PLATFORM_HOSTNAME/catalog/us-equities#XNAS.ITCH) Nasdaq disseminates full depth-of-book data for their exchange through their TotalView-ITCH feed over the MoldUDP64 protocol, which Databento receives over UDP multicast in our NY4 data center. PDF specifications are available on Nasdaq's website for [Nasdaq TotalView-ITCH](http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf) and [MoldUDP64](http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf). ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Nasdaq TotalView-ITCH messages received by Databento have at most one timestamp, which has nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The extra field `ts_in_delta` does not convey any additional information for this dataset, and is equivalent to `ts_recv - ts_event`. | Databento Field | TotalView Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### MBO normalization **Order lifecycle messages** There are seven message types in TotalView that change the state of the order book. Databento normalizes these messages to MBO records as follows: - *Add Order - No MPID Attribution*: **A**dd - *Add Order with MPID Attribution*: **A**dd, but the attribution field is not included - *Order Executed*: **T**rade, **F**ill, then **C**ancel - *Order Executed with Price*: **T**rade, **F**ill, then **C**ancel, but the initial **T**rade is omitted if the message is marked as non-printable - *Order Cancel*: **C**ancel, with the `quantity` indicating the number of shares to remove from the display size - *Order Delete*: **C**ancel, with the `quantity` indicating the current display size of the order - *Order Replace*: **C**ancel with the previous `order_id`, followed by **A**dd with the new `order_id` All **T**rade records will have `order_id` set to `0`, because Nasdaq does not provide information about the aggressor's order. Normalized records from the same TotalView message will have the same `sequence`. **Trade messages** TotalView also has two messages for providing execution details for match events against non-displayed orders or cross events. Databento normalizes each *Trade (Non-Cross)* message into a single MBO **T**rade record with side **N**one. *Cross Trade* messages are also normalized into MBO **T**rade records with side **N**one when shares are matched on the cross event. Nasdaq will always send a *Cross Trade* message following an opening, closing, or EMC cross event for an instrument. If no shares are matched during a cross event, the *Cross Trade* message is omitted from MBO and represented only in the [statistics schema](https://databento.com/docs/schemas/statistics.md). ### Statistics normalization TotalView provides information about the execution price and shares matched on completion of a cross event in their *Cross Trade* message. Databento normalizes each message into a record in the [statistics schema](https://databento.com/docs/schemas/statistics.md). Statistics are categorized by the following cross events: - Opening price: Nasdaq Opening Cross - Closing price: Nasdaq Closing Cross - Uncrossing price: EMC Cross (Halt/IPO) Nasdaq will always send a *Cross Trade* message following an opening, closing, or EMC cross event for an instrument. If no shares are matched during a cross event, `quantity` will be `0` and `price` will be `UNDEF_PRICE` (`0x7fffffffffffffff`, `9223372036854775808`). ### Imbalance normalization TotalView provides information about cross imbalances in their *Net Order Imbalance Indicator (NOII)* message. Databento normalizes each message into a record in the [imbalance schema](https://databento.com/docs/schemas/imbalance.md). | Databento Field | TotalView Field | Description | | `paired_qty` | Paired Shares | The number of shares that are eligible to match at the `ref_price`. | | `total_imbalance_qty` | Imbalance Shares | The number of shares not paired at the `ref_price`. | | `side` | Imbalance Direction | The market side of the order imbalance: **B**id (buy) imbalance, **A**sk (sell) imbalance, **N**o imbalance. | | `auct_interest_clr_price` | Far Price | Hypothetical auction-clearing price for cross orders only. | | `cont_book_clr_price` | Near Price | Hypothetical auction-clearing price for cross and continuous orders. | | `ref_price` | Current Reference Price | The price at which imbalance shares are being calculated. | | `auction_type` | Cross Type | The type of cross: **O**pening, **C**losing, un-**H**alt or IPO, **A**: Extended Trading Close. | | `significant_imbalance` | Price Variation Indicator | Indicates the deviation of `cont_book_clr_price` to the `ref_price`. Refer to the Nasdaq TotalView specification for values. | #### **Opening/Closing Cross** Nasdaq begins disseminating imbalance data at 9:25 ET for the Opening Cross and 15:50 ET for the Closing Cross. Between 9:25 and 9:28, and 15:50 and 15:55, imbalance data is sent every 10 seconds. The `auct_interest_clr_price` and `cont_book_clr_price` fields will not be populated during these times. After 9:28 and 15:55, imbalance data is sent every 1 second and will include `auct_interest_clr_price` and `cont_book_clr_price` data. At the conclusion of the Opening Cross at 9:30 and the Closing Cross at 16:00, a [statistics record](https://databento.com/docs/datasets/xnas-itch.md#statistics-normalization) will be published with the official opening/closing price. > **Info** > > For more information on the Nasdaq Opening and Closing Cross, see this [Nasdaq FAQ](https://nasdaqtrader.com/content/productsservices/trading/crosses/openclose_faqs.pdf). ### Definition normalization Definition records are sourced from Stock Directory messages. | Databento Field | TotalView Field | Description | | `exchange` | Market Category | Normalized to the MIC code of the listing venue. All Nasdaq-listed instruments will normalize to `XNAS`. | | `security_type` | Issue Classification | Identifies the security class for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix D [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf). | | `secsubtype` | Issue Sub-Type | Identifies the security sub-type for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix E [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf). | ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xbos-itch # Nasdaq Texas TotalView-ITCH **Dataset ID**: [XBOS.ITCH]($PLATFORM_HOSTNAME/catalog/us-equities#XBOS.ITCH) Nasdaq Texas disseminates full depth-of-book data for their exchange through their TotalView-ITCH feed over the MoldUDP64 protocol, which Databento receives over UDP multicast in our NY4 data center. PDF specifications are available on Nasdaq's website for [Nasdaq TotalView-ITCH](http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf) and [MoldUDP64](http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf). ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Nasdaq Texas TotalView-ITCH messages received by Databento have at most one timestamp, which has nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The extra field `ts_in_delta` does not convey any additional information for this dataset, and is equivalent to `ts_recv - ts_event`. | Databento Field | TotalView Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### MBO normalization **Order lifecycle messages** There are seven message types in TotalView that change the state of the order book. Databento normalizes these messages to MBO records as follows: - *Add Order - No MPID Attribution*: **A**dd - *Add Order with MPID Attribution*: **A**dd, but the attribution field is not included - *Order Executed*: **T**rade, **F**ill, then **C**ancel - *Order Executed with Price*: **T**rade, **F**ill, then **C**ancel, but the initial **T**rade is omitted if the message is marked as non-printable - *Order Cancel*: **C**ancel, with the `quantity` indicating the number of shares to remove from the display size - *Order Delete*: **C**ancel, with the `quantity` indicating the current display size of the order - *Order Replace*: **C**ancel with the previous `order_id`, followed by **A**dd with the new `order_id` All **T**rade records will have `order_id` set to `0`, because Nasdaq does not provide information about the aggressor's order. Normalized records from the same TotalView message will have the same `sequence`. **Trade messages** TotalView also has two messages for providing execution details for match events against non-displayed orders or cross events. Databento normalizes both *Trade (Non-Cross)* and *Cross Trade* messages into a single MBO **T**rade record with side **N**one. ### Definition normalization Definition records are sourced from Stock Directory messages. | Databento Field | TotalView Field | Description | | `exchange` | Market Category | Normalized to the MIC code of the listing venue. All Nasdaq-listed instruments will normalize to `XNAS`. | | `security_type` | Issue Classification | Identifies the security class for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix D [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf). | | `secsubtype` | Issue Sub-Type | Identifies the security sub-type for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix E [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf). | ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xpsx-itch # Nasdaq PSX TotalView-ITCH **Dataset ID**: [XPSX.ITCH]($PLATFORM_HOSTNAME/catalog/us-equities#XPSX.ITCH) Nasdaq PSX disseminates full depth-of-book data for their exchange through their TotalView-ITCH feed over the MoldUDP64 protocol, which Databento receives over UDP multicast in our NY4 data center. PDF specifications are available on Nasdaq's website for [Nasdaq TotalView-ITCH](http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf) and [MoldUDP64](http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf). ### Symbol convention Raw symbols in this dataset use the **Nasdaq** convention (e.g. `BRK.B`, `AAC+`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps Nasdaq PSX TotalView-ITCH messages received by Databento have at most one timestamp, which has nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The extra field `ts_in_delta` does not convey any additional information for this dataset, and is equivalent to `ts_recv - ts_event`. | Databento Field | TotalView Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Timestamp | The matching-engine-received timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### MBO normalization **Order lifecycle messages** There are seven message types in TotalView that change the state of the order book. Databento normalizes these messages to MBO records as follows: - *Add Order - No MPID Attribution*: **A**dd - *Add Order with MPID Attribution*: **A**dd, but the attribution field is not included - *Order Executed*: **T**rade, **F**ill, then **C**ancel - *Order Executed with Price*: **T**rade, **F**ill, then **C**ancel, but the initial **T**rade is omitted if the message is marked as non-printable - *Order Cancel*: **C**ancel, with the `quantity` indicating the number of shares to remove from the display size - *Order Delete*: **C**ancel, with the `quantity` indicating the current display size of the order - *Order Replace*: **C**ancel with the previous `order_id`, followed by **A**dd with the new `order_id` All **T**rade records will have `order_id` set to `0`, because Nasdaq does not provide information about the aggressor's order. Normalized records from the same TotalView message will have the same `sequence`. **Trade messages** TotalView also has two messages for providing execution details for match events against non-displayed orders or cross events. Databento normalizes both *Trade (Non-Cross)* and *Cross Trade* messages into a single MBO **T**rade record with side **N**one. ### Definition normalization Definition records are sourced from Stock Directory messages. | Databento Field | TotalView Field | Description | | `exchange` | Market Category | Normalized to the MIC code of the listing venue. All Nasdaq-listed instruments will normalize to `XNAS`. | | `security_type` | Issue Classification | Identifies the security class for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix D [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf). | | `secsubtype` | Issue Sub-Type | Identifies the security sub-type for the issue as assigned by Nasdaq. A list of the different values and their meanings can be found in Appendix E [here](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf). | ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xnys-pillar # NYSE Integrated **Dataset ID**: [XNYS.PILLAR]($PLATFORM_HOSTNAME/catalog/us-equities#XNYS.PILLAR) NYSE disseminates full depth-of-book data for their exchange through their Integrated Feed, which Databento receives over UDP multicast in our NY4 data center. The [Integrated Feed Client Specification](https://www.nyse.com/market-data/real-time/integrated-feed) is available as a PDF from NYSE's website. ### Symbol convention Raw symbols in this dataset use the **CMS** convention (e.g. `BRK B`, `AAC WS`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps NYSE messages received by Databento will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pillar Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | SourceTime | The matching-engine-received timestamp. | | `ts_in_delta` | SendTime (Common) | The matching-engine-sending timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Statistics normalization NYSE provides information about the execution price and shares matched on completion of a crossing auction in their *Cross Trade* message. Databento normalizes each message into a record in the [statistics schema](https://databento.com/docs/schemas/statistics.md). Statistics are categorized by the following auction types: - Opening price: Core Opening auction - Closing price: Closing auction - Uncrossing price: Halt/Re-opening auction ### Imbalance normalization NYSE provides information about opening and closing cross imbalances in their *Imbalance* message. Databento normalizes each message into a record in the [imbalance schema](https://databento.com/docs/schemas/imbalance.md). | Databento Field | Pillar Field | Description | | `ref_price` | ReferencePrice | The price at which imbalance shares are being calculated. | | `auction_time` | AuctionTime | The projected auction time, with second precision. | | `cont_book_clr_price` | ContinuousBookClearingPrice | The price closest to the reference price where the imbalance is 0. For significant imbalances, or if a continuous book clearing price is not reached, this field is defaulted to 0. | | `auct_interest_clr_price` | AuctionInterestClearingPrice | For the closing auction, the price closest to the reference price where the imbalance of closing-only interest is 0. If a closing-only clearing price is not reached, this field is defaulted to 0. | | `ssr_filing_price` | SSRFilingPrice | For non-Significant imbalances, if a Sell Short Restriction is in effect, the price at which Sell Short interest will be filed. | | `paired_qty` | PairedQty | The number of shares that are eligible to match at the `ref_price`. | | `total_imbalance_qty` | TotalImbalanceQty | The number of shares not paired at the `ref_price`. | | `unpaired_qty` | UnpairedQty | During the Closing Auction, the number of unpaired shares priced at or better than the `ref_price`. | | `auction_type` | AuctionType | The type of cross: **M**: Core opening, **H**: Halt resume, **C**: Closing, **P**: Extreme closing imbalance, **R**: Significant closing imbalance. | | `side` | ImbalanceSide | The market side of the order imbalance: **B**id (buy) imbalance, **A**sk (sell) imbalance, **N**o imbalance. | | `auction_status` | AuctionStatus | **0**: Will run as always for Open and Close. **1**: Will run, interest exists inside or at the collars or is fully paired off. **2**: Will not run because there is an imbalance through the collars. **3**: Will not run, will transition to the Closing Auction instead. | | `freeze_status` | FreezeStatus | **0**: Imbalance freeze not in effect. **1**: Imbalance freeze is in effect. | | `significant_imbalance` | SignificantImbalance | Indicates whether the current imbalance is significant. **Y** when significant, **' '** (space) otherwise. **~** prior to March 2019 when the Significant Imbalance field was added to the protocol. | #### **Auctions** During auction periods, an imbalance message will be published every 1 second, if changed from the previous second. Imbalance data will only be published for NYSE listed securities during these auction periods. At the conclusion of an auction, a [statistics record](https://databento.com/docs/datasets/xnys-pillar.md#statistics-normalization) will be published. NYSE conducts two scheduled auctions per trading day at the following times: - Core Open: 8:00 to 9:30 ET - Closing: 15:50 to 16:00 ET Auctions can also occur as the result of a trading halt. > **See also** > > See this [NYSE Pillar documentation](https://www.nyse.com/publicdocs/nyse/markets/nyse/Functional_Differences_NYSE_Pillar.pdf) for more information on auctions. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xase-pillar # NYSE American Integrated **Dataset ID**: [XASE.PILLAR]($PLATFORM_HOSTNAME/catalog/us-equities#XASE.PILLAR) NYSE American disseminates full depth-of-book data for their exchange through their Integrated Feed, which Databento receives over UDP multicast in our NY4 data center. The [Integrated Feed Client Specification](https://www.nyse.com/market-data/real-time/integrated-feed) is available as a PDF from NYSE's website. ### Symbol convention Raw symbols in this dataset use the **CMS** convention (e.g. `BRK B`, `AAC WS`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps NYSE American messages received by Databento will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pillar Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | SourceTime | The matching-engine-received timestamp. | | `ts_in_delta` | SendTime (Common) | The matching-engine-sending timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Statistics normalization NYSE American provides information about the execution price and shares matched on completion of a crossing auction in their *Cross Trade* message. Databento normalizes each message into a record in the [statistics schema](https://databento.com/docs/schemas/statistics.md). Statistics are categorized by the following auction types: - Opening price: Core Opening auction - Closing price: Closing auction - Uncrossing price: Early Opening and Halt/Re-opening auction ### Imbalance normalization NYSE American provides information about opening and closing cross imbalances in their *Imbalance* message. Databento normalizes each message into a record in the [imbalance schema](https://databento.com/docs/schemas/imbalance.md). | Databento Field | Pillar Field | Description | | `ref_price` | ReferencePrice | The price at which imbalance shares are being calculated. | | `auction_time` | AuctionTime | The projected auction time, with second precision. | | `cont_book_clr_price` | ContinuousBookClearingPrice | Hypothetical auction-clearing price for cross and continuous orders. | | `auct_interest_clr_price` | AuctionInterestClearingPrice | The price at which all eligible auction-only interest would trade, subject to auction collars. | | `ind_match_price` | IndicativeMatchPrice | The price at which the highest number of shares would trade, subject to auction collars. It includes the non-displayed quantity of Reserve Orders. | | `upper_collar` | UpperCollar | Upper auction collar. | | `lower_collar` | LowerCollar | Lower auction collar. | | `paired_qty` | PairedQty | The number of shares paired off at the `ref_price`. Before 2025-11-10, the number of shares paired off at the `ind_match_price`. | | `total_imbalance_qty` | TotalImbalanceQty | The total imbalance quantity at the `ref_price`. Before 2025-11-10, the total imbalance quantity at the `ind_match_price`. | | `market_imbalance_qty` | MarketImbalanceQty | The total market order imbalance quantity at the `ref_price`. Before 2025-11-10, the total market order imbalance quantity at the `ind_match_price`. | | `auction_type` | AuctionType | The type of cross: **O**: Early opening, **M**: Core opening, **H**: Halt resume, **C**: Closing. | | `side` | ImbalanceSide | The market side of the order imbalance: **B**id (buy) imbalance, **A**sk (sell) imbalance, **N**o imbalance. | | `auction_status` | AuctionStatus | **0**: Will run as always for Open and Close. **1**: Will run, interest exists inside or at the collars or is fully paired off. **2**: Will not run because there is an imbalance through the collars. **3**: Will not run, will transition to the Closing Auction instead. | | `freeze_status` | FreezeStatus | **0**: Imbalance freeze not in effect. **1**: Imbalance freeze is in effect. | | `num_extensions` | NumExtensions | Number of times the halt period has been extended. | | `significant_imbalance` | SignificantImbalance | Indicates whether the current imbalance is significant. **Y** when significant, **' '** (space) otherwise. **~** prior to March 2019 when the Significant Imbalance field was added to the protocol. | #### **Auctions** During auction periods, an imbalance message will be published every 1 second, if changed from the previous second. Imbalance data will only be published for NYSE American listed securities during these auction periods. At the conclusion of an auction, a [statistics record](https://databento.com/docs/datasets/xase-pillar.md#statistics-normalization) will be published. NYSE American conducts three scheduled auctions per trading day at the following times: - Early Opening: 6:30 to 7:00 ET - Core Opening: 8:00 to 9:30 ET - Closing: 15:00 to 16:00 ET Auctions can also occur as the result of a trading halt. > **See also** > > See the [NYSE Pillar documentation](https://www.nyse.com/publicdocs/nyse/markets/nyse/Functional_Differences_NYSE_Pillar.pdf) for more information on auctions. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/arcx-pillar # NYSE Arca Integrated **Dataset ID**: [ARCX.PILLAR]($PLATFORM_HOSTNAME/catalog/us-equities#ARCX.PILLAR) NYSE Arca disseminates full depth-of-book data for their exchange through their Integrated Feed, which Databento receives over UDP multicast in our NY4 data center. The [Integrated Feed Client Specification](https://www.nyse.com/market-data/real-time/integrated-feed) is available as a PDF from NYSE's website. ### Symbol convention Raw symbols in this dataset use the **CMS** convention (e.g. `BRK B`, `AAC WS`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps NYSE Arca messages received by Databento will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pillar Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | SourceTime | The matching-engine-received timestamp. | | `ts_in_delta` | SendTime (Common) | The matching-engine-sending timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Statistics normalization NYSE Arca provides information about the execution price and shares matched on completion of a crossing auction in their *Cross Trade* message. Databento normalizes each message into a record in the [statistics schema](https://databento.com/docs/schemas/statistics.md). Statistics are categorized by the following auction types: - Opening price: Core Opening auction - Closing price: Closing auction - Uncrossing price: Early Opening and Halt/Re-opening auction ### Imbalance normalization NYSE Arca provides information about opening and closing cross imbalances in their *Imbalance* message. Databento normalizes each message into a record in the [imbalance schema](https://databento.com/docs/schemas/imbalance.md). | Databento Field | Pillar Field | Description | | `ref_price` | ReferencePrice | The price at which imbalance shares are being calculated. | | `auction_time` | AuctionTime | The projected auction time, with second precision. | | `cont_book_clr_price` | ContinuousBookClearingPrice | Hypothetical auction-clearing price for cross and continuous orders. | | `auct_interest_clr_price` | AuctionInterestClearingPrice | The price at which all eligible auction-only interest would trade, subject to auction collars. | | `ind_match_price` | IndicativeMatchPrice | The price at which the highest number of shares would trade, subject to auction collars. It includes the non-displayed quantity of Reserve Orders. | | `upper_collar` | UpperCollar | Upper auction collar. | | `lower_collar` | LowerCollar | Lower auction collar. | | `paired_qty` | PairedQty | The number of shares paired off at the `ref_price`. Before 2025-11-13, the number of shares paired off at the `ind_match_price`. | | `total_imbalance_qty` | TotalImbalanceQty | The total imbalance quantity at the `ref_price`. Before 2025-11-13, the total imbalance quantity at the `ind_match_price`. | | `market_imbalance_qty` | MarketImbalanceQty | The total market order imbalance quantity at the `ref_price`. Before 2025-11-13, the total market order imbalance quantity at the `ind_match_price`. | | `auction_type` | AuctionType | The type of cross: **O**: Early opening, **M**: Core opening, **H**: Halt resume, **C**: Closing. | | `side` | ImbalanceSide | The market side of the order imbalance: **B**id (buy) imbalance, **A**sk (sell) imbalance, **N**o imbalance. | | `auction_status` | AuctionStatus | **0**: Will run as always for Open and Close. **1**: Will run, interest exists inside or at the collars or is fully paired off. **2**: Will not run because there is an imbalance through the collars. **3**: Will not run, will transition to the Closing Auction instead. | | `freeze_status` | FreezeStatus | **0**: Imbalance freeze not in effect. **1**: Imbalance freeze is in effect. | | `num_extensions` | NumExtensions | Number of times the halt period has been extended. | | `significant_imbalance` | SignificantImbalance | Indicates whether the current imbalance is significant. **Y** when significant, **' '** (space) otherwise. **~** prior to March 2019 when the Significant Imbalance field was added to the protocol. | #### **Auctions** During auction periods, an imbalance message will be published every 1 second, if changed from the previous second. At the conclusion of an auction, a [statistics record](https://databento.com/docs/datasets/arcx-pillar.md#statistics-normalization) will be published. NYSE Arca conducts three scheduled auctions per trading day at the following times: - Early Open: 2:30 to 4:00 ET for all UTP securities - Core Open: 8:00 to 9:30 ET for only NYSE Arca listed securities - Closing: 15:00 to 16:00 ET for all UTP securities Auctions can also occur as the result of a trading halt. Imbalance data will only be published for NYSE Arca listed securities during these trading halts. > **See also** > > See this [NYSE Pillar documentation](https://www.nyse.com/publicdocs/nyse/markets/nyse/Functional_Differences_NYSE_Pillar.pdf) for more information on auctions. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xcis-tradesbbo # NYSE National Trades and BBO **Dataset ID**: [XCIS.TRADESBBO]($PLATFORM_HOSTNAME/catalog/us-equities#XCIS.TRADESBBO) NYSE National disseminates top-of-book and last sale data through their BBO and Trades feeds, respectively. Databento receives these over UDP multicast in our NY4 data center. Because they come from separate feeds, BBO and trade events may be out-of-order, even within a single instrument. PDF specifications are available on NYSE's website for the [BBO](https://www.nyse.com/market-data/real-time/bbo) and [Trades](https://www.nyse.com/market-data/real-time/trades) feeds. > **Info** > > The Databento team is currently working on this article. ### Symbol convention Raw symbols in this dataset use the **CMS** convention (e.g. `BRK B`, `AAC WS`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps NYSE National messages received by Databento will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | BBO/Trades Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | SourceTime | The matching-engine-received timestamp. | | `ts_in_delta` | SendTime (Common) | The matching-engine-sending timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Normalization NYSE National Trades and BBO are sourced from two separate feeds: one for the BBO and one for trades. **A**dd records are normalized from BBO *Quote Message (140)* records in the BBO feed. An **A** record will be published anytime the best bid or offer quote is updated. **T**rade records are normalized from *Trade Message (220)* records in the Trades feed. When combined, this level of data matches Databento's [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md); however, unlike most other datasets, there is not a strict ordering between the events on the BBO feed and the Trades feed. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/xchi-pillar # NYSE Texas Integrated **Dataset ID**: [XCHI.PILLAR]($PLATFORM_HOSTNAME/catalog/us-equities#XCHI.PILLAR) NYSE Texas disseminates full depth-of-book data for their exchange through their Integrated Feed, which Databento receives over UDP multicast in our NY4 data center. The [Integrated Feed Client Specification](https://www.nyse.com/market-data/real-time/integrated-feed) is available as a PDF from NYSE's website. ### Symbol convention Raw symbols in this dataset use the **CMS** convention (e.g. `BRK B`, `AAC WS`). See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ### Timestamps NYSE Texas messages received by Databento will generally include two timestamps with nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. | Databento Field | Pillar Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | SourceTime | The matching-engine-received timestamp. | | `ts_in_delta` | SendTime (Common) | The matching-engine-sending timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Statistics normalization NYSE Texas provides information about the execution price and shares matched on completion of a crossing auction in their *Cross Trade* message. Databento normalizes each message into a record in the [statistics schema](https://databento.com/docs/schemas/statistics.md). Statistics are categorized by the following auction types: - Opening price: Core Opening auction - Closing price: Closing auction - Uncrossing price: Early Opening and Halt/Re-opening auction ### Imbalance normalization NYSE Texas provides information about opening and closing cross imbalances in their *Imbalance* message. Databento normalizes each message into a record in the [imbalance schema](https://databento.com/docs/schemas/imbalance.md). | Databento Field | Pillar Field | Description | | `ref_price` | ReferencePrice | The price at which imbalance shares are being calculated. | | `auction_time` | AuctionTime | The projected auction time, with second precision. | | `cont_book_clr_price` | ContinuousBookClearingPrice | Hypothetical auction-clearing price for cross and continuous orders. | | `auct_interest_clr_price` | AuctionInterestClearingPrice | The price at which all eligible auction-only interest would trade, subject to auction collars. | | `ind_match_price` | IndicativeMatchPrice | The price at which the highest number of shares would trade, subject to auction collars. It includes the non-displayed quantity of Reserve Orders. | | `upper_collar` | UpperCollar | Upper auction collar. | | `lower_collar` | LowerCollar | Lower auction collar. | | `paired_qty` | PairedQty | The number of shares paired off at the `ref_price`. Before 2025-11-20, the number of shares paired off at the `ind_match_price`. | | `total_imbalance_qty` | TotalImbalanceQty | The total imbalance quantity at the `ref_price`. Before 2025-11-20, the total imbalance quantity at the `ind_match_price`. | | `market_imbalance_qty` | MarketImbalanceQty | The total market order imbalance quantity at the `ref_price`. Before 2025-11-20, the total market order imbalance quantity at the `ind_match_price`. | | `auction_type` | AuctionType | The type of cross: **O**: Early opening, **M**: Core opening, **H**: Halt resume, **C**: Closing. | | `side` | ImbalanceSide | The market side of the order imbalance: **B**id (buy) imbalance, **A**sk (sell) imbalance, **N**o imbalance. | | `auction_status` | AuctionStatus | **0**: Will run as always for Open and Close. **1**: Will run, interest exists inside or at the collars or is fully paired off. **2**: Will not run because there is an imbalance through the collars. **3**: Will not run, will transition to the Closing Auction instead. | | `freeze_status` | FreezeStatus | **0**: Imbalance freeze not in effect. **1**: Imbalance freeze is in effect. | | `num_extensions` | NumExtensions | Number of times the halt period has been extended. | | `significant_imbalance` | SignificantImbalance | Indicates whether the current imbalance is significant. **Y** when significant, **' '** (space) otherwise. **~** prior to March 2019 when the Significant Imbalance field was added to the protocol. | #### **Auctions** During auction periods, an imbalance message will be published every 1 second, if changed from the previous second. Imbalance data will only be published for NYSE Texas listed securities during these auction periods. At the conclusion of an auction, a [statistics record](https://databento.com/docs/datasets/xchi-pillar.md#statistics-normalization) will be published. NYSE Texas conducts three scheduled auctions per trading day at the following times: - Early Open: 6:30 to 7:00 ET - Core Open: 8:00 to 9:30 ET - Closing: 15:00 to 16:00 ET Auctions can also occur as the result of a trading halt. > **See also** > > See this [NYSE Pillar documentation](https://www.nyse.com/publicdocs/nyse/markets/nyse/Functional_Differences_NYSE_Pillar.pdf) for more information on auctions. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/opra-pillar # OPRA **Dataset ID**: [OPRA.PILLAR]($PLATFORM_HOSTNAME/catalog/opra/OPRA.PILLAR) OPRA disseminates top-of-book, last sale, and daily statistics data for all US equity options exchanges through their Binary Output feed, which Databento receives over UDP multicast in our NY4 data center. The [OPRA Pillar Output Specification](https://cdn.opraplan.com/documents/OPRA_Pillar_Output_Specification.pdf) is available as a PDF from the OPRA website. ### Timestamps OPRA messages received by Databento include at most one timestamp, which has nanosecond precision. As with all of our datasets, we also collect a `ts_recv` timestamp when the packet was received by our capture server. The unused `ts_in_delta` field is set to `0` for all messages. | Databento Field | OPRA Field | Description | | `ts_recv` | N/A | The capture-server-received timestamp. | | `ts_event` | Block Timestamp | The consolidator-processed timestamp. | More details about our timestamps are available in our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md). ### Venues OPRA includes consolidated data for 20 options venues (as of September 2026). The `publisher_id` field on every record from Databento can be used to identify the venue associated with a trade event. | Value | MIC | Venue | | 20 | AMXO | NYSE American | | 21 | XBOX | BOX Options | | 22 | XCBO | Cboe Options | | 23 | EMLD | MIAX Emerald | | 24 | EDGO | Cboe EDGX Options | | 25 | GMNI | Nasdaq GEMX | | 26 | XISX | Nasdaq ISE | | 27 | MCRY | Nasdaq MRX | | 28 | XMIO | MIAX Options | | 29 | ARCO | NYSE Arca Options | | 30 | OPRA | Options Price Reporting Authority | | 31 | MPRL | MIAX Pearl | | 32 | XNDQ | Nasdaq Options | | 33 | XBXO | Nasdaq Texas Options | | 34 | C2OX | Cboe C2 Options | | 35 | XPHL | Nasdaq PHLX | | 36 | BATO | Cboe BZX Options | | 37 | MXOP | MEMX Options | | 61 | SPHR | MIAX Sapphire | | 108 | MXTO | MEMX MX2 Options | | 109 | IEXO | IEX Options | > **Info** > > These mappings are also available programmatically through the > [metadata.list\_publishers](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md) endpoint. ### Trades Trade records are published with the `publisher_id` corresponding to the venue where the trade occurred. OPRA does not disseminate the side of a trade, so `side` will always be `N`. The [TCBBO schema](https://databento.com/docs/schemas/tbbo.md#consolidated-bbo-on-trade-tcbbo) can be used to see the NBBO at the time of the trade. ### CMBP-1 Along with the 20 options venues, OPRA also publishes the OPRA consolidated BBO. This consolidated BBO is available through Databento as the [CMBP-1 schema](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1). OPRA generates this consolidated BBO based on the following criteria in this order: - **Price**: Venues with the highest bid and lowest offer have overall priority. Minimum price increments must be at least 1 cent higher or lower than the previous price. - **Size**: Largest size takes precedence when multiple venues submit the same bid and/or offer price. There are no initial minimum size requirements. Size increments must be 10 contracts or better than the previous size. - **Time**: Earliest time takes precedence when multiple venues submit the same bid and/or offer price with the same size. > **See also** > > More information on the OPRA consolidated BBO can be found in the [OPRA BBO > Guidelines FAQ](https://cdn.opraplan.com/documents/OPRA_BBO_Guidelines.pdf) OPRA consolidated BBO records are published with a `publisher_id` value of 30 (`OPRA_PILLAR_OPRA`). These records indicate the venues containing the best bid and best offer by including their respective `publisher_id` in the `bid_pb_00` and `ask_pb_00` fields. Trade records are published with the `publisher_id` corresponding to the venue where the trade happened. Databento does not publish BBO records for regional quotes. > **See also** > > [Examples - US equity options volume by venue](https://databento.com/docs/examples/options/options-venues.md) ### Historical CMBP-1 requests Due to the large quantity of data for the CMBP-1 schema, we recommend users request a batch download via [batch.submit_job](https://databento.com/docs/api-historical/batch.md) or the [User portal](https://databento.com/docs/portal/batch-download.md). ### Live CMBP-1 subscriptions Due to the large quantity of data for the CMBP-1 schema, the [slow reader behavior](https://databento.com/docs/api-live/basics/slow-reader-behavior.md) of CMBP-1 subscriptions will always be set to `skip` regardless of client settings. ### Symbology OPRA Pillar raw symbols are based on OCC symbology, also known as Options Symbology Initiative (OSI) symbology. These two terms are interchangeable. In OCC symbology, every raw symbol has exactly 21 characters and is composed of four parts. Using the SPY November 17, 2023 $486.00 put option (`SPY 231117P00486000`) as an example, these are: | Component | Name | Description | | `SPY ` | Parent symbol | The right space-padded six-character [parent symbol](https://databento.com/docs/conventions/symbology.md#parent). Often the symbol of the underlying, but not always such as with `SPXW` (SPX weekly options). | | `211117` | Expiration date | The YYMMDD expiration date of the option. | | `P` | Put or call | Whether the option is a `C`all or `P`ut. | | `00486000` | Strike | The zero-padded eight-digit strike price with three decimal places. | ### Adjusted symbol conventions Under OSI symbology, the parent symbol in most cases will be the same as the stock symbol. For example, `MSFT` will be the parent symbol for the underlying security `MSFT`. However, the parent symbol may change based on corporate actions. If a parent symbol change is necessary in a contract adjustment (e.g. in a 3-for-2 split), the symbol `MSFT` will change to `MSFT1`, with the numeric suffix identifying this option as an adjusted, "non-standard" contract. If the standard 100 share contract (`MSFT`) is reintroduced after the adjustment and another adjustment is subsequently necessary to this contract, then the adjusted symbol would use the next numeric suffix not already in use. For example, if `MSFT` and `MSFT1` are active contracts, `MSFT` could change to `MSFT2` if another adjustment is necessary. Parent symbol changes are necessary if the adjusted contract becomes "non-standard", i.e., containing a deliverable different from the "standard" option contract originally listed by an exchange. However, in some contract adjustments, the adjusted contracts remain identical to standard contracts (e.g. in a 2-for-1 split where the resulting contracts remain 100 share options). In these cases, the parent symbol would not change. See [OCC's memo](https://infomemo.theocc.com/infomemos?number=26853) on contract adjustments for more details. In addition, OCC provides a [searchable list of symbol adjustments](https://infomemo.theocc.com/infomemo/search-memo). ### Expirations OPRA only provides date-granularity expiration data so the `expiration` field in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) will always be at UTC midnight even when the option expires during or after the trading day. ### Differences between live and historical data The `instrument_id` to `raw_symbol` mappings are not the same between live and historical data. The live `instrument_id` mappings are stable within a week, while the historical mappings can change. ### Data prior to 2023-03-28 Data prior to 2023-03-28 was not generated from OPRA packet captures, but from subsampled data. This leads to the following differences on the data for those dates: * The `cmbp-1`, `cbbo-1s` and `tcbbo` schemas are not available. The highest granularity available for BBO data is the `cbbo-1m` schema. * Due to not being generated from packet captures, the `ts_recv` field has the event timestamp, and `F_BAD_TS_RECV` is always set. * The `bid_pb_00` and `ask_pb_00` fields do not contain the venue which is showing the NBBO. * The `statistics` schema only has information for Open Interest, which is published on an aggregated basis. ### Flags normalization The `F_PUBLISHER_SPECIFIC` flag provides no additional meaning in this dataset and can be safely ignored. --- Source: https://databento.com/docs/datasets/adjustment-factors # Adjustment factors The adjustment factors dataset covers global corporate actions capital events, spanning multiple currencies for the same event. Each record includes all necessary fields to identify the country, security, event, and adjustment factor. Each record represents a unique capital event at the [listing](https://databento.com/docs/faqs/instruments-and-products.md#listings) level. - **History.** From 2018-05-01 to present. - **Coverage.** Global. - **Update frequency.** Daily. The primary use case for this dataset is to back-adjust end-of-day (EOD) per-share price data, allowing the price series to be graphed with the effects of various events (e.g., bonus issues, rights issues, consolidations) factored into earlier prices. Individual events can receive multiple updates both prior to and after the ex-date, seamlessly accounting for potentially messy corporate actions, changes, and cancellations. All per-share series, including prices, earnings-per-share, dividends-per-share, assets-per-share, cash-flow-per-share, etc., need to be adjusted before meaningful conclusions can be drawn about growth rates and trends. > **See also** > > Adjustment factors [field descriptions](https://databento.com/docs/schemas/adjustment-factors.md#fields) for further details. ### Symbol convention The `nasdaq_symbol` field is normalized to the Nasdaq convention (e.g. `BF.B`). The `local_code` field keeps the listing exchange's native form, as published by the upstream reference-data vendor. For reliable linkage to time series data, identify the security by `isin`, `us_code` (CUSIP), or `figi`. See [Symbology](https://databento.com/docs/schemas/security-master.md#symbology) on the security master schema page for the full reference. ### Exchange coverage The adjustment factors dataset has coverage across 215 exchanges worldwide. - `primary_exchange` specifies the primary exchange for the security associated with the adjustment factor event. - `operating_mic` field specifies the [ISO 10383 MIC](https://www.iso20022.org/market-identifier-codes) of the listing exchange associated with the event. See the [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum lookup table for possible values. > **Tip** > > The `operating_mic` field will match the `venue` from the listed [publishers](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md) and the Definition schema `exchange` field, which is an ISO 10383 Market Identifier Code (MIC). ### Security continuity The `security_id` maintains continuity in the following scenarios: - **Name changes.** Issuer name changes which result in a ticker symbol change (e.g. Facebook: FB -> Meta Platforms: META). - **Mergers.** Either where the company operates under a new name, or one of the original company names. - **Demergers.** Either where the companies operate under new names, or one keeps the original company name. The `security_id` does **not** have continuity in the following scenarios: - **Spin-offs.** When the parent company continues operating (the spun on companies are new securities). - **Delistings.** When the security is never relisted. ### Timing The first generation of pending factors for a particular ex-date will happen close of market the day before the ex-date. Adjustment factors are provided on the ex-date of the event, except: - When the corporate action is incomplete (missing core fields such as "Ratio New", etc.) in which case the factor will be provided as soon as the corporate action becomes complete after the ex-date. - Or, the factor calculation requires the open price of a security that has not traded yet in which case a factor will be generated after the security trades. ### Reason codes Each record has a `reason` code which distinguishes between different event types. | Value | Description | | `1` | Rights (in same class) | | `2` | Rights (in different class) | | `3` | Entitlement (in same class) | | `4` | Entitlement (in different class) | | `5` | Subdivision | | `6` | Consolidation | | `7` | Reclassification | | `8` | Security Swap | | `9` | Demerger | | `10` | Capital Return | | `11` | Distribution | | `12` | Liquidation | | `13` | Bonus (same class) | | `14` | Bonus (different class) | | `16` | Capital Reduction | | `17` | Cash Dividend | | `18` | Scrip dividend (same class) | | `19` | Both Cash and Scrip dividend | | `20` | Scrip dividend (different class) | | `21` | Capital Call | ### Status codes Included with each record is a single character `status` code which indicates how the adjustment should be handled. | Value | Description | | `'A'` | Apply | | `'P'` | Pending, indicates that this adjustment will be finalized when additional information becomes available | | `'R'` | Rescind (remove/reverse), indicates that this adjustment has been cancelled or re-issued after an adjustment record has already been published | > **Info** > > Note that rescinded adjustments can happen years after the `ex_date` for various reasons. ### Option field Often the shareholder is given the choice of taking the event in a few different ways, for example a dividend might be paid in USD or GBP and the shareholder needs to choose how they wish to receive it, alternatively it might be paid as cash or script or a combination of both, again the shareholder needs to choose how they wish to receive the dividend. Also de-mergers and distributions might have different options for the shareholder to consider. In all cases, the `option` field will contain an integer value ranging from 1 to 9, representing the available choices. You must select one of these options to proceed. If you're unsure which option to choose, default to 1. > **Warning** > > Never apply multiple adjustments with the same `ex_date`, `event_id` and different options. ### Corporate actions relationship Generally, there is a one-to-one relationship to the underlying corporate action - however in a few circumstances this is not true. - When a dividend delivers both cash and script, the corporate action specifies this as a dividend type of "Both". This is split into two separate adjustment records, one for the cash component and the other for the script component, which can be recombined later as desired. - When an event is linked to a previous event they will be combined into a single adjustment record. For example in Stockholm, they often return cash to shareholders in the form of free redemption rights (RDR) followed by a return of capital on those redemption rights. When this happens only one adjustment record will be generated for both corporate actions. > **Tip** > > Where you see both cash and script dividend adjustment records with the same `event_id`, `ex_date` and `option` you can optionally recombine the two by multiplying the factors together. ### Applying adjustment factors > **See also** > > [Applying adjustment factors](https://databento.com/docs/examples/adjustment-factors/applying-adjustment-factors.md) tutorial. Decide which type of adjustments you want to apply (`DIV`, `BON`, etc) then use the `reason` field to access only those adjustments and ignore the others. Prior to using any adjustment factor check that no other adjustment record has the same `event_id` and `ex_date`, if it has then check the `option` - you should only use one of the options for the same `event_id` and `ex_date`. This is common when a shareholder has the choice of taking a dividend in multiple currencies and cash or script selection. Refer to the description of the [option](https://databento.com/docs/datasets/adjustment-factors.md#option-field) field above for further information. Compare the factor to the sentiment value, if available, they should be roughly the same if they are significantly different then the factor should be used with caution or even modified to bring it closer to sentiment (this is purely your prerogative). Multiply all historical prices, up to but not including the `ex_date`, by the `factor`. Optionally divide all historical volumes, up to but not including the `ex_date`, by the `factor` (for consolidations and subdivisions only). > **Info** > > Be prepared for corrections to come through, at a later date, in the form of rescinded adjustment records. In these cases you need to reverse or remove the original adjustment. ### Negative factors These are rare but can happen. For example, when a `DIV` (dividend) is paid that is greater than the share price, on the `ex_date`, or a `RCAP` (return of capital) that is also greater than the share price, on the `ex_date`. This type of event can happen when a fund is being closed down and final distributions made in the form of a dividend or return of capital. ### Factors of 1 A factor of "1" can either be a valid factor or default factor where one could not be calculated. Default factors are 1.0 because this is mathematically neutral and can be used as a placeholder when a factor cannot be calculated. Reasons for not being able to calculate factors include: - No stock close price when a stock close price required. - No issue open price when an issue open price required. - Unable to perform currency conversion when required. Factors of "1" are valid in following circumstances: - When a `RTS` (Rights) or `ENT` (Entitlement) issue makes the issue available at a price that is equal to or above the current stock price (issue price > stock price). - When a company does a `RTS` (Rights) or `ENT` (Entitlement) issue of options with an exercise date far in the future the impact may be of little significance or difficult to determine due to uncertainty. - `CAPRD` (Capital reduction) of 1 for 1. ### Reissued adjustments Adjustments will be re-issued and the original adjustment record rescinded, if any of the corporate action key fields change after the date the adjustment was first published. For example if the `ratio_new` or `ratio_old` change, after the date of first publication, then the original adjustment will be rescinded (after March 2010) and a new adjustment will be issued. ### Rescinded adjustments Adjustments are rescinded if the original adjustment is cancelled or re-issued after the publication date (usually the `ex-date`). Where an adjustment has been rescinded this will need to be removed/reversed out of your system. If you are loading the adjustment records into a database then you will need to remove the rescinded adjustment record from your database. If you are back-adjusting price series data then you will need to reverse the rescinded adjustment record from your price series data using the inverted factor to reverse the original adjustment. It is recommended to process rescinded adjustment records before processing any re-issued adjustment records. > **Info** > > During the day of publication, if an adjustment is cancelled or re-issued it is not rescinded; the change is simply reflected in the next update. > For example, if a dividend is cancelled on the day of publication then the interim pending record before cancellation will contain an adjustment record > for that dividend. A pending record after the cancellation will not contain an adjustment record (it is not rescinded, it is simply gone). --- Source: https://databento.com/docs/datasets/corporate-actions # Corporate actions The corporate actions dataset includes coverage from 215+ exchanges worldwide, 310,000+ listed and delisted securities from 100,000+ companies, categorized as 60+ event types. The dataset focuses on crucial events affecting global securities such as dividends, takeovers, mergers & acquisitions and rights issues. Each record represents a unique corporate actions event at the [listing](https://databento.com/docs/faqs/instruments-and-products.md#listings) level. - **History.** From 2018-05-01 to present. - **Coverage.** Global. - **Update frequency.** 4x per day. - **Listing level.** All records are maintained at the listing level, which means each individual listing for a security has its own set of records. - **Delisting and relisting.** The system continues to track securities even after they have been delisted. This is crucial because securities are occasionally relisted, so maintaining a continuous record prevents gaps in the time series thus avoiding ambiguity. > **See also** > > Corporate actions [field descriptions](https://databento.com/docs/schemas/corporate-actions.md#fields) for further details. ### Symbol convention The `nasdaq_symbol` field is normalized to the Nasdaq convention (e.g. `BF.B`). The `local_code` field keeps the listing exchange's native form, as published by the upstream reference-data vendor. For reliable linkage to time series data, identify the security by `isin`, `us_code` (CUSIP), or `figi`. See [Symbology](https://databento.com/docs/schemas/security-master.md#symbology) on the security master schema page for the full reference. ### Securities and listings A security is a fungible, tradable financial asset listed on regulated exchanges. Each security has a primary exchange listing and may be listed on multiple secondary exchanges. The corporate actions dataset utilizes a `security_id` to associate these various listings. Each public listing is identified by a unique `listing_id` that links together a series of point-in-time (PIT) records. The records are ordered by the `event_date` and `ts_record` timestamp. A `listing_group_id` groups all listings for the same security on a specific exchange, including secondary listings such as those in different trading currencies. ### Events Corporate actions can be broadly categorized into three types of events at both the **security** and **listing** levels: - **General events.** Announcements, company meetings, lawsuits/class actions, listing status changes, incorporation change, security description change, etc. - **Capital change events.** Mergers, stock splits / reverse stock splits, return of capital, franking, liquidation, divestments, dividends, currency redenomination, etc. - **Exchange events.** New listing, security reclassification, security swap, preferential offer, etc. The `event` field categorizes the type of corporate action and is the primary field for filtering and categorizing events at a granular level. Event records can be paired with an `event_id`, which links associated events, such as all payment rows. The **top ten** event types are listed in descending order based on their frequency of occurrence, see the [EVENT](https://databento.com/docs/conventions/reference-data-enums.md#event) enum lookup table for possible values, and [Corporate actions events](https://databento.com/docs/conventions/corporate-actions-events.md) for details about each event type. | Value | Description | | `SHOCH` | **Shares Outstanding Change**: The number of shares outstanding has been adjusted due to a capital event. | | `DIV` | **Capital Distribution**: A distribution of assets to shareholders, usually in the form of dividends. | | `AGM` | **Annual General Meeting**: A yearly gathering of a company's interested shareholders. | | `NLIST` | **New Listing**: The company's securities are now listed on an exchange. | | `ANN` | **Announcement**: Public communication regarding significant company events. | | `LSTAT` | **Listing Status**: Changes in the status of the company's listing on the exchange. | | `MKCHG` | **Market Segment Change**: The company has moved to a different market segment classification. | | `SDCHG` | **Sedol Change**: The company's SEDOL (**S**tock **E**xchange **D**aily **O**fficial **L**ist) identifier has been changed. | | `BBCC` | **Bloomberg Composite ID Change**: The Bloomberg Composite Identifier for the company has been changed. | | `BBEC` | **Bloomberg Exchange ID Change**: The Bloomberg Exchange Identifier for the company has been changed. | | `DIVRC` | **Dividend Reclassification**: The classification of dividend payments has been changed. | | `RD` | **Record Date**: The date on which a shareholder must own shares to be entitled to a dividend or other distribution. | | `ICC` | **International Code Change**: The international identification code for the company's securities has been changed. | | `DRIP` | **Dividend Reinvestment Plan**: A plan that allows shareholders to reinvest their cash dividends into additional shares. | | `LCC` | **Local Code Change**: The ticker symbol for the company's securities has been changed. | ### Event subtypes The `event_subtype` field is only used for a limited number of events which fall into distinct sub-groupings, see the [EVENTSUBTYPE](https://databento.com/docs/conventions/reference-data-enums.md#eventsubtype) enum lookup table for all possible values, and [Corporate actions events](https://databento.com/docs/conventions/corporate-actions-events.md) for a list of subtypes for each `event` code. | Value | Description | | `AGM` | **Annual General Meeting**: The table name implies Annual General Meeting, but there are several other possible meeting types categorized under this event subtype. | | `BB` | **Buyback**: Buyback events can also be used to transmit Dutch auction events. | | `BON` | **Bonus**: Bonus events can also be used to transmit depository receipt dividends, supplied by a depository bank in lieu of a bonus issue on the underlying security. | | `CONV` | **Conversion**: Conversion events are occasionally used to bundle data for different event types, e.g., offers. These need to be clearly distinguished from actual conversions. | | `DIV` | **Dividend**: Dividend events can also be used to transmit capital distribution and capital gain events. Certain interest payments are also mapped into dividends. | | `DIVIF` | **Dividend Income or Accumulation**: The subtype transmits accumulation or income, and also dividend, interest, or property-related events. | | `ENT` | **Entitlement**: Entitlement events can contain either non-renounceable rights or an offer event. | | `RCAP` | **Return of Capital**: Return of capital events have several distinct types, each of which can significantly impact client-side processing. | | `RTS` | **Rights**: Rights events can also be used to transmit depository receipt dividends, supplied by a depository bank in lieu of a rights issue on the underlying security. | | `REDEM` | **Redemption**: Redemption events are occasionally used to bundle data for different event types, e.g., offers. These need to be clearly distinguished from actual redemptions. | | `TKOVR` | **Takeover**: Takeover events are used as an umbrella term for several underlying event scenarios. | | `MRGR` | **Merger**: If a close date but no open date is populated, it indicates a mandatory event. | | `TKOVRMINI` | **Mini-Takeover**: If flagged as a Mini-Takeover, it is considered a voluntary event. | | `TENDMRGR` | **Tender Merger**: If close and open dates are populated and the event is still ongoing, or has lapsed, it is considered a voluntary event. | | `TEND` | **Tender Offer**: If close and open dates are populated and the event is still ongoing, or has lapsed, it is considered a voluntary event. | ### Security types The **top ten** security types covered are listed in descending order based on their frequency of occurrence, see the [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum lookup table for possible values. | Value | Description | | `ETF` | Exchange Traded Fund | | `EQS` | Equity Shares | | `DR` | Depository Receipts | | `PRF` | Preference Share | | `UNT` | Units | | `WAR` | Warrants | | `ETC` | Exchange Traded Commodities | | `TRT` | Tradeable Rights | | `STP` | Stapled Security | | `BND` | Bond | ### Exchange coverage The corporate actions dataset has coverage across 220+ exchanges worldwide. - `primary_exchange` specifies the primary exchange for the security associated with the event. - `exchange` field specifies the exchange for the listing and can be filtered by specifying a `dataset`. - `operating_mic` field specifies the [ISO 10383 MIC](https://www.iso20022.org/market-identifier-codes) of the listing exchange associated with the event. See the [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum lookup table for possible values. > **Tip** > > The `operating_mic` field will match the `venue` from the listed [publishers](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md) and the Definition schema `exchange` field, which is an ISO 10383 Market Identifier Code (MIC). ### Listing continuity The `listing_id` and `security_id` have continuity with changes to the underlying security in the following scenarios: - **Name changes.** Issuer name changes which result in a ticker symbol change (e.g. Facebook: FB -> Meta Platforms: META). - **Mergers.** Either where the company operates under a new name, or one of the original company names. - **Demergers.** Either where the companies operate under new names, or one keeps the original company name. The `listing_id` and `security_id` do **NOT** have continuity in the following scenarios: - **Spin-offs.** When the parent company continues operating, the spun-off companies become new listings. - **Delistings.** When the security is never relisted. ### Point-in-time (PIT) corporate actions The `corporate_actions.get_range` endpoint provides a time series of point-in-time (PIT) records for the queried symbols and events. This is ideal for research or backtesting where it's crucial to include records as they were known at specific points in time. Note that, like other APIs, the time ranges have an exclusive end. The Python client library offers a `pit` parameter to control record retention: - If True: All historical records for each `event_unique_id` will be retained, preserving the complete point-in-time history. - If False (default): The DataFrame will include only the most recent record for each `event_unique_id`, based on the `ts_record` timestamp. ### Split and restructuring events The corporate actions dataset includes normalization of several stock restructuring events, such as splits, spin-offs, and demergers, as shown in the following table: | Event type | Event subtype | Description | | `FSPLT` (Forward Split) | `DIV` (Dividend) | Stock dividend (represented as a forward split) | | `FSPLT` (Forward Split) | `BON` (Bonus) | Bonus (represented as a forward split) | | `RSPLT` (Reverse Split) | `CONSD` (Consolidation) | Consolidation (represented as a reverse split) | | `SOFF` (Spin-Off) | `DIST` (Distribution) | Distribution resulting from a spin-off | | `SOFF` (Spin-Off) | `DMRGR` (Demerger) | Demerger resulting in a spin-off | | `FSPLT` (Forward Split) | `SD` (Subdivision) | Subdivision (represented as a forward split) | ### Payment ratios For `FSPLT`, `DIV`, and `BON` events, the `ratio_new` includes the existing holdings. As a result, some reported ratios may differ from the actual figures in official announcements. --- Source: https://databento.com/docs/datasets/security-master # Security master > **Info** > > The Databento team is currently working on this article. The security master provides point-in-time (PIT) static information on securities across international markets for 860,000+ listed and delisted securities. This provides a means for resolving securities attributes, identification, symbology and informational challenges efficiently. Some common use cases for the security master include research, backtesting, compliance and report generation, risk management, and keeping portfolios current. - **History.** From 2005-01-01 to present. - **Coverage.** Global. - **Update frequency.** Once per week. - **Listing level.** All records are maintained at the listing level, which means each individual listing for a security has its own set of records. - **Delisting and relisting.** The system continues to track securities even after they have been delisted. This is crucial because securities are occasionally relisted, so maintaining a continuous record prevents gaps in the time series thus avoiding ambiguity. > **See also** > > Security master [field descriptions](https://databento.com/docs/schemas/security-master.md#fields) for further details. ### Symbol convention The `nasdaq_symbol` field is normalized to the Nasdaq convention (e.g. `BF.B`). The `local_code` field keeps the listing exchange's native form, as published by the upstream reference-data vendor. For reliable linkage to time series data, identify the security by `isin`, `us_code` (CUSIP), or `figi`. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the convention reference and per-dataset map. > **Warning** > > For symbols with suffix codes (class shares, warrants, preferreds, etc.) listed on `exchange=USNYSE`, the upstream reference-data vendor delivers the symbol in a no-delimiter form that the normalization step cannot recover. > In these cases, `nasdaq_symbol` is left in the no-delimiter form (e.g. `BRKB` for Berkshire Hathaway Class B, `HEIA` for Heico Class A) and does not match the time series `raw_symbol` for the same security. > Cross-listings of the same security on other US exchanges (e.g. `USCIN`, `USAMEX`) are unaffected. > Matching by `isin`, `us_code`, or `figi` is the reliable workaround. ### Securities and listings A security is a fungible, tradable financial asset listed on regulated exchanges. Each security has a primary exchange [listing](https://databento.com/docs/faqs/instruments-and-products.md#listings) and may be listed on multiple secondary exchanges. The security master utilizes a `security_id` to associate these various listings. Each public listing is identified by a unique `listing_id` that links together a series of point-in-time (PIT) records. The records are ordered by the `ts_effective` timestamp, which is when the information was valid from. A `listing_group_id` groups all listings for the same security on a specific exchange, including secondary listings such as those in different trading currencies. ### Security types The **top ten** security types covered are listed in descending order based on their frequency of occurrence: | Value | Description | | `EQS` | Equity Shares | | `ETF` | Exchange Traded Fund | | `MF` | Mutual Fund | | `DR` | Depository Receipts | | `TRT` | Tradeable Rights | | `WAR` | Warrants | | `BND` | Bond | | `PRF` | Preference Share | | `UNT` | Units | | `ETC` | Exchange Traded Commodities | | `STP` | Stapled Security | See the [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum lookup table for possible values. ### Exchange coverage The security master has coverage across 220+ exchanges worldwide. - `primary_exchange` field specifies the primary exchange for the security. - `exchange` field specifies the exchange for the listing and can be filtered by specifying a `dataset`. - `operating_mic` field specifies the [ISO 10383 MIC](https://www.iso20022.org/market-identifier-codes) of the listing exchange. See the [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum lookup table for possible values. > **Tip** > > The `operating_mic` field will match the `venue` from the listed [publishers](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md) and the Definition schema `exchange` field, which is an ISO 10383 Market Identifier Code (MIC). ### Listing continuity The `listing_id` and `security_id` have continuity with changes to the underlying security in the following scenarios: - **Name changes.** Issuer name changes which result in a ticker symbol change (e.g. Facebook: `FB` -> Meta Platforms: `META`). - **Mergers.** Either where the company operates under a new name, or one of the original company names. - **Demergers.** Either where the companies operate under new names, or one keeps the original company name. The `listing_id` and `security_id` do **NOT** have continuity in the following scenarios: - **Spin-offs.** When the parent company continues operating, the spun-off companies become new listings. - **Delistings.** When the security is never relisted. ### Enriching the definition schema The security master data can be used to enrich instrument definitions by merging two pandas dataframes by an inner join, using the `definition` schema 'symbol' field and the security master 'symbol' field (where `stype_in` is `raw_symbol`). ```{python} df_combined = pd.merge( df_definition, df_security_master, left_on="symbol", right_on="symbol", how="inner", ) ``` This assumes Nasdaq Integrated Platform Suffix is used for the definition schema instrument (such as those obtained from the `XNAS.ITCH` dataset), otherwise joining on the definitions `raw_symbol` may be necessary. > **See also** > > [Enriching instrument definitions](https://databento.com/docs/examples/security-master/enrich-instrument-definitions.md) tutorial. --- Source: https://databento.com/docs/examples?lang=python # Examples ", "[Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md)", "[Custom OHLCV bars from trades](https://databento.com/docs/examples/basics-historical/custom-ohlcv.md)" ], "LIVE DATA": [ "[Stream live data to a file](https://databento.com/docs/examples/basics-live/live-stream-to-file.md)", "[Handle multiple record types](https://databento.com/docs/examples/basics-live/live-dispatch.md)", "[Estimate latency](https://databento.com/docs/examples/basics-live/live-latency.md)" ] } }, { "title": "Advanced applications", "subtitle": "Develop systematic strategies and algorithmic trading applications.", "columns": { "HISTORICAL DATA": [ "[Limit order book construction](https://databento.com/docs/examples/order-book/limit-order-book.md)", "[Execution slippage and markouts](https://databento.com/docs/examples/algo-trading/execution-slippage.md)", "[Machine learning prediction models](https://databento.com/docs/examples/algo-trading/machine-learning.md)" ], "LIVE DATA": [ "[A high-frequency liquidity-taking strategy](https://databento.com/docs/examples/algo-trading/high-frequency.md)", "[Symbology mapping for live data](https://databento.com/docs/examples/symbology/live-symbol-mapping.md)", "[Calculate TICK and TRIN indicators](https://databento.com/docs/examples/basics-live/live-tick-trin.md)" ] } } ] } }) ", "[Create a candlestick plot](https://databento.com/docs/examples/basics-historical/candlestick-plot.md)", "[Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md)", "[Top pre-market movers](https://databento.com/docs/examples/equities/equities-premarket-movers.md)", "[Matching engine latencies](https://databento.com/docs/examples/algo-trading/latency.md)", "[Estimate implied volatility](https://databento.com/docs/examples/options/estimating-implied-volatility.md)" ] } ], [ { "title": "C++", "columns": [ "[Best bid, best offer, and midprice](https://databento.com/docs/examples/basics-historical/midprice.md)", "[Programmatic batch downloads](https://databento.com/docs/examples/basics-historical/programmatic-batch-download.md)" ] }, { "title": "Rust", "columns": [ "[Types of order book events](https://databento.com/docs/examples/order-book/order-actions.md)", "[End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md)" ] } ] ] } }) **Can't find an example for your use case?** Our team can help. **[Contact support](https://databento.com/support)** or **[join our Slack community](https://to.dbn.to/slack)** to chat with developers. --- Source: https://databento.com/docs/examples/equities?lang=python ## Equities | Title | Subtitle | Image | Supported languages | | [Equities: Introduction](https://databento.com/docs/examples/equities/equities-introduction.md) | Get started with equities data by fetching top-of-book quotes and last sale. | ![Equities introduction image](https://databento.com/docs/equities-introduction.svg.md) | Python, Equities, Historical | | [Top pre-market movers](https://databento.com/docs/examples/equities/equities-premarket-movers.md) | Get a ranking of the biggest gainers in the Nasdaq pre-market session. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Rust, Equities, Historical | | [Find average spread for a symbol](https://databento.com/docs/examples/equities/spread-boxplots.md) | Find the average spread for a symbol across the different equity venues. | ![Spread boxplots](https://databento.com/docs/spread-boxplots-thumbnail.svg.md) | Python, Equities, Historical | | [Calculate synthetic NBBO from prop feeds](https://databento.com/docs/examples/equities/consolidated-bbo.md) | Consolidate MBP-1 from prop feeds to get an improved microprice vs. SIP NBBO. | ![Image placeholder](https://databento.com/docs/consolidated-bbo-thumbnail.svg.md) | Python, Historical | | [Get daily closing prices for equities](https://databento.com/docs/examples/equities/closing-prices.md) | Retrieve the official consolidated closing prices for US equities. | ![Image placeholder](https://databento.com/docs/closing-prices-thumbnail.svg.md) | Python, Historical | | [Analyze auction imbalance dynamics](https://databento.com/docs/examples/equities/auction-imbalance.md) | Compare the closing auction process for different US equity exchanges. | ![Image placeholder](https://databento.com/docs/auction-imbalance-thumbnail.svg.md) | Python, Historical | --- Source: https://databento.com/docs/examples/equities/equities-introduction?lang=python # Equities: Introduction > **Info** > > This article introduces additional concepts to get started with equities data on > Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll show how to: - Find an equities dataset - Fetch top-of-book quotes and last sale - Stream live data OHLCV data - Explain special conventions for equities vs. other asset classes on Databento ## Finding an equities dataset Our US Equity data provides full coverage of all RegNMS symbols, which includes but is not limited to: - Stocks (e.g. AAPL) - ETFs (e.g. SPY) - Warrants (e.g. GME+) While a symbol has a primary listing on one exchange, it may trade on all exchanges. We have coverage of the individual prop feeds from the various equity exchanges. Each prop feed will have its own dataset. To use equities data on Databento, first navigate to our [equities catalog page]($PLATFORM_HOSTNAME/catalog/us-equities). ![Databento Equities landing](https://databento.com/docs/equities-landing.png.md) Next, click on the dataset of your choice. ![Dataset Name](https://databento.com/docs/equities-dataset-id.png.md) For this example, we'll use the [Nasdaq TotalView-ITCH]($PLATFORM_HOSTNAME/portal/catalog/us-equities#XNAS.ITCH) dataset, whose dataset name is `XNAS.ITCH`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## Getting MBP-1 data The easiest way to fetch top-of-book quotes and last sale is by requesting the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md), which gives all trades and all book updates that affect the top-of-book; some data providers refer to this as "L1" data. > **Info** > > Note that MBP-1 should not be mistaken with the TBBO schema. > MBP-1 is a superset of the TBBO schema, which is essentially MBP-1 > sampled in trade space. See our [FAQ page](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md) for more information on the differences between these schemas. ```python import databento as db import matplotlib.pyplot as plt client = db.Historical("$YOUR_API_KEY") dataset = "XNAS.ITCH" symbol = "AAPL" start = "2024-10-28T12:30:00" end = "2024-10-28T12:45:00" data = client.timeseries.get_range( dataset=dataset, schema="mbp-1", symbols=symbol, start=start, end=end, ) df = data.to_df() df = df.rename(columns={"bid_px_00": "bid", "ask_px_00": "ask"}) ax = df[["bid", "ask"]].plot(drawstyle="steps-post") df[df["action"] == "T"]["price"].plot( ax=ax, style=".", markersize=10, label="trade", color="C2", zorder=2, ylabel="Price", xlabel="Time", title=symbol, ) plt.legend() plt.show() ``` ![AAPL price plot](https://databento.com/docs/equities-price-plot-0.svg.md) ## Streaming live OHLCV data The [Databento US Equities Mini](https://databento.com/docs/datasets/equs-mini.md) (`EQUS.MINI`) dataset is our most cost-effective solution for real-time, top-of-book US equities data. It combines multiple trading venues into one composite dataset that provides accurate best bid and offer (BBO) quotes, liquidity, and trade volume data--all without exchange license fees. In this example, we will use the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md) and print out records every second. A full list of available schemas can be found on the [dataset documentation page](https://databento.com/docs/datasets/equs-mini.md). > **Info** > > This example requires a live data subscription to Databento US Equities. For more information, please visit our [equities catalog page]($PLATFORM_HOSTNAME/portal/catalog/us-equities). ```python import databento as db # Enable some basic logging db.enable_logging("INFO") # Create a live client and connect live_client = db.Live("$YOUR_API_KEY") # Subscribe to the OHLCV-1s schema for AMZN live_client.subscribe( dataset="EQUS.MINI", schema="ohlcv-1s", symbols="AMZN", ) # We add a print callback to view each record live_client.add_callback(print) # Start the live client to begin data streaming live_client.start() # Run the stream for 15 seconds before closing live_client.block_for_close(timeout=15) ``` > **Info** > > If you do not see any output, it could be because the markets are closed. See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) for using [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=853, ts_event=1769047202984118060, pretty_ts_event='2026-01-22T02:00:02.984118060Z', stype_in=None, stype_in_symbol='AMZN', stype_out=None, stype_out_symbol='AMZN', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768913037000000000, pretty_ts_event='2026-01-20T12:43:57.000000000Z', open=233480000000, pretty_open=233.48, high=233480000000, pretty_high=233.48, low=233480000000, pretty_low=233.48, close=233480000000, pretty_close=233.48, volume=15) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914152000000000, pretty_ts_event='2026-01-20T13:02:32.000000000Z', open=233370000000, pretty_open=233.37, high=233390000000, pretty_high=233.39, low=233370000000, pretty_low=233.37, close=233390000000, pretty_close=233.39, volume=103) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914160000000000, pretty_ts_event='2026-01-20T13:02:40.000000000Z', open=233490000000, pretty_open=233.49, high=233490000000, pretty_high=233.49, low=233490000000, pretty_low=233.49, close=233490000000, pretty_close=233.49, volume=65) ... OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914714000000000, pretty_ts_event='2026-01-20T13:11:54.000000000Z', open=233500000000, pretty_open=233.5, high=233500000000, pretty_high=233.5, low=233490000000, pretty_low=233.49, close=233490000000, pretty_close=233.49, volume=100) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914724000000000, pretty_ts_event='2026-01-20T13:12:04.000000000Z', open=233570000000, pretty_open=233.57, high=233570000000, pretty_high=233.57, low=233570000000, pretty_low=233.57, close=233570000000, pretty_close=233.57, volume=50) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768915049000000000, pretty_ts_event='2026-01-20T13:17:29.000000000Z', open=233320000000, pretty_open=233.32, high=233320000000, pretty_high=233.32, low=233320000000, pretty_low=233.32, close=233320000000, pretty_close=233.32, volume=25) ``` ## Special conventions for equities on Databento - **Odd lots.** All order quantities are expressed in units of 1 share. Databento's equities data generally includes odd lots as we source our data directly from prop feeds. If you've switched to Databento from a data provider that only distributes CTA/UTP SIP data, you may have never encountered order quantities for less than 100 shares as the SIPs only provide round lots. - **Explicit trade aggressor side.** Equities prop feeds usually provide the aggressor side of each trade explicitly. Hence, the `side` field in our data is exact; it is not inferred using a trade classification algorithm. If the aggressor side is not known, we mark it with a side of `N`. If you've switched to Databento from a data provider that only distributes CTA/UTP SIP data, you may not have dealt with explicit trade aggressor side before but should try to make use of this field as it provides valuable premia over the SIPs. - **Ephemeral and Databento-assigned permanent instrument IDs.** Several equities venues do not have a system of numeric instrument IDs that correspond to each ticker symbol; these venues rely solely on the ticker symbols to identify each instrument. On certain venues like Nasdaq, the venue does assign instrument IDs to each instrument that can be used alongside the ticker symbols, but such venue-assigned instrument IDs are ephemeral and rotate daily. This is unlike venues like CME Globex, where the venue-assigned instrument IDs are typically permanent until the instrument expires. Where the venue assigns an instrument ID, we pass this directly through--so the instrument IDs of a venue like Nasdaq may be difficult to work with. Where the venue doesn't assign an instrument ID, we'll generate our own synthetic instrument ID for consistency across all venues. - **Atomic executions and passive-side fills.** On most equities venues, the fill event and order deletion is published at the same time as a single, atomic execution event. In such cases, Databento represents each execution event with action `F`, implying the passive-side fill, and provides a synthetic trade event with action `T` to represent the aggressor side of the trade. Note that this is merely for convenience, so you can reuse the same book management logic for all delta one products like equities and futures--in such cases that the aggressor size is not explicit since we only know the sizes of the passive fills corresponding to the trade, and hence the actual aggressor size needs to be inferred. - **Trade conditions.** We do not provide trade condition codes as those are not provided on the prop feeds. - **Instrument definitions.** Equities instrument definitions tend to be much sparser than those of derivatives like options and futures, so you should expect to see many null values when you fetch instrument definitions. ## Symbology In US equities, the same stock may be traded on different venues, but have slightly different ticker symbols on each venue. Examples include `BRK.B` and `AAC+` in the Nasdaq convention vs. `BRK B` and `AAC WS` in the CMS convention. In addition to the `raw_symbol` symbology, which uses the publisher's convention, we also offer the `cms` and `nasdaq` symbology types to allow explicitly using one convention. For Databento consolidated equities datasets and Nasdaq TotalView-ITCH, `raw_symbol` and `nasdaq` symbology types are equivalent. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ## Numeric publisher IDs Consolidated equities feeds include data from several publishers. The `publisher_id` field included with each record indicates the source venue of the record. These IDs can be converted to publisher names through the `Publisher` enum. > **See also** > > [End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md) and > [Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md). --- Source: https://databento.com/docs/examples/equities/equities-premarket-movers?lang=python # Top pre-market movers ## Overview This example demonstrates how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve OHLCV-1m bar aggregate data to analyze the largest movers in the Nasdaq pre-market. Equities which are trading in the early pre-market may be reacting to recent events or news announcements, relevant to a single stock, an industry, or an entire sector. This notable pre-market activity usually manifests in large price changes and volume, which significantly exceed the normal range, and may be indicative of subsequent moves after the session open. For our purposes we'll define the pre-market as 4:00 AM to 9:30 AM in the 'US/Eastern' time zone, which accounts for changes for daylight savings. You can find more information on Nasdaq trading hours [here](https://www.nasdaq.com/stock-market-trading-hours-for-nasdaq). ## Nasdaq dataset For this example, we'll use the [Nasdaq TotalView-ITCH]($PLATFORM_HOSTNAME/datasets/XNAS.ITCH) dataset, whose dataset ID is `XNAS.ITCH`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## OHLCV-1m schema The data we need will come from the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low, and closing prices, as well as the aggregated volume of trades occurring within each time interval. Since we are interested in a specific market session, we'll use an interval of one minute, which is specified by the suffix `-1m`. ## Example ```python import databento as db import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, request OHLCV-1m bars for the time range we're interested in data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="ohlcv-1m", symbols="ALL_SYMBOLS", start=pd.Timestamp("2023-06-06T04:00", tz="US/Eastern"), end=pd.Timestamp("2023-06-06T09:30", tz="US/Eastern"), ) # We need to obtain the symbology mappings for 'ALL_SYMBOLS' symbology = data.request_symbology(client) # This allows us to populate the 'symbol' column when we convert to a DataFrame data.insert_symbology_json(symbology) # Convert to dataframe with timestamps in the US/Eastern timezone df = data.to_df(tz="US/Eastern") # Combine the first open, last close and volume sum into a DataFrame df_moves = df.groupby("symbol").agg({"open": "first", "close": "last", "volume": "sum"}) # Calculate the change from start to end of pre-market df_moves["change"] = (df_moves["close"] - df_moves["open"]) / df_moves["open"] df_moves = df_moves.sort_values(by="change", ascending=False) # Display the top 10 largest movers print(df_moves.head(10)) ``` ## Result ```plaintext open close volume change symbol CRGOW 0.1564 0.3200 1977 1.046036 ZVZZT 13.0200 25.0000 369 0.920123 MLECW 0.0974 0.1798 3394 0.845996 HOTH 2.0300 3.5400 1600550 0.743842 BOAC+ 0.0458 0.0782 700 0.707424 RCRT 0.1918 0.2953 575273 0.539625 ATMCR 0.1489 0.2237 400 0.502351 SBFMW 0.5938 0.8915 305 0.501347 RCRTW 0.0229 0.0302 300 0.318777 TCBPW 0.0436 0.0573 300 0.314220 ``` > **See also** > > [End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md) and > [Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md). --- Source: https://databento.com/docs/examples/equities/spread-boxplots?lang=python # Find average spread for a symbol There are 16 US equity exchanges that provide trading for RegNMS symbols. Knowing the liquidity characteristics for each venue is necessary in understanding the microstructure of the equity markets. Cboe publishes the [US Equities Market Volume Summary report](https://www.cboe.com/us/equities/market_statistics/) that indicates the relative market volume share for each venue. Databento provides coverage for all equity exchanges except LTSE, which provides <0.01% of total market volume. ## Overview In this example, we will use the [historical client](https://databento.com/docs/api-historical/client.md) to get the average spread for SPY across the different equity venues before and after an increase in volatility. We'll use the [BBO-1s schema](https://databento.com/docs/schemas/bbo.md) which provides the bid and ask price subsampled in 1-second intervals to calculate the spread in basis-points (BPS). We'll compare these pre-volatility and post-volatility figures for the different venues. ## Example ```python import datetime as dt import databento as db import matplotlib.pyplot as plt import pandas as pd import seaborn as sns # Set parameters symbol = "SPY" prevol_date = dt.date(2025, 4, 1) # Tuesday before volatility spike postvol_date = dt.date(2025, 4, 8) # Tuesday after volatility spike # Regular Trading Hours (RTH) for US equities rth_start = dt.time(9, 30) rth_end = dt.time(16, 0) equity_datasets = [ "XNAS.ITCH", # Nasdaq "XBOS.ITCH", # Nasdaq Texas "XPSX.ITCH", # Nasdaq PSX "XNYS.PILLAR", # NYSE "ARCX.PILLAR", # NYSE Arca "XASE.PILLAR", # NYSE American "XCHI.PILLAR", # NYSE Texas "XCIS.TRADESBBO", # NYSE National "MEMX.MEMOIR", # Members Exchange "EPRL.DOM", # MIAX Pearl "IEXG.TOPS", # IEX "BATS.PITCH", # Cboe BZX "BATY.PITCH", # Cboe BYX "EDGA.PITCH", # Cboe EDGA "EDGX.PITCH", # Cboe EDGX # Databento does not provide coverage for LTSE (MIC: LTSE) ] client = db.Historical(key="$YOUR_API_KEY") venue_map = {x["dataset"]: x["venue"] for x in client.metadata.list_publishers()} def calculate_spread( dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """ Calculate spread size in basis points (bps) """ df = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="bbo-1s", # BBO-1s is required for this example start=date, ).to_df(tz="America/New_York") df = df.resample("1s").ffill().loc[rth_start:rth_end] # Calculate spread in basis points df["midpoint"] = (df["ask_px_00"] + df["bid_px_00"]) / 2 df["spread_abs"] = (df["ask_px_00"] - df["bid_px_00"]).clip(lower=0) df["spread_bps"] = df["spread_abs"] / df["midpoint"] * 1e4 df["Venue"] = venue_map[dataset] df["Date"] = df.index.date return df[["Venue", "Date", "spread_bps"]] df_list: list[pd.DataFrame] = [] for date in (prevol_date, postvol_date): for dataset in equity_datasets: df_list.append(calculate_spread(dataset, symbol, date)) df = pd.concat(df_list, ignore_index=True) plt.figure(figsize=(16, 8)) ax = sns.boxplot( data=df, x="Venue", y="spread_bps", hue="Date", order=df[df["Date"] == postvol_date] .groupby("Venue")["spread_bps"] .median() .sort_values() .index, gap=0.1, width=0.8, showfliers=False, ) # Set colors for each venue n_venues = len(equity_datasets) venue_colors = sns.color_palette("Set1", n_colors=n_venues) for i, patch in enumerate(ax.patches): patch.set_facecolor(venue_colors[i % n_venues]) # Color the post-volatility boxplots a lighter shade if i >= n_venues: patch.set_facecolor((*patch.get_facecolor()[:3], 0.4)) # Adjust legend prevol_patch, postvol_patch = ax.get_legend().get_patches() prevol_patch.set_facecolor(venue_colors[0]) postvol_patch.set_facecolor((venue_colors[0], 0.4)) plt.legend(handles=[prevol_patch, postvol_patch], labels=[f"{prevol_date}", f"{postvol_date}"]) plt.xticks(rotation=45) plt.ylabel("Spread size (bps)") plt.title(f"{symbol} spreads before and after April 2025 tariff crash") plt.tight_layout() plt.show() ``` The venues with the highest market share, such as NYSE Arca (`ARCX`) and Nasdaq (`XNAS`), have spreads that remain relatively tight even after the increase in volatility. Venues with a low market share see the largest increase in spreads post-volatility. ![Spread boxplot](https://databento.com/docs/spread-boxplots-0.svg.md) --- Source: https://databento.com/docs/examples/equities/consolidated-bbo?lang=python # Calculate synthetic NBBO from prop feeds ## Overview In this example, we will calculate a synthetic NBBO by taking the best bid and offer seen across the different exchanges. Unlike the official SIP NBBO which ignores odd lots and is calculated by CTA and UTP in their datacenters, this synthetic NBBO is derived from the direct prop feeds. Databento provides top-of-book coverage for all equity exchanges (except LTSE, which only provides <0.01% of total market volume). Databento captures all equity data in the NY4 datacenter with PTP timestamps. See our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md) for more information. ## Design We'll use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) and download data for the different equity exchanges. Next, we'll process the data sequentially based on [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv), which is the timestamp when Databento received the data from the exchange. We'll calculate the synthetic NBBO by taking the best bid and offer across these exchanges. After that, we'll plot this over a 100-millisecond window to take a closer look at how the synthetic NBBO reacts when price moves. ## Example ```python import os from collections import defaultdict from dataclasses import dataclass, field from heapq import merge import databento as db import matplotlib.dates as mdates import matplotlib.pyplot as plt import pandas as pd @dataclass(slots=True) class PriceLevel: price: float size: int = 0 count: int = 0 update_time: int = 0 def __str__(self) -> str: return f"{self.size:4} @ {self.price:6.2f} | {self.count:2} order(s)" @staticmethod def _bid_sort(r) -> tuple[float, int, int]: return r.price, r.size, -r.update_time @staticmethod def _ask_sort(r) -> tuple[float, int, int]: return -r.price, r.size, -r.update_time @dataclass(slots=True) class MbpBook: offer: PriceLevel | None = field(default=None) bid: PriceLevel | None = field(default=None) def bbo(self) -> tuple[PriceLevel | None, PriceLevel | None]: return self.bid, self.offer def apply(self, mbp1: db.MBP1Msg) -> None: level = mbp1.levels[0] ts_recv = mbp1.ts_recv if level.bid_px == db.UNDEF_PRICE: self.bid = None else: self.bid = PriceLevel(level.pretty_bid_px, level.bid_sz, level.bid_ct, ts_recv) if level.ask_px == db.UNDEF_PRICE: self.offer = None else: self.offer = PriceLevel(level.pretty_ask_px, level.ask_sz, level.ask_ct, ts_recv) @dataclass(slots=True) class MbpMarket: books: defaultdict[int, defaultdict[int, MbpBook]] = field( default_factory=lambda: defaultdict(lambda: defaultdict(MbpBook)), ) def get_book(self, instrument_id: int, publisher_id: int) -> MbpBook: return self.books[instrument_id][publisher_id] def bbo( self, instrument_id: int, publisher_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: return self.books[instrument_id][publisher_id].bbo() def aggregated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: """Calculate the aggregated BBO across all venues""" agg_bbo: list[PriceLevel | None] = [None] all_bbos = list(zip(*(book.bbo() for book in self.books[instrument_id].values()))) for idx, reducer in ((0, max), (1, min)): all_best = [b for b in all_bbos[idx] if b] if all_best: best_price = reducer(b.price for b in all_best) best = [b for b in all_best if b.price == best_price] agg_bbo[idx] = PriceLevel( price=best_price, size=sum(b.size for b in best), count=sum(b.count for b in best), ) return tuple(agg_bbo) def consolidated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: all_bids, all_offers = zip(*(book.bbo() for book in self.books[instrument_id].values())) best_bid = max((b for b in all_bids if b), key=PriceLevel._bid_sort, default=None) best_offer = max((o for o in all_offers if o), key=PriceLevel._ask_sort, default=None) return best_bid, best_offer def apply(self, msg: db.MBP1Msg) -> None: book = self.books[msg.instrument_id][msg.publisher_id] book.apply(msg) if __name__ == "__main__": equity_datasets = [ "XNAS.ITCH", # Nasdaq "XBOS.ITCH", # Nasdaq Texas "XPSX.ITCH", # Nasdaq PSX "XNYS.PILLAR", # NYSE "ARCX.PILLAR", # NYSE Arca "XASE.PILLAR", # NYSE American "XCHI.PILLAR", # NYSE Texas "XCIS.TRADESBBO", # NYSE National "MEMX.MEMOIR", # Members Exchange "EPRL.DOM", # MIAX Pearl "IEXG.TOPS", # IEX "BATS.PITCH", # Cboe BZX "BATY.PITCH", # Cboe BYX "EDGA.PITCH", # Cboe EDGA "EDGX.PITCH", # Cboe EDGX ] symbol = "NVDA" start = pd.Timestamp(2025, 6, 17, 9, 30, tz="US/Eastern") end = pd.Timestamp(2025, 6, 17, 10, 0, tz="US/Eastern") schema = "mbp-1" client = db.Historical(key="$YOUR_API_KEY") # Get data for all datasets dataset_data_dict: dict[str, db.DBNStore] = {} for dataset in equity_datasets: dataset_name = dataset.replace(".", "-").lower() data_path = ( f"{dataset_name}-{symbol}-{start.date().isoformat().replace('-', '')}.{schema}.dbn.zst" ) if os.path.exists(data_path): data = db.DBNStore.from_file(data_path) else: data = client.timeseries.get_range( dataset=dataset, start=start, end=end, symbols=symbol, schema=schema, path=data_path, ) dataset_data_dict[dataset] = data # Merge all datasets into one stream sorted by ts_recv data = merge(*dataset_data_dict.values(), key=lambda x: x.ts_recv) # Iterate over the records and calculate the consolidated BBO cbbo_list: list[tuple[pd.Timestamp, float, float]] = [] market = MbpMarket() for record in data: market.apply(record) best_bid, best_offer = market.consolidated_bbo(record.instrument_id) cbbo_list.append( ( record.pretty_ts_recv, best_bid.price if best_bid is not None else float("Nan"), best_offer.price if best_offer is not None else float("Nan"), ), ) # Create DataFrame df = pd.DataFrame(cbbo_list, columns=["Timestamp", "Bid", "Offer"]) df = df.set_index("Timestamp") df["is_crossed"] = df["Bid"] >= df["Offer"] # Now we'll plot a small slice of time when the book is crossed start_time = pd.Timestamp(2025, 6, 17, 9, 56, 31, 650000, tz="US/Eastern") end_time = pd.Timestamp(2025, 6, 17, 9, 56, 31, 750000, tz="US/Eastern") df = df.loc[start_time:end_time] fig, ax = plt.subplots(figsize=(11, 6)) # Shade periods where book is not crossed green plt.fill_between( df.index, df["Bid"], df["Offer"], where=~df["is_crossed"], alpha=0.2, linewidth=0, color="green", step="post", label="Normal market", ) # Shade periods where book is crossed red plt.fill_between( df.index, df["Offer"], df["Bid"], where=df["is_crossed"], alpha=0.2, linewidth=0, color="red", step="post", label="Crossed market", ) # Plot BBO lines for col, color in [("Offer", "C1"), ("Bid", "C0")]: plt.hlines( y=df[col][:-1], xmin=df.index[:-1], xmax=df.index[1:], colors=color, label=col, ) plt.gca().xaxis.set_major_formatter(mdates.DateFormatter("%H:%M:%S.%f", tz="US/Eastern")) plt.ylabel("Price (USD)") plt.xlabel("Timestamp (ET)") plt.title(f"{symbol} synthetic NBBO") plt.legend() plt.tight_layout() plt.show() ``` ## Results Notice that the synthetic NBBO may occasionally appear crossed, unlike the regulatory NBBO published by the SIPs. This is expected behavior due to two factors: first, proprietary feeds include odd lot quotations, which are excluded from SIP NBBO calculations; second, the feeds originate from different data centers, resulting in receive-time deltas. These characteristics can temporarily produce crossed markets, but they also enable the construction of a more predictive microprice. ![Consolidated BBO](https://databento.com/docs/consolidated-bbo-0.svg.md) --- Source: https://databento.com/docs/examples/equities/closing-prices?lang=python # Get daily closing prices for equities Official consolidated end-of-day summary data for all US equities is published by the Nasdaq NLS+ feed. This data is normalized into our [Databento US Equities Summary dataset](https://databento.com/docs/datasets/equs-summary.md). ## Overview We'll use the [historical client](https://databento.com/docs/api-historical/client.md) to request daily data for multiple equity symbols. For this example, we'll look at a few popular companies like AMZN, NFLX, and TSLA. We'll calculate the cumulative percent change for these symbols over time, then plot the data to compare their relative performance. ## Example ```python import databento as db import matplotlib.pyplot as plt # Set parameters dataset = "EQUS.SUMMARY" symbols = ["AMZN", "NFLX", "TSLA"] start = "2024-10-01" end = "2025-10-01" client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1d data for the selected symbols df = client.timeseries.get_range( dataset=dataset, symbols=symbols, schema="ohlcv-1d", start=start, end=end, ).to_df() # Calculate cumulative percent change over the request range df["cum_pct_change"] = df.groupby("symbol")["close"].transform( lambda x: (x - x.iloc[0]) / x.iloc[0] * 100, ) print(df) # Plot the relative performance of these symbols df.groupby("symbol")["cum_pct_change"].plot( figsize=(12, 6), legend=True, xlabel="Date", ylabel="Cumulative % Change", title="Relative Performance\nOctober 2024 - October 2025", ) plt.axhline(y=0, color="grey", linestyle="-", zorder=1) plt.grid(True) plt.tight_layout() plt.show() ``` ## Results ```plaintext rtype publisher_id instrument_id open high low close volume symbol cum_pct_change ts_event 2024-10-01 00:00:00+00:00 35 90 853 184.90 186.1900 183.4519 185.13 36044906 AMZN 0.000000 2024-10-01 00:00:00+00:00 35 90 11275 713.64 717.7600 698.5900 706.13 2813482 NFLX 0.000000 2024-10-01 00:00:00+00:00 35 90 16244 262.67 263.9800 248.5300 258.02 87397613 TSLA 0.000000 2024-10-02 00:00:00+00:00 35 90 16244 247.55 251.1585 241.5000 249.02 93983930 TSLA -3.488102 2024-10-02 00:00:00+00:00 35 90 853 184.44 186.6000 184.0400 184.76 23704056 AMZN -0.199860 ... ... ... ... ... ... ... ... ... ... ... 2025-09-29 00:00:00+00:00 35 90 16244 444.35 450.9800 439.5000 443.21 79491510 TSLA 71.773506 2025-09-29 00:00:00+00:00 35 90 853 220.08 222.6000 219.3000 222.17 44259177 AMZN 20.007562 2025-09-30 00:00:00+00:00 35 90 853 222.03 222.2400 217.8900 219.57 48396369 AMZN 18.603144 2025-09-30 00:00:00+00:00 35 90 11275 1206.41 1208.5000 1178.0000 1198.92 3830304 NFLX 69.787433 2025-09-30 00:00:00+00:00 35 90 16244 441.52 445.0000 433.1200 444.72 74357960 TSLA 72.358732 [750 rows x 10 columns] ``` > **See also** > > The prices seen in the EQUS.SUMMARY dataset are unadjusted for any splits or dividends. > This data can be enriched with our [corporate actions and adjustment factors datasets]($PLATFORM_HOSTNAME/docs/portal/live-data). ![Daily prices](https://databento.com/docs/closing-prices-0.svg.md) --- Source: https://databento.com/docs/examples/equities/auction-imbalance?lang=python # Analyze auction imbalance dynamics The closing auction for US Equities is one of the most important events of the trading day. With nearly 10% of average daily volume transacting on the close, the dynamics around price discovery are critical, as these closing prices are used in various use cases, such as benchmark pricing for index funds. While different exchanges share many common behaviors during the closing auction, there are some microstructure differences, such as accepted order types, that can affect the price discovery mechanism. ## Overview We'll use the [historical client](https://databento.com/docs/api-historical/client.md) to request data for the closing auction. The [imbalance schema](https://databento.com/docs/schemas/imbalance.md) provides auction imbalance data such as reference price, paired quantity, and auction status. We'll also use the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md) to look at the price and volume transacting on the limit order book. We'll compare the closing auction process for SPY on NYSE Arca and QQQ on Nasdaq. Due to differences in the data disseminated by the exchange, we'll use different fields for Nasdaq and NYSE. For both venues, the `ref_price` field represents the price at which imbalance shares are being calculated. This price is generally within the current BBO on the limit order book. For Nasdaq, we'll look at the `cont_book_clr_price` field, which represents the price at which auction and continuous orders would transact. For NYSE Arca, we'll look at the `ind_match_price` field instead. This field represents the price at which the highest number of shares would trade, subject to auction collars. ## Example ```python import datetime as dt from zoneinfo import ZoneInfo import databento as db import pandas as pd from matplotlib import dates as mdates from matplotlib import pyplot as plt def get_imbalance_df( client: db.Historical, dataset: str, symbol: str, start: dt.datetime, end: dt.datetime, ) -> pd.DataFrame: """Get imbalance schema data""" imbalance_df = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="imbalance", start=start, end=end, ).to_df(tz=start.tzinfo) return imbalance_df def get_ohlcv_df( client: db.Historical, dataset: str, symbol: str, start: dt.datetime, end: dt.datetime, ) -> pd.DataFrame: """Get OHLCV schema data""" ohlcv_df = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="ohlcv-1s", start=start, end=end, ).to_df(tz=start.tzinfo) return ohlcv_df def plot_closing_auction( venue: str, symbol: str, imbalance_df: pd.DataFrame, ohlcv_df: pd.DataFrame, start_time: dt.time, end_time: dt.time, ) -> None: imbalance_df = imbalance_df.between_time(start_time, end_time) ohlcv_df = ohlcv_df.between_time(start_time, end_time) ohlcv_df["cumulative_volume"] = ohlcv_df["volume"].cumsum() # Create the subplots _, (ax1, ax2) = plt.subplots( 2, 1, figsize=(14, 9), sharex=True, gridspec_kw={"height_ratios": [3, 1]}, ) # First subplot will display prices ax1.plot( imbalance_df.index, imbalance_df["ref_price"], label="Reference Price", drawstyle="steps-pre", ) ax1.plot( ohlcv_df.index, ohlcv_df["close"], label="Last Price", drawstyle="steps-post", ) ax1.plot( imbalance_df.index, imbalance_df["uncrossing_price"], label="Theoretical Uncrossing Price", drawstyle="steps-post", ) ax1.set_ylabel("Price") ax1.grid(True) ax1.legend(loc="best") ax1.set_title(f"{venue} Closing Auction - {symbol}") # Second subplot will display traded volume and auction paired quantity ax2.plot( ohlcv_df.index, ohlcv_df["cumulative_volume"], label="Cumulative Volume", drawstyle="steps-post", ) ax2.plot( imbalance_df.index, imbalance_df["paired_qty"], label="Paired Quantity", drawstyle="steps-post", ) ax2.set_xlabel("Time (ET)") ax2.set_ylabel("Volume") ax2.grid(True) ax2.legend(loc="best") ax2.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M:%S", tz=start_time.tzinfo)) plt.xticks(rotation=45) plt.show() # Set parameters tz = ZoneInfo("America/New_York") start = dt.datetime(2025, 11, 5, 15, 58, 0, tzinfo=tz) end = dt.datetime(2025, 11, 5, 16, 0, 1, tzinfo=tz) client = db.Historical("$YOUR_API_KEY") # First, we'll look at SPY data on NYSE Arca (ARCX.PILLAR) arcx_symbol = "SPY" arcx_dataset = "ARCX.PILLAR" arcx_ohlcv_df = get_ohlcv_df(client, arcx_dataset, arcx_symbol, start, end) arcx_imbalance_df = get_imbalance_df(client, arcx_dataset, arcx_symbol, start, end) arcx_imbalance_df = arcx_imbalance_df.rename(columns={"ind_match_price": "uncrossing_price"}) arcx_imbalance_df = arcx_imbalance_df[["ref_price", "uncrossing_price", "paired_qty", "symbol"]] print(arcx_imbalance_df) plot_closing_auction( "ARCX", arcx_symbol, arcx_imbalance_df, arcx_ohlcv_df, dt.time(15, 58, 0, tzinfo=tz), dt.time(16, 0, 0, tzinfo=tz), ) # Next, we'll look at QQQ data on Nasdaq (XNAS.ITCH) xnas_symbol = "QQQ" xnas_dataset = "XNAS.ITCH" xnas_ohlcv_df = get_ohlcv_df(client, xnas_dataset, xnas_symbol, start, end) xnas_imbalance_df = get_imbalance_df(client, xnas_dataset, xnas_symbol, start, end) xnas_imbalance_df = xnas_imbalance_df.rename(columns={"cont_book_clr_price": "uncrossing_price"}) xnas_imbalance_df = xnas_imbalance_df[["ref_price", "uncrossing_price", "paired_qty", "symbol"]] print(xnas_imbalance_df) plot_closing_auction( "XNAS", xnas_symbol, xnas_imbalance_df, xnas_ohlcv_df, dt.time(15, 58, 0, tzinfo=tz), dt.time(16, 0, 0, tzinfo=tz), ) ``` ## Results First, we'll take a look at the closing auction for SPY on NYSE Arca. > **See also** > > See the [NYSE Arca auction documentation](https://www.nyse.com/markets/nyse-arca/trading-info#auctions) for more information. ```plaintext ref_price uncrossing_price paired_qty symbol ts_recv 2025-11-05 15:58:00.061423353-05:00 677.715 684.49 951885 SPY 2025-11-05 15:58:01.055768239-05:00 677.760 676.22 1193528 SPY 2025-11-05 15:58:02.054678442-05:00 677.760 676.50 1194940 SPY 2025-11-05 15:58:03.056152198-05:00 677.760 676.50 1195106 SPY 2025-11-05 15:58:04.054268311-05:00 677.720 676.18 1200714 SPY ... ... ... ... ... 2025-11-05 15:59:55.052843213-05:00 677.670 677.81 1401805 SPY 2025-11-05 15:59:56.050469422-05:00 677.600 677.76 1402565 SPY 2025-11-05 15:59:57.051653917-05:00 677.590 677.72 1402565 SPY 2025-11-05 15:59:58.052726733-05:00 677.530 677.68 1403065 SPY 2025-11-05 15:59:59.050924091-05:00 677.470 677.60 1403218 SPY ``` ![NYSE Arca Auction imbalance](https://databento.com/docs/auction-imbalance-0.svg.md) Next, we'll look at the closing auction for QQQ on Nasdaq. You'll notice that in both examples, these theoretical uncrossing prices will converge towards the last traded price as we approach the close. > **See also** > > See the [Nasdaq Closing Cross documentation](https://www.nasdaqtrader.com/content/productsservices/Trading/ClosingCrossfaq.pdf) for more information. ```plaintext ref_price uncrossing_price paired_qty symbol ts_recv 2025-11-05 15:58:00.060035531-05:00 623.49 621.42 677169 QQQ 2025-11-05 15:58:01.019052477-05:00 623.52 622.10 677696 QQQ 2025-11-05 15:58:02.055023604-05:00 623.52 622.06 678796 QQQ 2025-11-05 15:58:03.010276615-05:00 623.54 622.20 678836 QQQ 2025-11-05 15:58:04.040411490-05:00 623.47 622.00 699336 QQQ ... ... ... ... ... 2025-11-05 15:59:55.008542818-05:00 623.43 623.43 827157 QQQ 2025-11-05 15:59:56.099171134-05:00 623.37 623.37 826774 QQQ 2025-11-05 15:59:57.059011900-05:00 623.36 623.36 826774 QQQ 2025-11-05 15:59:58.025345576-05:00 623.27 623.27 826393 QQQ 2025-11-05 15:59:59.032786745-05:00 623.31 623.31 826393 QQQ ``` ![Nasdaq Auction imbalance](https://databento.com/docs/auction-imbalance-1.svg.md) --- Source: https://databento.com/docs/examples/futures?lang=python ## Futures | Title | Subtitle | Image | Supported languages | | [Futures: Introduction](https://databento.com/docs/examples/futures/futures-introduction.md) | Get started with futures and symbology. Volumes, tick size, expirations, and more. | ![Futures introduction image](https://databento.com/docs/futures-introduction.svg.md) | Python, Futures, Historical | | [Volume, open interest, and settlement prices](https://databento.com/docs/examples/futures/retrieving-oi-and-settlement-prices.md) | Retrieve volume, open interest, and settlement prices from the statistics schema. | ![Futures statistics](https://databento.com/docs/retrieving-oi-and-settlement-prices-thumbnail.svg.md) | Python, Futures, Historical | | [Futures trading hours](https://databento.com/docs/examples/futures/trading-hours.md) | Get trading hours for a futures product. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Futures, Historical | | [Get options chain for a futures product](https://databento.com/docs/examples/futures/futures-product-options.md) | Get daily, weekly, and monthly options for a futures product. | ![Image placeholder](https://databento.com/docs/futures-product-options-thumbnail.svg.md) | Python, Futures, Historical | | [Determine publishing times for statistics](https://databento.com/docs/examples/futures/statistics-schedule.md) | Discover when different types of statistics are published for an instrument. | ![Image placeholder](https://databento.com/docs/statistics-schedule-thumbnail.svg.md) | Python, Futures, Historical | --- Source: https://databento.com/docs/examples/futures/futures-introduction?lang=python # Futures: Introduction > **Info** > > This article introduces additional concepts to get started with futures data on > Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll show how to: - Find a futures dataset - Find the 10 futures contracts with the highest volume - Use instrument definitions to get the tick size, expiration, and matching algorithm of an instrument - Stream live BBO data - Use parent symbology to fetch all contracts expirations - Use continuous contract symbology to get the lead month contract We'll also highlight any special conventions of futures datasets on Databento that differ from those of other asset classes. ## Finding a futures dataset To use futures data on Databento, first identify the dataset that you want from our [data catalog]($PLATFORM_HOSTNAME) and go to its detail page. Here, you can find its dataset ID at the top left of the page. ![Dataset ID](https://databento.com/docs/futures-dataset-id.png.md) For this example, we'll use the [CME Globex MDP 3.0]($PLATFORM_HOSTNAME/datasets/GLBX.MDP3) dataset, whose dataset ID is `GLBX.MDP3`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## Finding futures contracts with highest volume A quick way to find the most actively-traded futures contracts, across all expirations, is to fetch the daily volumes from the OHLCV-1d schema. (You can also get a similar result using the `statistics` schema, which provides the official daily settlement prices and trade volumes.) ```python import databento as db # Create historical client client = db.Historical("$YOUR_API_KEY") def rank_by_volume(top=10): # Request OHLCV-1d data data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols="ALL_SYMBOLS", schema="ohlcv-1d", start="2023-08-15" ) # Convert to DataFrame and filter for top 10 instruments by volume df = data.to_df() return df.sort_values(by="volume", ascending=False)["instrument_id"].to_list()[:top] top_instruments = rank_by_volume() print(top_instruments) ``` This returns the following list of numeric instrument IDs, from the `instrument_id` field. ```json [338574, 3445, 404144, 9235, 2922, 2130, 72156, 399495, 225833, 1562] ``` ## Using instrument definitions to get tick size, expiration, and matching algorithm In the case of CME Globex MDP 3.0, these instrument IDs are sourced from tag 48-SecurityID of the original [Security Definition messages](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Security+Definition+-+Futures). Instrument IDs are necessary for many order routing and post-trade scenarios with the exchange, but can be hard to work with, so we will print their raw symbols instead. Let's extract useful properties of these instruments, like their tick sizes, expiration dates, and matching algorithms. ```python def get_symbol_properties(instrument_id_list): # Request definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", stype_in="instrument_id", symbols=instrument_id_list, schema="definition", start="2023-08-15", ) # Convert to DataFrame df = data.to_df() return df[["instrument_id", "raw_symbol", "min_price_increment", "match_algorithm", "expiration"]] print(get_symbol_properties(top_instruments)) ``` ```json instrument_id raw_symbol min_price_increment match_algorithm expiration ts_recv 2023-08-15 00:00:00+00:00 225833 ZBU3 0.031250 F 2023-09-20 17:01:00+00:00 2023-08-15 00:00:00+00:00 1562 SR3Z3 0.005000 A 2024-03-19 21:00:00+00:00 2023-08-15 00:00:00+00:00 2922 MESU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 399495 TNU3 0.015625 F 2023-09-20 17:01:00+00:00 2023-08-15 00:00:00+00:00 338574 ZNU3 0.015625 F 2023-09-20 17:01:00+00:00 2023-08-15 00:00:00+00:00 3445 ESU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 72156 ZTU3 0.003906 K 2023-09-29 17:01:00+00:00 2023-08-15 00:00:00+00:00 9235 MNQU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 2130 NQU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 404144 ZFU3 0.007812 F 2023-09-29 17:01:00+00:00 ``` Observe that the `matching_algorithm` values are in their raw values passed through from the exchange. The full list of supported instrument definition fields and info about the output can be found on the exchange's [specifications page](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Security+Definition+-+Futures). ## Streaming live BBO data While highly liquid futures contracts generally maintain narrow bid-ask spreads, certain market conditions can lead to this spread widening. Stream our [BBO-1s schema](https://databento.com/docs/schemas/bbo.md) to monitor the current best bid and best offer, subsampled at 1-second intervals. This example uses the BBO-1s schema for top-of-book information, but other similar schemas exist that may better suit your use case. Read more about these schemas on our [MBP-1 vs. TBBO vs. BBO schemas page](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). > **Info** > > This example requires a live license to `GLBX.MDP3`. Visit our [live data portal]($PLATFORM_HOSTNAME/portal/live-data#manage) > to sign up. ```python import databento as db # Enable basic logging db.enable_logging("INFO") # Create a live client live_client = db.Live("$YOUR_API_KEY") # Subscribe to the BBO-1s schema for the continuous NQ contract live_client.subscribe( dataset="GLBX.MDP3", schema="bbo-1s", symbols="NQ.v.0", stype_in="continuous", ) # Add a print callback live_client.add_callback(print) # Start the live client to begin streaming live_client.start() # Run the stream for 15 seconds before closing live_client.block_for_close(timeout=15) ``` > **Info** > > If you do not see any output, it could be because the markets are closed. > See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) to utilize [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=42002475, ts_event=1769047300419171930, pretty_ts_event='2026-01-22T02:01:40.419171930Z', stype_in=None, stype_in_symbol='NQ.v.0', stype_out=None, stype_out_symbol='NQH6', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') BboMsg(ts_recv=1768960877000000000, pretty_ts_recv='2026-01-21T02:01:17.000000000Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42002475, ts_event=1768960876926278195, pretty_ts_event='2026-01-21T02:01:16.926278195Z', side='B', price=25201000000000, pretty_price=25201, size=1, flags=LAST (128), sequence=151272098, bid_px_00=25200750000000, pretty_bid_px_00=25200.75, ask_px_00=25201500000000, pretty_ask_px_00=25201.5, bid_sz_00=2, ask_sz_00=1, bid_ct_00=2, ask_ct_00=1) SystemMsg(rtype=, publisher_id=0, instrument_id=0, ts_event=1768960877000000000, pretty_ts_event='2026-01-21T02:01:17.000000000Z', msg='End of interval for bbo-1s', code=) ... BboMsg(ts_recv=1768960880000000000, pretty_ts_recv='2026-01-21T02:01:20.000000000Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42002475, ts_event=1768960879783497307, pretty_ts_event='2026-01-21T02:01:19.783497307Z', side='B', price=25203250000000, pretty_price=25203.25, size=1, flags=LAST (128), sequence=151273402, bid_px_00=25202750000000, pretty_bid_px_00=25202.75, ask_px_00=25203750000000, pretty_ask_px_00=25203.75, bid_sz_00=1, ask_sz_00=2, bid_ct_00=1, ask_ct_00=2) SystemMsg(rtype=, publisher_id=0, instrument_id=0, ts_event=1768960880000000000, pretty_ts_event='2026-01-21T02:01:20.000000000Z', msg='End of interval for bbo-1s', code=) BboMsg(ts_recv=1768960881000000000, pretty_ts_recv='2026-01-21T02:01:21.000000000Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42002475, ts_event=1768960879783497307, pretty_ts_event='2026-01-21T02:01:19.783497307Z', side='B', price=25203250000000, pretty_price=25203.25, size=1, flags=LAST (128), sequence=151273771, bid_px_00=25202250000000, pretty_bid_px_00=25202.25, ask_px_00=25203000000000, pretty_ask_px_00=25203, bid_sz_00=2, ask_sz_00=1, bid_ct_00=2, ask_ct_00=1) ``` ## Parent symbology On futures trading venues, it can be tedious to fetch every child instrument and contract expiration (like ESU3, ESZ3, ESU3-ESZ3) for a given parent instrument (like ES). You can use our [parent symbology type](https://databento.com/docs/conventions/symbology.md#parent) to do this, by passing in `stype_in="parent"`. ```python def get_child_instruments(parents=["ZB.FUT", "SR3.FUT"]): # Request definition data for parent symbols data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=parents, stype_in="parent", schema="definition", start="2023-08-15", ) # Convert to DataFrame df = data.to_df() return df[["instrument_id", "raw_symbol"]] print(get_child_instruments().head()) ``` ```json instrument_id raw_symbol ts_recv 2023-08-15 00:00:00+00:00 34810 SR3:AB 01Y M8 2023-08-15 00:00:00+00:00 45040 SR3M6-SR3Z7 2023-08-15 00:00:00+00:00 51186 SR3Z4-SR3M6 2023-08-15 00:00:00+00:00 22508 SR3:DF H7Z7U8M9 2023-08-15 00:00:00+00:00 350151 SR3:SB PK M4-M5 ``` Alternatively, you can replicate this logic by requesting definition data for all symbols and filtering on the `asset` field. The `asset` field is populated with the root of the parent symbol. Databento's web portal only exposes parent products, and not child instruments. When you set up a batch download through the web portal, note that you're making a parent symbology request and you'll receive interleaved data from multiple instruments. If you want to set up a batch download of individual child instruments, you must use our API instead. ## Continuous contract symbology Likewise, it's tedious to track the lead month contract of a futures product over a long period of time, due to rollovers. You can use our [continuous contract symbology type](https://databento.com/docs/conventions/symbology.md#continuous) to get a single symbol that is pegged to the lead month contract, by passing in `stype_in="continuous"`. For example, let's plot the two lead month ES contracts `ES.n.0` and `ES.n.1`. ```python import databento as db import matplotlib.pyplot as plt client = db.Historical("$YOUR_API_KEY") dataset = "GLBX.MDP3" symbols = ["ES.n.0", "ES.n.1"] start = "2024" data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="ohlcv-1d", stype_in="continuous", symbols=symbols, start=start, ) df = data.to_df() df.groupby("symbol")["close"].plot( xlabel="Date", ylabel="Price", ) plt.legend() plt.show() ``` ![ES plot](https://databento.com/docs/futures-continuous-0.svg.md) > **Tip** > > If you'd like to see the two calendar front month contracts instead, you may use `ES.c.0` and > `ES.c.1`. However, in most cases these will resolve to the same symbols as `ES.n.0` and `ES.n.1` > respectively because open interest tends to decay with increasing time to expiration. Many futures products reflect seasonality in commodities or term structure in fixed income, so the nearest calendar month may not be the lead month. In such cases, you need to specify how you want to resolve the lead month, also known as a **roll rule**. For example, you could use `ES.v.0` to resolve the lead month by volume instead of open interest. ## Special conventions for futures on Databento - **Weekly trading session.** Unlike many venues, CME Globex has a weekly trading session. This affects how you process MBO data. We provide a synthetic snapshot of the book at the start of each UTC day to make it easier to start from any day of the week. - **User-defined instruments and spreads.** CME Globex has a large number of [user-defined instruments](https://www.cmegroup.com/confluence/display/EPICSANDBOX/User+Defined+Spread+-+UDS). While many vendors do not expose these and their raw symbols may be foreign to a user who's seeing these for the first time, Databento includes all of them as many are highly liquid and active. - **Asynchronous trade publication.** CME Globex prints fills and order deletions associated with the fills asynchronously, with the fills published before the deletions. This is unlike most venues, which treat the individual fill and corresponding order deletion as a single atomic event. You may choose to preemptively update your book based on trades or fills, or wait until the corresponding deletes, which we represent with action `C`. - **Implied book.** CME Globex has implied matching. If a trade is partially filled by contra liquidity on the implied book, we show the full quantity of the trade but only the fill quantities on the direct book. A full implied trade will have trade side `N`. - **Inverted spreads.** CME Globex's matching engine has various price limits and circuit breakers. The spread may be inverted during a trading halt. - **No rollover back-adjustments.** Our continuous contract symbology is a notation that maps to an actual, tradable instrument on any given date. The continuous contract prices provided are the original, unadjusted prices. We don't create a synthetic time series by back-adjusting the prices to remove jumps during rollovers. > **See also** > > [Options on futures introduction](https://databento.com/docs/examples/options/options-on-futures-introduction.md) and [continuous contracts](https://databento.com/docs/conventions/symbology.md#continuous). --- Source: https://databento.com/docs/examples/futures/retrieving-oi-and-settlement-prices?lang=python # Volume, open interest, and settlement prices The [statistics schema](https://databento.com/docs/schemas/statistics.md) provides official summary statistics of each instrument that's published by the venue. Each statistics record contains the value or values associated with only a single statistics type, indicated by the `stat_type` field. This makes the schema flexible and forwards-compatible, but can make it harder to work with. ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve daily statistics for a [parent symbol](https://databento.com/docs/conventions/symbology.md#parent). We'll download statistics data for a parent symbol and use the `StatType` enum to filter for cleared volume, open interest, and settlement price statistics. Next, we'll filter for statistics on outright futures only, using data found in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). Then, we'll group statistics by symbol and date to create a table for each date and symbol. In the end, we'll generate a plot which shows open interest and cleared volume over time. ## ts_ref Many statistics are published daily or refer to a specific trading session. When provided by the publisher, the trading session associated with the statistic is indicated with the `ts_ref` field. Depending on the dataset, statistics can be published at various times, so you may need to look at the next UTC date for a given statistic. For statistics that are published multiple times, you should always refer to the most recently published value. > **See also** > > See our [statistics schedule example](https://databento.com/docs/examples/futures/statistics-schedule.md) which shows how you can determine when the various statistics types are published. ## Example ```python import databento as db import matplotlib.pyplot as plt # Set parameters dataset = "GLBX.MDP3" product = "ZC" start = "2025-05-04" end = "2025-05-19" # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Request definition data and convert to DataFrame def_df = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=f"{product}.FUT", stype_in="parent", start=start, end=end, ).to_df() # Filter for outright futures only and keep the last definition def_df = def_df[def_df["instrument_class"] == db.InstrumentClass.FUTURE] def_df = def_df[["instrument_id", "raw_symbol", "expiration"]] def_df = def_df.drop_duplicates(subset=["instrument_id"], keep="last") # Request statistics data and convert to DataFrame stats_df = client.timeseries.get_range( dataset=dataset, schema="statistics", symbols=f"{product}.FUT", stype_in="parent", start=start, end=end, ).to_df() # Filter for daily statistics on outright futures stats_df = stats_df.merge(def_df, on="instrument_id") stats_df = stats_df.rename(columns={"raw_symbol": "Symbol"}) # Only keep the date component of `ts_ref` stats_df["Trade date"] = stats_df["ts_ref"].dt.date # Create columns for each statistic stats_df["Settlement price"] = stats_df[stats_df["stat_type"] == db.StatType.SETTLEMENT_PRICE][ "price" ] stats_df["Cleared volume"] = stats_df[stats_df["stat_type"] == db.StatType.CLEARED_VOLUME][ "quantity" ] stats_df["Open interest"] = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST]["quantity"] # Groupby and keep the last published record for each statistic stats_df = ( stats_df.groupby(["Trade date", "Symbol"]).agg("last").sort_values(["Trade date", "expiration"]) ) stats_df["Cleared volume"] = stats_df["Cleared volume"].fillna(0).astype(int) stats_df["Open interest"] = stats_df["Open interest"].fillna(0).astype(int) stats_df = stats_df[["Settlement price", "Cleared volume", "Open interest"]] print(stats_df) # Now, we'll plot aggregated volume and open interest statistics for this product date_df = stats_df[["Cleared volume", "Open interest"]].groupby("Trade date").agg("sum") ax_vol = date_df["Cleared volume"].plot.bar(ylabel="Cleared volume") ax_oi = ( date_df["Open interest"] .reset_index() .plot.line( ax=ax_vol, color="C1", marker="o", ylabel="Open interest", secondary_y=True, mark_right=False, legend=False, ) ) plt.gcf().legend(bbox_to_anchor=(0, 1), loc="upper left", bbox_transform=ax_vol.transAxes) plt.setp([ax_vol.get_xticklabels(), ax_oi.get_xticklabels()], rotation=45) plt.title(f"{product} cleared volume and open interest") plt.tight_layout() plt.show() ``` ## Result ```plaintext Settlement price Cleared volume Open interest Trade date Symbol 2025-05-02 ZCK5 461.25 2624 1966 ZCN5 469.00 163656 665836 ZCU5 440.00 60030 314718 ZCZ5 450.25 93701 432738 ZCH6 464.50 10615 50264 ... ... ... ... 2025-05-16 ZCN7 475.25 12 139 ZCU7 453.75 5 12 ZCZ7 455.50 17 630 ZCN8 474.50 0 7 ZCZ8 458.00 0 18 [174 rows x 3 columns] ``` The table is similar to the one found on the [CME website](https://www.cmegroup.com/markets/agriculture/grains/corn.volume.html) that shows these statistics for every active contract. The plot shows the aggregated volume and open interest for all active contracts for a product. ![Statistics plot](https://databento.com/docs/retrieving-oi-and-settlement-prices-0.svg.md) --- Source: https://databento.com/docs/examples/futures/trading-hours?lang=python # Futures trading hours Tracking exchange trading hours is an essential task for all market participants. Venues can modify their regular trading hours over time, so having an up-to-date record of these schedules can aid in organizing historical data. Since trading hours are typically modified during holidays, it's important to be aware of these altered schedules. ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical.md) to get trading hours for ES futures. We'll look at two holidays to show how trading hours are modified. We'll use the [status schema](https://databento.com/docs/schemas/status.md) for this example. This schema provides updates about the trading session. These updates can be scheduled changes, or they can be unscheduled, such as volatility halts or other market integrity controls. We'll filter on the `reason` field to look for scheduled changes. We'll also look at the `is_trading` column, since not all changes in market state result in a change to trading functionality. ## Example ```python import datetime as dt import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2024-12-22" end = "2025-01-05" # Get status data status_data = client.timeseries.get_range( dataset=dataset, schema="status", symbols=f"{product}.v.0", stype_in="continuous", start=start, end=end, ) # Convert to DataFrame status_df = status_data.to_df() # Filter out midnight snapshots status_df = status_df[status_df.index.time != dt.time.min] # Keep scheduled changes resulting in a change in trading state status_df = status_df[status_df["reason"] == db.StatusReason.SCHEDULED] status_df = status_df[status_df["trading_event"] == db.TradingEvent.NONE] status_df = status_df[status_df["is_trading"] != status_df["is_trading"].shift()] # Clean up for readability status_df = status_df.set_index("ts_event") status_df = status_df[["symbol", "is_trading"]] print(status_df) ``` CME Globex can have different trading hours for different products. For ES futures, there is an early close on Christmas Eve (2024-12-24), as well as a full close on Christmas (2024-12-25) and New Year's Day (2025-01-01). ```plaintext symbol is_trading ts_event 2024-12-22 23:00:00+00:00 ES.v.0 Y 2024-12-23 22:00:00+00:00 ES.v.0 N 2024-12-23 23:00:00+00:00 ES.v.0 Y 2024-12-24 18:15:00+00:00 ES.v.0 N 2024-12-25 23:00:00+00:00 ES.v.0 Y 2024-12-26 22:00:00+00:00 ES.v.0 N 2024-12-26 23:00:00+00:00 ES.v.0 Y 2024-12-27 22:00:00+00:00 ES.v.0 N 2024-12-29 23:00:00+00:00 ES.v.0 Y 2024-12-30 22:00:00+00:00 ES.v.0 N 2024-12-30 23:00:00+00:00 ES.v.0 Y 2024-12-31 22:00:00+00:00 ES.v.0 N 2025-01-01 23:00:00+00:00 ES.v.0 Y 2025-01-02 22:00:00+00:00 ES.v.0 N 2025-01-02 23:00:00+00:00 ES.v.0 Y 2025-01-03 22:00:00+00:00 ES.v.0 N ``` --- Source: https://databento.com/docs/examples/futures/futures-product-options?lang=python # Get options chain for a futures product Identifying all options for a futures product can be difficult because exchange naming conventions may split the options chain across different [root symbols](https://databento.com/docs/conventions/symbology.md#parent). On CME, this can occur in two ways: - Futures and options may use entirely different roots, e.g., `CL.FUT` and `LO.OPT` for crude oil. - Certain expirations may appear under a different root symbol than their underlying. `E1AQ5 C5125` is a weekly option on `ESU5` but has the root `E1A`, so it is included in `E1A.OPT` instead of `ES.OPT`. > **See also** > > CME's Weekly Options on Futures page provides more examples of the second scenario above, where weekly contracts are denoted by different root symbols than their underlying futures. ## Overview This example demonstrates how to use the [historical API](https://databento.com/docs/api-historical/client.md) to find all options linked to a futures product, even when they're listed under different root symbols. We'll request data using the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) for all symbols, where the `asset` field represents the product's root symbol and the `instrument_class` field identifies its outright futures. We'll then retrieve every option whose `underlying` matches any of those futures for the complete options chain. We'll also use the `expiration` field to demonstrate how to filter these options by their Days to Expiration (DTE). ## Example ```python import databento as db import matplotlib.pyplot as plt import pandas as pd from matplotlib.patches import Patch def get_futures( product: str, def_df: pd.DataFrame, ) -> pd.DataFrame: """ Get definitions for all outright futures for a product. """ outright_futures = def_df[ (def_df["instrument_class"] == db.InstrumentClass.FUTURE) & (def_df["asset"] == product) ] return outright_futures def get_options( product: str, def_df: pd.DataFrame, expiration_filter: pd.Interval | None = None, ) -> pd.DataFrame: """ Get definitions for all options for a product. Optionally filter by expiration. """ # Get the symbols for outright futures for the product outright_futures = get_futures(product, def_df)["raw_symbol"].to_list() # Get all options that have any of the outright futures as an underlying options_df = def_df[ (def_df["underlying"].isin(outright_futures)) & (def_df["instrument_class"].isin(["C", "P"])) ].copy() # Create a `days_to_expiration` field from `ts_recv` and `expiration` options_df["days_to_expiration"] = ( options_df["expiration"].dt.normalize() - options_df.index.normalize() ).dt.days if expiration_filter is None: return options_df # Filter for specified expiration range filtered_options = options_df[ options_df["days_to_expiration"].between( expiration_filter.left, expiration_filter.right, inclusive="both", ) ] return filtered_options def plot_volume_by_dte( product: str, date: str, def_df: pd.DataFrame, ohlcv_df: pd.DataFrame, ) -> None: dte_ranges = [ (0, 0, "0"), (1, 2, "1-2"), (3, 4, "3-4"), (5, 9, "5-9"), (10, 1000, "10+"), ] # Create pie chart fig, ax = plt.subplots(figsize=(10, 6)) dte_handles = [] dte_volumes = [] for idx, (start, end, label) in enumerate(dte_ranges): defs = get_options(product, def_df, pd.Interval(start, end, closed="both")) vol = ohlcv_df[ohlcv_df["instrument_id"].isin(defs["instrument_id"])]["volume"].sum() patch = Patch(facecolor=f"C{idx}", label=f"{label} DTE") dte_handles.append(patch) dte_volumes.append(vol) ax.pie(dte_volumes, autopct="%1.1f%%", startangle=90, pctdistance=1.2) ax.set_title(f"{product} options volume on {date} by Days To Expiration (DTE)") fig.legend(handles=dte_handles, loc="upper left", bbox_to_anchor=(0.1, 0.9)) plt.show() # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2025-08-04" # Create a historical client client = db.Historical("$YOUR_API_KEY") # Download ohlcv-1d data for ALL_SYMBOLS for one day ohlcv_df = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols="ALL_SYMBOLS", start=start, ).to_df() # Download definition data for ALL_SYMBOLS for one day def_df = client.timeseries.get_range( dataset=dataset, schema="definition", symbols="ALL_SYMBOLS", start=start, ).to_df() # Create a range of DTE to filter for # You can filter for 1-3 DTE with pd.Interval(1, 3, closed="both") zero_dte_options = get_options(product, def_df, pd.Interval(0, 0, closed="both")) print(zero_dte_options[["raw_symbol", "expiration", "strike_price", "days_to_expiration"]]) # Now, plot of total volume by different DTE ranges plot_volume_by_dte(product, start, def_df, ohlcv_df) ``` ## Result ```plaintext raw_symbol expiration strike_price days_to_expiration ts_recv 2025-08-04 00:00:00+00:00 E1AQ5 C5125 2025-08-04 20:00:00+00:00 5125.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P6600 2025-08-04 20:00:00+00:00 6600.0 0 2025-08-04 00:00:00+00:00 E1AQ5 C4100 2025-08-04 20:00:00+00:00 4100.0 0 2025-08-04 00:00:00+00:00 E1AQ5 C7200 2025-08-04 20:00:00+00:00 7200.0 0 2025-08-04 00:00:00+00:00 E1AQ5 C4700 2025-08-04 20:00:00+00:00 4700.0 0 ... ... ... ... ... 2025-08-04 00:00:00+00:00 E1AQ5 P4400 2025-08-04 20:00:00+00:00 4400.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P7600 2025-08-04 20:00:00+00:00 7600.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P5735 2025-08-04 20:00:00+00:00 5735.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P5715 2025-08-04 20:00:00+00:00 5715.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P5075 2025-08-04 20:00:00+00:00 5075.0 0 [626 rows x 4 columns] ``` You could use the `raw_symbol` values from this filtered list and pass them into another data request. This is demonstrated below which uses the [OHLCV-1d](https://databento.com/docs/schemas/ohlcv.md) schema to calculate the total volume for all options based on Days to Expiration (DTE). ![DTE volume plot](https://databento.com/docs/futures-product-options-0.svg.md) --- Source: https://databento.com/docs/examples/futures/statistics-schedule?lang=python # Determine publishing times for statistics Official exchange-published statistics for an instrument can be published at various times. There are statistics that are published continuously throughout the day, while other statistics are only published a few times per day. Additionally, this publishing behavior may be different between venues. ## Overview We'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve all statistics for an instrument to determine when statistics are published. We'll request data from the [statistics schema](https://databento.com/docs/schemas/statistics.md), which contains all official statistics published by the venue. We'll also request data from the [status schema](https://databento.com/docs/schemas/status.md). This status schema data will show how changes in market state can affect publishing times for various statistics. We'll plot this data to show when each statistic is published. ## Example ```python import datetime as dt import databento as db import matplotlib.pyplot as plt import pandas as pd from matplotlib.dates import DateFormatter, MinuteLocator # Create mapping of statistic type to human-readable name stat_type_dict = {x.value: x.name.replace("_", " ").capitalize() for x in db.StatType.variants()} def plot_statistics_distribution( symbol: str, stats_data: pd.DataFrame, status_data: pd.DataFrame, ): """Plot publishing time distribution of each statistic type""" stats_data = ( stats_data.groupby("stat_type") .resample("15min", include_groups=False) .size() .unstack(level=0) .fillna(0) ).transpose() stats_data = stats_data.rename(index=stat_type_dict) _, ax = plt.subplots(figsize=(16, 8)) times = pd.to_datetime(stats_data.columns) time_width = pd.Timedelta(minutes=15) # Plot horizontal bars for each statistic type for idx, stat_type in enumerate(stats_data.index): mask = stats_data.loc[stat_type] > 0 if mask.any(): ax.barh( idx, width=time_width * 1.1, left=times[mask] - time_width / 2, label=stat_type, ) # Plot vertical lines for changes in market state for idx, row in status_data.iterrows(): ax.axvline( x=idx, color="dimgray", linestyle=":", linewidth=1.5, ) ax.text( x=idx, y=ax.get_ylim()[1], s=str(row["action"]), ha="left", va="top", rotation=-90, color="dimgray", ) # Format axes ax.set( yticks=range(len(stats_data)), yticklabels=stats_data.index, xlabel="Time (UTC)", ylabel="Stat Type", title=f"{symbol} Statistics Message Distribution", ) ax.xaxis.set_major_formatter(DateFormatter("%H:%M")) ax.xaxis.set_major_locator(MinuteLocator([0, 60])) ax.set_axisbelow(True) ax.grid() plt.setp(ax.xaxis.get_majorticklabels(), rotation=45, ha="right") plt.tight_layout() plt.show() # Set parameters dataset = "IFUS.IMPACT" product = "CC" start = "2025-09-17" client = db.Historical("$YOUR_API_KEY") # Request statistics data for the continuous contract stats_df = client.timeseries.get_range( dataset=dataset, symbols=f"{product}.v.0", stype_in="continuous", schema="statistics", start=start, ).to_df() # Request status data for the continuous contract status_df = client.timeseries.get_range( dataset=dataset, symbols=f"{product}.v.0", stype_in="continuous", schema="status", start=start, ).to_df() # Filter out midnight snapshot status_df = status_df[status_df.index.time != dt.time.min] # Only keep certain market states # For a full list of variants, see https://databento.com/docs/schemas-and-data-formats/status#status-action-variants status_map = {1: "Pre-Open", 7: "Open", 12: "Close"} status_df = status_df[status_df["action"].isin(status_map)] status_df = status_df[status_df["trading_event"] != 1] status_df["action"] = status_df["action"].map(status_map) # Print out the number of occurrences for each statistic type print(stats_df["stat_type"].map(stat_type_dict).value_counts()) # Plot distribution of statistic publishing time plot_statistics_distribution( symbol=product, stats_data=stats_df, status_data=status_df, ) ``` ## Result Taking a look at this example for Cocoa futures on ICE Futures US, you will notice how certain statistics, such as cleared volume and VWAP, are published continuously throughout the day. Other statistics, such as open interest and settlement price, are published only a few times per day. ```plaintext stat_type Trading session high price 4482 Trading session low price 4482 Vwap 4482 Cleared volume 4482 Indicative opening price 3060 Opening price 3 Open interest 3 Settlement price 3 Name: count, dtype: int64 ``` ![Statistics publishing times](https://databento.com/docs/statistics-schedule-0.svg.md) --- Source: https://databento.com/docs/examples/indices?lang=python ## Indices | Title | Subtitle | Image | Supported languages | | [Indices: Introduction](https://databento.com/docs/examples/indices/indices-introduction.md) | Get started with index data from the Databento Core Indices dataset. | ![Index interval distribution](https://databento.com/docs/indices-introduction-thumbnail.svg.md) | Python, Indices, Historical | | [Implied SPX index from ES futures curve](https://databento.com/docs/examples/indices/implied-spx-index.md) | Estimate an S&P 500 spot proxy from the front two ES contracts. | ![ES-implied SPX proxy](https://databento.com/docs/implied-spx-index-thumbnail.svg.md) | Python, Indices, Futures, Historical | | [Filter options by moneyness](https://databento.com/docs/examples/indices/option-moneyness.md) | Calculate moneyness across the SPXW chain and set a volatility-adjusted strike range. | ![ATM Strike Activity](https://databento.com/docs/option-moneyness-thumb-thumbnail.svg.md) | Python, Indices, Options, Historical | | [Filter SPXW strikes by VIX-implied expected move](https://databento.com/docs/examples/indices/spx-and-the-vix.md) | Interpolate the VIX to set a volatility-adjusted strike range for SPXW expirations. | ![Strikes filtered by volatility](https://databento.com/docs/spx-and-the-vix-thumb-thumbnail.svg.md) | Python, Indices, Options, Historical | --- Source: https://databento.com/docs/examples/indices/indices-introduction?lang=python # Indices: Introduction > **Info** > > This article introduces additional concepts to get started with indices data on > Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll introduce: - The Databento Core Indices dataset - How to discover index update frequencies - How to plot index prices - Special conventions for indices vs. other asset classes offered by Databento ## Using the indices dataset Our [Databento Core Indices feed]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE) includes, but is not limited to: - Flagship benchmarks (e.g. SPX and VIX) - Strategy indices (e.g. PUTR and BXR) - ETF fair values (e.g. TQQQIV and SSOIV) - Crypto values (e.g. BTC1RP and ETH100RP) - US benchmarks (e.g. RUT and RUI) - US market and style indices (e.g. MSTAR and MWMF) - Global equity indices (e.g. MXEA and MXWLD) To use index data on Databento, navigate to our [indices catalog page]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE) and find its dataset ID at the top left of the page. ![Find dataset id](https://databento.com/docs/core-indices-dataset-id.png.md) For this example, we'll use the [Databento Core Indices]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE) (`DBIX.CORE`) dataset. This ID will be passed into the `dataset` parameter of any API or client method. ## Update frequency Unlike market data for order-driven instruments, which updates in response to individual trades or order book updates, index values are calculated and published according to an index-specific schedule. The publication frequency varies by index, ranging from intraday updates every few seconds to values published only once per day. Since these records are not sent out at perfect intervals, there is some variance in the send times. We can estimate the observed update cadence for any index using Databento's [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md), which provides every single update to the index price at the moment it was released. For instance, to find the update frequency of the Morningstar US Target Market Exposure Index (MUFT): ```python import databento as db client = db.Historical(key="$YOUR_API_KEY") muft = client.timeseries.get_range( dataset="DBIX.CORE", symbols=["MUFT"], start="2026-02-18", schema="mbp-1" ).to_df() intervals = muft["ts_event"].diff().dt.total_seconds() print(intervals.describe()) ``` ```plaintext count 1638.000000 mean 14.999921 std 0.305004 min 11.611000 25% 14.961000 50% 15.000000 75% 15.042750 max 18.405000 Name: ts_event, dtype: float64 ``` We can see that MUFT updates approximately every 15 seconds, with small variations in the observed interval between updates. We can use this method for any index in the DBIX dataset. ## Plotting index bid, ask, and price values The official price of an index is calculated from last sale data. However, for some indices, a bid and ask value may also be calculated which is similar to the index value but is calculated from bid and ask prices instead of last sale prices. Databento normalizes these fields to `bid_px_00`, `ask_px_00`, and `price` in the [bbo-1s schema](https://databento.com/docs/schemas/bbo.md). We can plot these values for SPX: > **Info** > > Although these fields may resemble market prices, the bid and ask values are calculated index values > rather than executable orders, and there is no underlying order book depth. ```python import databento as db import matplotlib.pyplot as plt client = db.Historical(key="$YOUR_API_KEY") spx = client.timeseries.get_range( dataset="DBIX.CORE", symbols=["SPX"], start="2026-02-18", schema="bbo-1s", limit=400 ).to_df() prices = spx[["ask_px_00", "price", "bid_px_00"]] prices = prices.rename(columns={ "ask_px_00": "ask", "bid_px_00": "bid"}) prices.plot() plt.show() ``` ![Index price plot](https://databento.com/docs/indices-introduction-0.svg.md) ## Special conventions for indices on Databento - **Source timestamp precision.** Although Databento timestamps are represented in nanoseconds, some venues within DBIX may not provide that same granularity natively. Databento adds no additional source precision beyond that which is provided by the underlying venues. - **Instrument definitions.** DBIX instruments are always normalized with `instrument_class` set to `INDEX`, and `match_algorithm` set to `UNDEFINED`, since there is no matching engine associated with an index feed. - **Indicative index values.** Some DBIX indices under the the `publisher_id` `DBIX.CORE.CGI` may publish indicative values, which are estimated or preliminary values marked as not being final. For these records, Databento applies the `F_PUBLISHER_SPECIFIC` flag, allowing them to be distinguished from normal index values. - See more on our [venue details page](https://databento.com/docs/datasets/dbix-cgif.md). --- Source: https://databento.com/docs/examples/indices/implied-spx-index?lang=python # Implied SPX index from ES futures curve ## Overview In this example, we'll explore how to estimate the implied spot price for the S&P 500 Index using best bid and offer (BBO) data from the ES curve with Databento's [Historical client](https://databento.com/docs/api-historical/client.md). We will then validate the implied price against the true underlying index value from our [Databento Core Indices dataset]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE). Because ES futures trade nearly continuously and update more frequently than the published SPX cash index, they can be used to construct a real-time proxy for the underlying index level. ## Futures overview and cost of carry A [futures contract](https://databento.com/docs/examples/futures/futures-introduction.md) is a standardized legal agreement to buy or sell an underlying asset at a specified price on a future date. Some futures, including ES futures, are cash-settled rather than settled through delivery of the underlying asset. The price of a futures contract at a given expiration is determined by the current spot price, adjusted for factors associated with carrying exposure to the underlying asset until expiration. This adjustment is commonly referred to as the "net cost of carry." For equity index futures, the net cost of carry primarily reflects the financing cost of purchasing the underlying portfolio and the dividend income forgone by holding the futures contract instead. Under the standard model, the futures price can be expressed using the risk-free interest rate and the continuous dividend yield as follows: \[ F_i = S \cdot e^{(R - D)\, T_i} \] Where: - \(F_i\) is the *i*-th futures contract price - \(S\) is the current spot price of the underlying asset - \(R\) is the risk-free interest rate - \(D\) is the continuous dividend yield - \(T_i\) is the time to expiration of the *i*-th futures contract ## Solving for the spot price Interest rates and expected dividends may be difficult to source. However, because ES futures trade across multiple expirations, we can use a mathematical trick to infer their combined effect from the futures curve. If we assume that the continuously compounded net carry rate is approximately constant across the first two maturities, we can estimate an implied net carry rate from their observed prices and use it to recover the underlying spot price. > **Info** > > It is important to note that this assumption simplifies the true nature of the net cost of carry. > In practice, dividends are paid on discrete dates and may be concentrated unevenly across quarters. > Similarly, interest rates also vary by term and are often modeled as piece-wise functions with knots on FOMC days. > > We are also assuming that the implied net carry rate remains constant across maturities. For the first and second contracts: \[ F_1 = S \cdot e^{(R - D)\, T_1} \] \[ F_2 = S \cdot e^{(R - D)\, T_2} \] Dividing the second equation by the first eliminates the unknown spot price: \[ \frac{F_2}{F_1} = e^{(R - D)(T_2 - T_1)} \] Taking the natural logarithm of both sides yields: \[ \ln\!\left(\frac{F_2}{F_1}\right) = (R - D)(T_2 - T_1) \] We can therefore estimate the net carry rate as: \[ R - D = \frac{\ln\!\left(\dfrac{F_2}{F_1}\right)}{T_2 - T_1} \] Now that we have found an estimate for the net carry rate, we can substitute it into the original futures-pricing equation and solve for spot: \[ S = \frac{F_1}{e^{(R - D)\, T_1}} \] Substituting the estimated value of \(R - D\), we obtain an ES-implied proxy for the SPX spot price: \[ S = \frac{F_1}{\exp\!\left(\dfrac{\ln\!\left(\dfrac{F_2}{F_1}\right)}{T_2 - T_1} \cdot T_1\right)} \] With this equation, implementing the calculation using the API becomes straightforward: ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Set time range start = "2026-01-05T00:00:00" end = "2026-01-05T21:00:00" # Set dataset and symbols dataset = "GLBX.MDP3" symbols = ["ES.v.0", "ES.v.1"] # Get 1-second BBO for the front two continuous ES contracts (ES.v.0, ES.v.1) es = client.timeseries.get_range( dataset=dataset, symbols=symbols, start=start, end=end, schema="bbo-1s", stype_in="continuous", stype_out="instrument_id", ) es_df = es.to_df() # Get definitions to map each instrument to its expiration date definitions = client.timeseries.get_range( dataset=dataset, symbols=symbols, start=start, end=end, schema="definition", stype_in="continuous", ) definitions_df = definitions.to_df() # Build a map from instrument_id to expiration expiration_map = definitions_df.groupby("instrument_id")["expiration"].first().to_dict() # Attach each row's expiration to the BBO DataFrame es_df["expiration"] = es_df["instrument_id"].map(expiration_map) # Time to expiration in years (T_i), and the mid price as our price proxy es_df["time_to_expire"] = (es_df["expiration"] - es_df["ts_event"]).dt.total_seconds() / (365 * 24 * 60 * 60) es_df["mid"] = (es_df["bid_px_00"] + es_df["ask_px_00"]) / 2 # Split into the first (F_1) and second (F_2) contracts es0 = es_df[es_df["symbol"] == "ES.v.0"][["mid", "time_to_expire"]] es1 = es_df[es_df["symbol"] == "ES.v.1"][["mid", "time_to_expire"]] # Estimate the net carry rate (R - D) = ln(F_2/F_1) / (T_2 - T_1) r_d = (np.log(es1["mid"] / es0["mid"])) / (es1["time_to_expire"] - es0["time_to_expire"]) # Back out the implied spot: S = F_1 / e^[(R - D) * T_1] spx = es0["mid"] / np.exp(r_d * es0["time_to_expire"]) # Combine the prices into a combined DataFrame proxy_df = pd.DataFrame({"es0": es0["mid"], "es1": es1["mid"], "spx": spx}) print(proxy_df.head(5)) ``` ```plaintext es0 es1 spx ts_recv 2026-01-05 00:00:00+00:00 6907.125 6958.375 6864.956430 2026-01-05 00:00:01+00:00 6907.375 6958.375 6865.410758 2026-01-05 00:00:02+00:00 6907.375 6958.625 6865.206449 2026-01-05 00:00:03+00:00 6907.500 6958.625 6865.433613 2026-01-05 00:00:04+00:00 6907.250 6958.375 6865.183636 ``` In just a few lines of code, we have created an SPX index proxy that updates more quickly than the true index value. However, we still need to validate its accuracy. ## Validation We can use Databento's Core Indices dataset to request the actual underlying index value, check the accuracy of our estimate, and plot the two time series against one another: ```python # Create a request using the DBIX.CORE dataset. index = client.timeseries.get_range( dataset="DBIX.CORE", symbols="SPX", start=start, end=end, schema="bbo-1m", ) index_df = index.to_df() index_df["idx"] = (index_df["bid_px_00"] + index_df["ask_px_00"]) / 2 # Combine data, keeping only rows from the DBIX DataFrame prices_df = pd.merge(left=index_df[["idx"]], right=proxy_df, how="left", left_index=True, right_index=True) prices_df = prices_df[prices_df["spx"].notnull()] # Plot the implied SPX proxy against the true index price plt.figure(figsize=(16, 8)) prices_df["spx"].plot(label="SPX Proxy") prices_df["idx"].plot(label="SPX Index") plt.title("ES-implied SPX spot proxy vs. true SPX spot") plt.xlabel("Date") plt.ylabel("Price") plt.legend() plt.show() ``` ![SPX implied proxy](https://databento.com/docs/implied-spx-index-0.svg.md) ### Error To measure the error between our SPX index proxy and the true SPX index, we can use the Root Mean Squared Error (RMSE), which quantifies the differences (residuals) between predicted and actual values. It squares each difference to penalize larger errors and eliminate negative errors, averages the squared differences, and takes the square root to return the error to the original units. The exact equation is: \[ \text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} \left(\hat{S}_i - S_i\right)^2} \] Where: - \(\hat{S}_i\) is the ES-implied proxy at observation *i* - \(S_i\) is the true index value at observation *i* - \(n\) is the number of observations We can achieve this quite easily in Python with no additional packages: ```python # Calculate RMSE rmse = np.sqrt(((prices_df["spx"] - prices_df["idx"])**2).sum() / len(prices_df)) print("Root Mean Squared Error:", round(rmse, 4)) ``` ```python Root Mean Squared Error: 0.6386 ``` So our estimate is only off by an RMSE of ~0.64 index points. This is an excellent result, but to ensure this metric is reasonable, we still need to inspect the distribution of errors. ### Residuals For a better idea of how the errors are distributed, we can create a residual plot. This plot displays each error as a point on a graph, allowing us to visually inspect whether our estimate is systematically biased in one direction. We can find a consistent bias by measuring the average size of our residuals: \[ \text{Bias} = \frac{1}{n} \sum_{i=1}^{n} \left(\hat{S}_i - S_i\right) \] We can calculate this value in Python and plot it alongside our residuals plot: ```python # Calculate mean error bias = (prices_df["spx"] - prices_df["idx"]).mean() # Plot residuals vs. true index price. plt.scatter(x=prices_df["idx"], y=prices_df["spx"] - prices_df["idx"] ) plt.axhline(bias, color="C1", linestyle="--", label=f"Bias = {bias:.3f}") plt.axhline(0, color="C2", linestyle=":", label="Zero") plt.xlabel("SPX Index") plt.ylabel("Residuals (SPX Proxy - SPX Index)") plt.title("SPX proxy residuals vs. true SPX index") plt.legend() plt.show() print("Bias:", round(bias, 4)) ``` ![SPX residual plot](https://databento.com/docs/implied-spx-index-1.svg.md) ```python Bias: 0.517 ``` This plot shows that our estimate was consistently about half an index point too high, indicating that our simplified cost-of-carry model may introduce structural bias. This bias may vary across days due to the non-linear nature of the cost of carry. However, given that error is measured as follows: \[ \text{RMSE}^2 = \text{Bias}^2 + \sigma^2 \] We can calculate the standard deviation of our estimate around the bias. \[ \sigma = \sqrt{\text{RMSE}^2 - \text{Bias}^2} \] Then, we can calculate this value in Python: ```python variance = rmse**2 - bias**2 print("Standard Deviation:", round(variance**0.5)) ``` ```python Standard Deviation: 0.375 ``` Our estimate has a standard deviation of less than 1/2 of an index point around our bias, meaning that our proxy is quite robust. We can use this proxy to find at-the-money options strikes, find a continuous index level outside of regular trading hours, and compute realized volatility on a tick-by-tick basis. --- Source: https://databento.com/docs/examples/indices/option-moneyness?lang=python # Filter options by moneyness ## Overview In this example, we'll calculate the moneyness of SPXW options and filter to strikes within 5% of the ATM price. We'll calculate moneyness for each option by combining SPXW option definitions and statistics from Databento's [OPRA dataset]($PLATFORM_HOSTNAME/portal/catalog/opra) with SPX index values from the [Databento Core Indices dataset]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE). ## Moneyness Moneyness describes where an option's strike sits relative to the current price of the underlying. The moneyness of an option is separated into three trading states: - **In-the-money (ITM)**: The option has positive intrinsic value. - **At-the-money (ATM)**: The option's strike price is almost identical to the market price of the underlying asset. - **Out-of-the-money (OTM)**: The option has no intrinsic value. Options closer to ATM typically have higher trading volume. For any option: \[ \text{Moneyness} = \frac{K}{S} - 1 \] Where: - \(K\) is the strike price of the contract - \(S\) is the current price of the underlying asset A negative value places the strike below spot, which is in-the-money for a call and out-of-the-money for a put. The opposite is true for a positive value. Both the sign and the contract type determine the state of an option. For easier readability, we will multiply this value by 100 to express moneyness as a percentage rather than a decimal. ## Calculating moneyness To calculate moneyness, we'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) to retrieve the necessary information for each option, such as `expiration`, `strike_price`, and `instrument_class`. We'll then pull the spot price at market open using the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md). ```python import databento as db import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Fetch data for all expirations for the week of Monday, April 27 start = "2026-04-27" # Fetch the SPX index price from market open spx_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1m", symbols=["SPX"], start=start + "T13:30", end=start + "T13:31", ).to_df() spx = spx_df[spx_df["symbol"] == "SPX"]["open"].iloc[0] # Fetch SPXW options definitions spxw_definitions = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Filter to outright contracts expiring by the end of the week friday_expiry = pd.to_datetime(start, utc=True) + pd.Timedelta(days=4) spxw_definitions = spxw_definitions[spxw_definitions["expiration"] <= friday_expiry] spxw_definitions = spxw_definitions[spxw_definitions["instrument_class"].isin(["P", "C"])] # Drop all unnecessary rows for this process spxw_metadata = spxw_definitions[["instrument_id", "strike_price", "expiration", "instrument_class"]] # Calculate Moneyness per contract (For the sake of this example, we convert to a percentage by multiplying by 100) spxw_metadata["moneyness"] = (spxw_metadata["strike_price"] / spx - 1) * 100 print(spxw_metadata.head()) ``` ```plaintext instrument_id strike_price expiration instrument_class moneyness ts_recv 2026-04-27 10:30:04.059606168+00:00 1191185088 6500.0 2026-05-01 00:00:00+00:00 C -9.125480 2026-04-27 10:30:04.069618621+00:00 1191185095 7150.0 2026-05-01 00:00:00+00:00 C -0.038027 2026-04-27 10:30:04.069618621+00:00 1191185096 5200.0 2026-05-01 00:00:00+00:00 C -27.300384 2026-04-27 10:30:04.079627286+00:00 1191185103 5950.0 2026-05-01 00:00:00+00:00 P -16.814862 2026-04-27 10:30:04.079627286+00:00 1191185104 6300.0 2026-05-01 00:00:00+00:00 C -11.921619 ``` ## Visualizing activity by strike Now that we have calculated the moneyness of each strike price, we can visualize the trading activity within 5% of the opening ATM price. A single SPXW expiration lists strikes across a range far wider than the underlying is likely to move, and most of them barely trade, so any workflow that consumes the chain could benefit from cutting it down to an active region. ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Fetch data for all expirations for the week of Monday, April 27 start = "2026-04-27" # Fetch the SPX index price from market open spx_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1m", symbols=["SPX"], start=start + "T13:30", end=start + "T13:31", ).to_df() spx = spx_df[spx_df["symbol"] == "SPX"]["open"].iloc[0] # Fetch SPXW options definitions spxw_definitions = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Filter to outright contracts expiring by the end of the week friday_expiry = pd.to_datetime(start, utc=True) + pd.Timedelta(days=4) spxw_definitions = spxw_definitions[spxw_definitions["expiration"] <= friday_expiry] spxw_definitions = spxw_definitions[spxw_definitions["instrument_class"].isin(["P", "C"])] # Drop all unnecessary rows for this process spxw_metadata = spxw_definitions[["instrument_id", "strike_price", "expiration", "instrument_class"]] # Calculate Moneyness per contract (For the sake of this example, we convert to a percentage by multiplying by 100) spxw_metadata["moneyness"] = (spxw_metadata["strike_price"] / spx - 1) * 100 # Fetch SPXW options statistics spxw_statistics = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="statistics", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Include only daily volume statistics spxw_volumes = spxw_statistics[spxw_statistics["stat_type"] == db.StatType.CLEARED_VOLUME][["instrument_id", "quantity"]] spxw_volumes = spxw_volumes.groupby("instrument_id").last().reset_index().rename(columns={"quantity": "volume"}) # Merge the metadata and volume dataframes spxw = pd.merge(spxw_volumes, spxw_metadata, on="instrument_id") # Create a reusable function to plot the volume data for each expiration date def volume_plots(axes, plot_df, expiries, alpha=1): for ax, expiry in zip(axes, expiries): # Filter the dataframe to only include options for the current expiration date expiry_df = plot_df[plot_df["expiration"] == expiry] puts = expiry_df[expiry_df["instrument_class"] == "P"] calls = expiry_df[expiry_df["instrument_class"] == "C"] x_values = np.sort(expiry_df["moneyness"].unique()) bar_width = np.median(np.diff(x_values)) * 0.85 # Bar width will be 85% of the median price difference between strikes # Plot the volume for calls and puts as bars, with calls above the x-axis and puts below the x-axis ax.bar(calls["moneyness"], calls["volume"], width=bar_width, label="Calls", alpha=alpha, color="C0") ax.bar(puts["moneyness"], -puts["volume"], width=bar_width, label="Puts", alpha=alpha, color="C1") # Add a vertical line at moneyness = 0 and a horizontal line at volume = 0 ax.axvline(0, linestyle="--", linewidth=1) ax.axhline(0, linewidth=0.8, color="black") ax.set_title(pd.Timestamp(expiry).strftime("%a %b %d"), loc="left") ax.set_ylabel("Trading Volume") # Filter to only include options that are within 5% of the SPX index price plot_df = spxw[spxw["moneyness"].between(-5, 5)] # Create a list of unique expiration dates for the options expiries = sorted(plot_df["expiration"].unique()) # Create the subplots for each expiration date fig, axes = plt.subplots(nrows=len(expiries), ncols=1, figsize=(15, 12), sharex=True) volume_plots(axes, plot_df, expiries) axes[-1].set_xlabel("Moneyness (%)") axes[0].legend() fig.suptitle("SPXW Volume by Expiration", fontsize=16) plt.show() ``` ![Strikes by volume](https://databento.com/docs/option-moneyness-2-0.svg.md) --- Source: https://databento.com/docs/examples/indices/spx-and-the-vix?lang=python # Filter SPXW strikes by VIX-implied expected move ## Overview In this example, we will use the VIX to define a volatility-adjusted active strike range across SPXW expirations from the [OPRA feed]($PLATFORM_HOSTNAME/portal/catalog/opra) using SPX and VIX index values from Databento's [Databento Core Indices dataset]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE). > **Info** > > This article is a continuation of our [Filter options by moneyness example](https://databento.com/docs/examples/indices/option-moneyness.md). For more details on moneyness, visit that page first. ## The VIX The VIX, or Cboe Volatility Index, is a market index that measures the S&P 500's anticipated volatility in the near future. VIX is derived from the SPX option chain and expresses expected 30-day volatility in annualized terms. For this example, we will focus on 0DTE-5DTE options contracts, so we will instead use shorter-term versions of the VIX: VIX1D and VIX9D, which measure the S&P 500's expected volatility for the current trading day and the next 9 days respectively. Then, we will use these two points to interpolate the implied total variance for each expiration. We use variance instead of volatility here because variance is additive over time. To do this, we first need to convert volatility to annualized variance by squaring: \[ Var_{1D} = (\text{VIX1D})^2 \] \[ Var_{9D} = (\text{VIX9D})^2 \] Next, we will account for the time fraction of each index to get the actual total variance expected over their respective horizons: \[ \text{Total Var}_{1D} = Var_{1D} \times \frac{1}{365} \] \[ \text{Total Var}_{9D} = Var_{9D} \times \frac{9}{365} \] We then linearly interpolate for the target day by calculating where it sits proportionally between day 1 and day 9: \[\text{Total\ Var}_{ND}=\text{Total\ Var}_{1D}+\left(\frac{N-1}{9-1}\right)\times (\text{Total\ Var}_{9D}-\text{Total\ Var}_{1D})\] Finally, we take the square root of the total variance to get the expected move over n days. This converts the variance back into a percentage move, the same unit as moneyness: \[ \text{Expected Move}_{ND} = \sqrt{\text{Total Var}_{ND}} \] We can achieve this very simply in Python: ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Fetch data for all expirations for the week of Monday, April 27 start = "2026-04-27" # Get the opening values for the VIX (not available exactly at open, since the VIX is calculated from SPX options) vix_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1s", symbols=["VIX1D", "VIX9D"], start=start + "T13:31", end=start + "T13:32", ).to_df() vix1d = vix_df[vix_df["symbol"] == "VIX1D"]["open"].iloc[0] vix9d = vix_df[vix_df["symbol"] == "VIX9D"]["open"].iloc[0] # Square the indices to get annualized variances var_1d = vix1d ** 2 var_9d = vix9d ** 2 # Convert to annualized variances for the respective horizons total_var_1d = var_1d * (1 / 365) total_var_9d = var_9d * (9 / 365) # Interpolate days 1 through 5 expected_moves = {} for n in range(1, 6): weight = (n - 1) / (9 - 1) total_var_n = (total_var_1d + weight * (total_var_9d - total_var_1d)) # Convert to expected move and map to expiration expected_moves[pd.to_datetime(start, utc=True) + pd.Timedelta(days=n - 1)] = total_var_n ** 0.5 # Now that we have interpolated the expected moves for each expiry, # we can filter for all strikes within one standard deviation of the spot. # Under normal assumptions, there is a ~68% chance that the spot price will settle within this band. # Fetch the SPX index price from market open to calculate moneyness spx_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1m", symbols=["SPX"], start=start + "T13:30", end=start + "T13:31", ).to_df() spx = spx_df[spx_df["symbol"] == "SPX"]["open"].iloc[0] # Fetch SPXW options definitions spxw_definitions = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Filter to outright contracts expiring by the end of the week friday_expiry = pd.to_datetime(start, utc=True) + pd.Timedelta(days=4) spxw_definitions = spxw_definitions[spxw_definitions["expiration"] <= friday_expiry] spxw_definitions = spxw_definitions[spxw_definitions["instrument_class"].isin(["P", "C"])] # Drop all unnecessary rows for this process spxw_metadata = spxw_definitions[["instrument_id", "strike_price", "expiration", "instrument_class"]] # Calculate Moneyness per contract (For the sake of this example, we convert to a percentage by multiplying by 100) spxw_metadata["moneyness"] = (spxw_metadata["strike_price"] / spx - 1) * 100 # Fetch SPXW options statistics spxw_statistics = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="statistics", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Include only daily volume statistics spxw_volumes = spxw_statistics[spxw_statistics["stat_type"] == db.StatType.CLEARED_VOLUME][["instrument_id", "quantity"]] spxw_volumes = spxw_volumes.groupby("instrument_id").last().reset_index().rename(columns={"quantity": "volume"}) # Merge the metadata and volume dataframes spxw = pd.merge(spxw_volumes, spxw_metadata, on="instrument_id") # Create a reusable function to plot the volume data for each expiration date def volume_plots(axes, plot_df, expiries, alpha=1): for ax, expiry in zip(axes, expiries): # Filter the dataframe to only include options for the current expiration date expiry_df = plot_df[plot_df["expiration"] == expiry] puts = expiry_df[expiry_df["instrument_class"] == "P"] calls = expiry_df[expiry_df["instrument_class"] == "C"] x_values = np.sort(expiry_df["moneyness"].unique()) bar_width = np.median(np.diff(x_values)) * 0.85 # Bar width will be 85% of the median price difference between strikes # Plot the volume for calls and puts as bars, with calls above the x-axis and puts below the x-axis ax.bar(calls["moneyness"], calls["volume"], width=bar_width, label="Calls", alpha=alpha, color="C0") ax.bar(puts["moneyness"], -puts["volume"], width=bar_width, label="Puts", alpha=alpha, color="C1") # Add a vertical line at moneyness = 0 and a horizontal line at volume = 0 ax.axvline(0, linestyle="--", linewidth=1) ax.axhline(0, linewidth=0.8, color="black") ax.set_title(pd.Timestamp(expiry).strftime("%a %b %d"), loc="left") ax.set_ylabel("Trading Volume") # Filter to only include options that are within 5% of the SPX index price plot_df = spxw[spxw["moneyness"].between(-5, 5)] # Create a list of unique expiration dates for the options expiries = sorted(plot_df["expiration"].unique()) # Create a mask for options that are within the expected move range and those that are outside of it inside_range_mask = abs(plot_df["moneyness"]) < plot_df["expiration"].map(expected_moves) outside_range_mask = abs(plot_df["moneyness"]) > plot_df["expiration"].map(expected_moves) # Use the function defined above to plot the options that are within the expected move range # and those that are outside of it, with different alpha values for each fig, axes = plt.subplots(nrows=len(expiries), ncols=1, figsize=(15, 12), sharex=True) volume_plots(axes, plot_df[inside_range_mask], expiries, alpha=1) axes[0].legend() # Avoid duplicate legends by only adding it to the first subplot volume_plots(axes, plot_df[outside_range_mask], expiries, alpha=0.3) # Add vertical lines to indicate the expected move range for each expiration date for i in range(len(expiries)): axes[i].axvspan(-expected_moves[expiries[i]], expected_moves[expiries[i]], alpha=0.08) axes[i].axvline(-expected_moves[expiries[i]], linestyle=":") axes[i].axvline( expected_moves[expiries[i]], linestyle=":") axis_title = f"{pd.Timestamp(expiries[i]).strftime('%a %b %d')} -- Volatility range: ±{expected_moves[expiries[i]]:.2f}%" axes[i].set_title(axis_title, loc="left") axes[-1].set_xlabel("Moneyness (%)") fig.suptitle("SPXW Volume by Expiration", fontsize=16) plt.show() ``` ![Strikes filtered by volatility](https://databento.com/docs/spx-and-the-vix-0.svg.md) Each shaded band now scales with the expiration's expected volatility rather than by a fixed percentage. Because we have the moneyness mapped to each `instrument_id`, we can easily subscribe to only the contracts we have identified as liquid for each expiration. --- Source: https://databento.com/docs/examples/options?lang=python ## Options | Title | Subtitle | Image | Supported languages | | [Equity options: Introduction](https://databento.com/docs/examples/options/equity-options-introduction.md) | Get started with stock options data. Fetch options chains, strikes, expirations, NBBO. | ![example1](https://databento.com/docs/equity-options-introduction.svg.md) | Python, Options, Historical | | [Options on futures: Introduction](https://databento.com/docs/examples/options/options-on-futures-introduction.md) | Get started with options on futures data. Fetch options chains, strikes, expirations. | ![example2](https://databento.com/docs/options-on-futures-introduction.svg.md) | Python, Options, Historical | | [All options with a given underlying](https://databento.com/docs/examples/options/options-with-underlying.md) | Find all the quarterly, monthly (serial), weekly, and daily options of an underlying. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Options, Historical | | [Join options with underlying prices](https://databento.com/docs/examples/options/options-and-futures.md) | Get front month outright options and join with the latest underlying futures quotes. | ![Image placeholder](https://databento.com/docs/options-and-futures-thumbnail.svg.md) | Python, Options, Historical | | [US equity options volume by venue](https://databento.com/docs/examples/options/options-venues.md) | Compare volume across all venues in the US equity options market using pandas. | ![Image placeholder](https://databento.com/docs/options-venues-thumbnail.svg.md) | Python, Options, Historical | | [Resample US equity options NBBO](https://databento.com/docs/examples/options/nbbo-resampling.md) | Fetch NBBO and downsample to 1-minute intervals for all options in a chain. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Options, Historical | | [Estimate implied volatility](https://databento.com/docs/examples/options/estimating-implied-volatility.md) | Compute and graph implied volatility from options prices. | ![example6](https://databento.com/docs/estimating-implied-volatility-thumbnail.svg.md) | Python, Options, Historical | | [Get symbols for 0DTE options](https://databento.com/docs/examples/options/zero-dte-options.md) | Get a list of 0DTE options for a product. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Options, Historical, Live | | [Daily statistics for equity options](https://databento.com/docs/examples/options/equity-open-interest.md) | Calculate volume and open interest for an equity options chain. | ![Image placeholder](https://databento.com/docs/equity-open-interest-thumbnail.svg.md) | Python, Options, Historical | | [Get end of day option spreads](https://databento.com/docs/examples/options/option-spreads.md) | Calculate end of day options spread by expirations. | ![Image placeholder](https://databento.com/docs/option-spreads-thumbnail.svg.md) | Python, Options, Historical | --- Source: https://databento.com/docs/examples/options/equity-options-introduction?lang=python # Equity options: Introduction > **Info** > > This article introduces additional concepts to get started with equity options data > on Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll introduce: - The OPRA dataset - Using instrument definitions to get symbols, strike prices, and expirations - Options Clearing Corporation (OCC) symbology for raw symbols - Using parent instrument symbology to fetch an option chain - Stream live options trades We'll also highlight any special conventions of equity options datasets on Databento that differ from those of other asset classes. ## Finding an equity options dataset To use equity options data on Databento, first identify the dataset that you want from our [data catalog]($PLATFORM_HOSTNAME) and go to its detail page. Here, you can find its dataset ID at the top left of the page. ![Dataset ID](https://databento.com/docs/equity-options-dataset-id.png.md) For this example, we'll use the [OPRA]($PLATFORM_HOSTNAME/datasets/OPRA.PILLAR) dataset, whose dataset ID is `OPRA.PILLAR`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## OPRA OPRA provides consolidated last sale and national BBO across all US equity options exchanges. Various types of options are covered, including: - Single name stock options (e.g. AAPL, TSLA) - ETF options (e.g. SPY, QQQ) - Index options (e.g. SPX/SPXW, VIX) > **Info** > > The "Pillar" suffix in the `OPRA.PILLAR` dataset ID is a reference to the SIAC Pillar > SIP, the latest infrastructure behind the binary OPRA feed since 2021. ## Using instrument definitions to get symbols, strike prices, and expirations A full list of instrument definitions can be fetched with the `definition` schema and passing in `symbols="ALL_SYMBOLS"` to the [timeseries.get\_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md?lang=python) method. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols="ALL_SYMBOLS", start="2024-08-09", limit=10000, ) df = data.to_df() print(df[["raw_symbol", "strike_price", "expiration"]]) ``` ```plaintext raw_symbol strike_price expiration ts_recv 2024-08-09 10:30:00.446956866+00:00 BKNG 250117P01770000 1770.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.446956866+00:00 BKNG NaN NaT 2024-08-09 10:30:00.446972667+00:00 BLK 250117C00430000 430.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.446972667+00:00 BLK NaN NaT 2024-08-09 10:30:00.448575941+00:00 UNH 250117P00640000 640.0 2025-01-17 00:00:00+00:00 ... ... ... ... 2024-08-09 10:30:00.586069426+00:00 SPY 240814P00561000 561.0 2024-08-14 00:00:00+00:00 2024-08-09 10:30:00.586076435+00:00 SMCI 250117C01080000 1080.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.586081478+00:00 SMCI 250117C01440000 1440.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.586087572+00:00 QQQ 250331C00415000 415.0 2025-03-31 00:00:00+00:00 2024-08-09 10:30:00.586107573+00:00 SHY 250117P00085000 85.0 2025-01-17 00:00:00+00:00 [10000 rows x 3 columns] ``` Note that we used a timeseries method here and the results are indexed in time. These are called **point-in-time instrument definitions**. Many data providers treat instrument listings and definitions as a static table, so this may initially feel unintuitive if you're new to this approach. However, for options, this is important as new instruments are added intraday as the underlying moves, and knowing the strike prices of these instruments may result in lookahead effects. ## OCC symbology Databento's raw symbols for equity options are based on OCC symbology, also known as Options Symbology Initiative (OSI) symbology. The [OPRA dataset specification](https://databento.com/docs/datasets/opra-pillar.md#symbology) provides an overview of this symbology format. You can specify the symbology type to use for your input symbols with the `stype_in` parameter. To demonstrate this, let's fetch the raw symbol `SPY 241115P00525000` with `stype_in="raw_symbol"`: the SPY 525.00 put expiring on 2024-11-15. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="cmbp-1", stype_in="raw_symbol", symbols=["SPY 241115P00525000"], start="2024-08-09T09:30-04:00", end="2024-08-09T10:00-04:00", ) df = data.to_df() print(df[["symbol", "bid_px_00", "ask_px_00"]].head()) ``` ```plaintext symbol bid_px_00 ask_px_00 ts_recv 2024-08-09 13:30:00.025137089+00:00 SPY 241115P00525000 16.33 16.53 2024-08-09 13:30:00.167110712+00:00 SPY 241115P00525000 16.33 16.52 2024-08-09 13:30:00.182316856+00:00 SPY 241115P00525000 16.34 16.51 2024-08-09 13:30:00.238455572+00:00 SPY 241115P00525000 16.34 16.50 2024-08-09 13:30:00.340439305+00:00 SPY 241115P00525000 16.34 16.51 ``` ## Using parent symbology to fetch an option chain Working with individual raw symbols can be tedious. Many users find it easier to specify the underlying's symbol and fetch all option symbols for that underlying in an **option chain** at once. This can be done using our [parent symbology](https://databento.com/docs/conventions/symbology.md#parent) by passing in `stype_in="parent"` to your request. For example, we can print all tick-by-tick trades of options on AAPL for a given time range as follows: ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="trades", stype_in="parent", symbols=["AAPL.OPT"], start="2024-08-09T09:30-04:00", end="2024-08-09T10:00-04:00", ) df = data.to_df() print(df[["symbol", "price"]]) ``` ```plaintext symbol price ts_recv 2024-08-09 13:30:00.008689345+00:00 AAPL 240809C00217500 0.06 2024-08-09 13:30:00.021880156+00:00 AAPL 240816C00230000 0.10 2024-08-09 13:30:00.032886565+00:00 AAPL 240920P00175000 0.77 2024-08-09 13:30:00.036991612+00:00 AAPL 240809P00202500 0.08 2024-08-09 13:30:00.109699333+00:00 AAPL 240816C00215000 2.23 ... ... ... 2024-08-09 13:59:58.801315993+00:00 AAPL 240816C00207500 6.80 2024-08-09 13:59:58.962219427+00:00 AAPL 240809C00212500 1.24 2024-08-09 13:59:59.036621416+00:00 AAPL 240816C00240000 0.02 2024-08-09 13:59:59.124512360+00:00 AAPL 240809C00212500 1.24 2024-08-09 13:59:59.858997181+00:00 AAPL 240809C00220000 0.02 [22451 rows x 2 columns] ``` ## Streaming live equity options trades For this example, we will stream the [TCBBO schema](https://databento.com/docs/schemas/tbbo.md#consolidated-bbo-on-trade-tcbbo) for all AAPL options. This schema shows every trade event along with the NBBO immediately *before* the effect of each trade. Over 99.9% of events on the OPRA feed are CMBP-1 updates. It's significantly easier to work with another schema like [TCBBO](https://databento.com/docs/schemas/tbbo.md#consolidated-bbo-on-trade-tcbbo) or [trades](https://databento.com/docs/schemas/trades.md). While the OPRA feed does not disseminate the trade aggressor side for a trade (`side` will be always be `N`), you can use the NBBO before the trade executes to help make a determination on whether the trade was initiated by the buyer or seller. > **Info** > > This example requires a live license to `OPRA.PILLAR`. Visit our [live data portal]($PLATFORM_HOSTNAME/portal/live-data#manage) > to sign up. ```python import databento as db # Enable some basic logging db.enable_logging("INFO") # Create a live client and connect live_client = db.Live("$YOUR_API_KEY", compression=db.Compression.ZSTD) # Subscribe to the TCBBO schema for all active AAPL options live_client.subscribe( dataset="OPRA.PILLAR", schema="tcbbo", symbols="AAPL.OPT", stype_in="parent", ) # We add a print callback to view each record live_client.add_callback(print) # Start the live client to begin data streaming live_client.start() # Run the stream for 5 seconds before closing live_client.block_for_close(timeout=5) ``` > **Info** > > If you do not see any output, it could be because the markets are closed. See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) for using [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=16777235, ts_event=1769047399503117383, pretty_ts_event='2026-01-22T02:03:19.503117383Z', stype_in=None, stype_in_symbol='AAPL.OPT', stype_out=None, stype_out_symbol='AAPL 270115P00380000', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') ... Cmbp1Msg(ts_recv=1769005800009884453, pretty_ts_recv='2026-01-21T14:30:00.009884453Z', rtype=, publisher_id=OPRA.PILLAR.XPHL (35), instrument_id=16787902, ts_event=1769005800009683740, pretty_ts_event='2026-01-21T14:30:00.009683740Z', action='T', side='N', price=260000000, pretty_price=0.26, size=7, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=260000000, pretty_bid_px_00=0.26, ask_px_00=300000000, pretty_ask_px_00=0.3, bid_sz_00=13, ask_sz_00=3, bid_pb_00=35, ask_pb_00=35) Cmbp1Msg(ts_recv=1769005800010490784, pretty_ts_recv='2026-01-21T14:30:00.010490784Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16787347, ts_event=1769005800010290179, pretty_ts_event='2026-01-21T14:30:00.010290179Z', action='T', side='N', price=4950000000, pretty_price=4.95, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=9223372036854775807, pretty_bid_px_00=NaN, ask_px_00=9223372036854775807, pretty_ask_px_00=NaN, bid_sz_00=0, ask_sz_00=0, bid_pb_00=0, ask_pb_00=0) Cmbp1Msg(ts_recv=1769005800023091148, pretty_ts_recv='2026-01-21T14:30:00.023091148Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16787347, ts_event=1769005800022890199, pretty_ts_event='2026-01-21T14:30:00.022890199Z', action='T', side='N', price=5000000000, pretty_price=5, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=4900000000, pretty_bid_px_00=4.9, ask_px_00=5000000000, pretty_ask_px_00=5, bid_sz_00=2, ask_sz_00=1, bid_pb_00=26, ask_pb_00=26) ... Cmbp1Msg(ts_recv=1769005800202002589, pretty_ts_recv='2026-01-21T14:30:00.202002589Z', rtype=, publisher_id=OPRA.PILLAR.XBXO (33), instrument_id=16794630, ts_event=1769005800201802008, pretty_ts_event='2026-01-21T14:30:00.201802008Z', action='T', side='N', price=1920000000, pretty_price=1.92, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=1920000000, pretty_bid_px_00=1.92, ask_px_00=2030000000, pretty_ask_px_00=2.03, bid_sz_00=6, ask_sz_00=6, bid_pb_00=27, ask_pb_00=27) Cmbp1Msg(ts_recv=1769005800215279430, pretty_ts_recv='2026-01-21T14:30:00.215279430Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16790992, ts_event=1769005800215078713, pretty_ts_event='2026-01-21T14:30:00.215078713Z', action='T', side='N', price=410000000, pretty_price=0.41, size=3, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=350000000, pretty_bid_px_00=0.35, ask_px_00=420000000, pretty_ask_px_00=0.42, bid_sz_00=3, ask_sz_00=6, bid_pb_00=37, ask_pb_00=37) Cmbp1Msg(ts_recv=1769005800215680274, pretty_ts_recv='2026-01-21T14:30:00.215680274Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16780046, ts_event=1769005800215479700, pretty_ts_event='2026-01-21T14:30:00.215479700Z', action='T', side='N', price=480000000, pretty_price=0.48, size=6, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=460000000, pretty_bid_px_00=0.46, ask_px_00=910000000, pretty_ask_px_00=0.91, bid_sz_00=20, ask_sz_00=1, bid_pb_00=25, ask_pb_00=25) ``` ## Special conventions for equity options on Databento - **Empty levels and null prices.** Many option contracts are illiquid and may have an empty book, especially at the start of day. Empty levels will have null prices and zero depth. We represent null prices with type max, e.g. 2^63 - 1, as a sentinel value. Our client libraries will generally replace these sentinel values with a more idiomatic null representation--for example, if you pass `price_type="float"` (default) in to the [DBNStore.to\_df](https://databento.com/docs/api-historical/helpers/dbn-store-to-df.md?lang=python) method of the Python client, these sentinel values will be replaced with `NaN` in the resulting DataFrame. - **Bandwidth.** Due to the significant size of OPRA data, your live feed may get backed up if your client is not reading fast enough or network bandwidth is limited--this is a result of TCP flow control. Hence, we recommend setting up [dedicated connectivity](https://databento.com/docs/architecture/dedicated-connectivity-guide.md) if you intend to consume a large number of symbols or the entire dataset in real-time. - **Intraday listings.** Our instrument definitions are timestamped on both historical and live data. This point-in-time treatment is especially useful if you intend to subscribe to and trade new strikes as they get listed; to backtest this, and to avoid lookahead effects of knowing the underlying's future price changes. ## Numeric publisher IDs The source of each trade and quote is identified by a numeric publisher ID. > **See also** > > [US equity options volume by venue](https://databento.com/docs/examples/options/options-venues.md) to see how to > identify exchange from publisher ID. --- Source: https://databento.com/docs/examples/options/options-on-futures-introduction?lang=python # Options on futures: Introduction > **Info** > > This article builds off [Futures: Introduction](https://databento.com/docs/examples/futures/futures-introduction.md) and > introduces additional concepts to get started with options on futures. > If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) guide. ## Overview In this example, we'll demonstrate: - Using instrument definitions to get symbols, strike prices, and expirations. - Using parent instrument symbology to fetch an option chain. ## Finding an options on futures dataset Several futures exchanges also provide options on futures. You can follow the [Futures: Introduction](https://databento.com/docs/examples/futures/futures-introduction.md) guide to get the dataset ID of such an exchange. For this example, we'll use the [CME Globex MDP 3.0]($PLATFORM_HOSTNAME/datasets/GLBX.MDP3) dataset, whose dataset ID is `GLBX.MDP3`. ## Using instrument definitions to get symbols, strike prices, and expirations A full list of instrument definitions can be fetched with the `definition` schema and passing in `symbols='ALL_SYMBOLS'` to the [timeseries.get\_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md?lang=python) method. ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="definition", symbols="ALL_SYMBOLS", start="2023-08-25", limit=10000, ) # Convert to DataFrame df = data.to_df() print(df[["raw_symbol", "strike_price", "expiration"]]) ``` ```plaintext raw_symbol strike_price expiration ts_recv 2023-08-25 00:00:00+00:00 OZSH4 C1520 1520.00 2024-02-23 19:20:00+00:00 2023-08-25 00:00:00+00:00 MEOF4-MEOK4 NaN 2024-01-31 19:30:00+00:00 2023-08-25 00:00:00+00:00 LOX4 P250 2.50 2024-10-17 18:30:00+00:00 2023-08-25 00:00:00+00:00 AHMG4 NaN 2024-02-01 04:59:00+00:00 2023-08-25 00:00:00+00:00 OBK4 C29300 2.93 2024-04-25 18:30:00+00:00 ... ... ... ... 2023-08-25 00:00:00+00:00 L0BZ3 C-0070 -7.00 2023-12-01 19:00:00+00:00 2023-08-25 00:00:00+00:00 OZNV3 C1135 113.50 2023-09-22 21:00:00+00:00 2023-08-25 00:00:00+00:00 R2AU3 P1970 1970.00 2023-09-11 20:00:00+00:00 2023-08-25 00:00:00+00:00 G1MV3 P2100 2100.00 2023-10-02 17:30:00+00:00 2023-08-25 00:00:00+00:00 CHIG5-WDCG5 NaN 2025-02-24 20:00:00+00:00 [10000 rows x 3 columns] ``` Note that we used a timeseries method here and the results are indexed in time. These are called **point-in-time instrument definitions**. Many data providers treat instrument listings and definitions as a static table, so this may initially feel unintuitive if you're new to this approach. However, for options, this is important as new instruments are added intraday as the underlying moves, and knowing the strike prices of these instruments may result in lookahead effects. ## Using parent symbology to fetch an option chain Working with individual raw symbols can be tedious. Many users find it easier to specify the underlying's symbol and fetch all option symbols for that underlying in an **option chain** at once. This can be done using our [parent symbology](https://databento.com/docs/conventions/symbology.md#parent) by passing in `stype_in="parent"` to your request. For example, we can print all tick-by-tick trades of quarterly options on ES for a given time range as follows: ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request trades data data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols=["ES.OPT"], start="2023-08-25T12:00", end="2023-08-25T13:00", ) # Convert to DataFrame df = data.to_df() print(df[["symbol", "price"]].tail(10)) ``` ```plaintext symbol price ts_recv 2023-08-25 12:58:44.446885515+00:00 ESU3 P4390 53.25 2023-08-25 12:58:44.447489339+00:00 ESU3 P4410 61.50 2023-08-25 12:58:58.034159513+00:00 ESU3 C4460 30.50 2023-08-25 12:58:58.052849806+00:00 ESU3 P4390 53.25 2023-08-25 12:59:16.789312686+00:00 ESU3 C4500 17.25 2023-08-25 12:59:27.661545003+00:00 ESU3 C4420 49.00 2023-08-25 12:59:28.337200208+00:00 ESU3 P4410 61.50 2023-08-25 12:59:30.609532264+00:00 ESU3 C4460 30.50 2023-08-25 12:59:50.095229136+00:00 ESU3 C4440 39.25 2023-08-25 12:59:50.141519076+00:00 ESU3 P4380 49.25 ``` ## Special conventions for options on futures on Databento Most of the [conventions for futures](https://databento.com/docs/examples/futures/futures-introduction.md#special-conventions-for-futures-on-databento) will also apply to options on futures. - **Empty levels and null prices.** Many option contracts are illiquid and may have an empty book, especially at the start of day. Empty levels will have null prices and zero depth. We represent null prices with type max, e.g. 2^63 - 1, as a sentinel value. Our client libraries will generally replace these sentinel values with a more idiomatic null representation--for example, if you pass `price_type="float"` (default) in to the [DBNStore.to\_df](https://databento.com/docs/api-historical/helpers/dbn-store-to-df.md?lang=python) method of the Python client, these sentinel values will be replaced with `NaN` in the resulting DataFrame. - **Intraday listings.** Our instrument definitions are timestamped on both historical and live data. This point-in-time treatment is especially useful if you intend to subscribe to and trade new strikes as they get listed; to backtest this, and to avoid lookahead effects of knowing the underlying's future price changes. > **See also** > > [All options with a given underlying](https://databento.com/docs/examples/options/options-with-underlying.md). --- Source: https://databento.com/docs/examples/options/options-with-underlying?lang=python # All options with a given underlying ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve instrument definitions for all options on futures with a given underlying. This is useful for complex assets such as the options on [E-mini S&P 500 Futures](https://www.cmegroup.com/markets/equities/sp/e-mini-sandp500.html) which have different symbols for their quarterly, monthly (serial), weekly, and daily options. We'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties such as the security type and underlying of the instrument. We'll request the definitions for all symbols, and then filter the instruments using the `security_type` and `underlying` fields. We'll show the `asset` field, which refers to the parent symbol for that option, as well as `strike price` and `expiration` . > **Info** > > See the [Options on E-Mini S&P 500 Futures](https://www.cmegroup.com/trading/equity-index/weekly-eom-options-faq.html) > FAQ page for more information on the ES Options symbology. ## Example ```python import databento as db # Set underlying instrument symbol = "ESH5" # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Get definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="definition", symbols="ALL_SYMBOLS", start="2025-03-10", ) # Convert to DataFrame df = data.to_df() # Filter for instruments with a security type of "OOF" (options-on-futures) df = df[df["security_type"] == "OOF"] # Select only options for the specified underlying df = df[df["underlying"] == symbol] # Sort by strike price and select specific columns df = df.sort_values("strike_price") df = df[["raw_symbol", "underlying", "asset", "instrument_class", "strike_price", "expiration"]] print(f"{len(df):,d}", f"option(s) for {symbol}") print(df) ``` ```plaintext 6,494 option(s) for ESH5 ``` > **See also** > > Check out the [0DTE symbols](https://databento.com/docs/examples/options/zero-dte-options.md) example on how to filter these > options based on `expiration`. ```plaintext raw_symbol underlying asset instrument_class strike_price expiration ts_recv 2025-03-10 00:00:00+00:00 E2AH5 P100 ESH5 E2A P 100.0 2025-03-10 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 C100 ESH5 E3D C 100.0 2025-03-20 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3AH5 C100 ESH5 E3A C 100.0 2025-03-17 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 P100 ESH5 E3D P 100.0 2025-03-20 20:00:00+00:00 2025-03-10 00:00:00+00:00 E2DH5 P100 ESH5 E2D P 100.0 2025-03-13 20:00:00+00:00 ... ... ... ... ... ... ... 2025-03-10 00:00:00+00:00 ESH5 P8500 ESH5 ES P 8500.0 2025-03-21 13:30:00+00:00 2025-03-10 00:00:00+00:00 ESH5 P8600 ESH5 ES P 8600.0 2025-03-21 13:30:00+00:00 2025-03-10 00:00:00+00:00 ESH5 C8600 ESH5 ES C 8600.0 2025-03-21 13:30:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 C10100 ESH5 E3D C 10100.0 2025-03-20 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 P10100 ESH5 E3D P 10100.0 2025-03-20 20:00:00+00:00 [6494 rows x 6 columns] ``` --- Source: https://databento.com/docs/examples/options/options-and-futures?lang=python # Join options with underlying prices > **Info** > > This example builds off the [Join schemas on instrument ID](https://databento.com/docs/examples/basics-historical/joining-schemas.md) example. ## Overview In this example, we'll use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to get data for futures and options. First, we'll request [trades schema](https://databento.com/docs/schemas/trades.md) data for all options on the volume-based continuous contract and join them with the most recent quote from the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) for the underlying future. We'll also request [BBO-1m schema](https://databento.com/docs/schemas/bbo.md) data for the underlying future and a single option and plot the midprice for a full session. ## Example ```python import databento as db import matplotlib.dates as mdates import matplotlib.pyplot as plt import pandas as pd def get_bbo_data( dataset: str, instrument_id: int, start: str, end: str, ) -> pd.DataFrame: mbp_df = client.timeseries.get_range( dataset=dataset, schema="bbo-1m", symbols=instrument_id, stype_in="instrument_id", start=start, end=end, ).to_df(map_symbols=False) return mbp_df[["instrument_id", "bid_px_00", "ask_px_00"]] def plot_option_with_underlying( opt_df: pd.DataFrame, fut_df: pd.DataFrame, ) -> None: opt_df["mid_price"] = (opt_df["bid_px_00"] + opt_df["ask_px_00"]) / 2 fut_df["mid_price"] = (fut_df["bid_px_00"] + fut_df["ask_px_00"]) / 2 full_index = pd.date_range( start=start, end=end, freq="1min", tz="UTC", inclusive="right", ) opt_df = opt_df.reindex(full_index).ffill().astype(opt_df.dtypes) fut_df = fut_df.reindex(full_index).ffill().astype(fut_df.dtypes) opt_symbol = opt_df["raw_symbol"].iloc[0] fut_symbol = opt_df["underlying"].iloc[0] fig, ax1 = plt.subplots(figsize=(14, 6)) ax1.plot(fut_df.index, fut_df["mid_price"], color="C0", label=fut_symbol) ax1.set_ylabel(f"Futures midprice ({fut_symbol})", color="C0") ax1.tick_params(axis="y", labelcolor="C0") strike = opt_df["strike_price"].iloc[0] ax1.axhline(y=strike, color="gray", linestyle="--") ax1.text( x=fut_df.index[0], y=strike, s=f"Strike price: {strike:.0f}", color="gray", va="bottom", ha="left", ) ax2 = ax1.twinx() ax2.plot(opt_df.index, opt_df["mid_price"], color="C1", label=opt_symbol) ax2.set_ylabel(f"Options midprice ({opt_symbol})", color="C1") ax2.tick_params(axis="y", labelcolor="C1") ax1.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M")) ax1.xaxis.set_major_locator(mdates.HourLocator(interval=1)) ax1.set_title("Futures vs. Options Midprice") ax1.set_xlabel("Time (UTC)") lines1, labels1 = ax1.get_legend_handles_labels() lines2, labels2 = ax2.get_legend_handles_labels() ax1.legend(lines1 + lines2, labels1 + labels2, loc="upper left") ax1.grid(True) ax2.grid(False) fig.tight_layout() plt.show() # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" symbol = "CL.v.0" start = "2026-04-12T22:00:00" end = "2026-04-13T21:00:00" # Get symbol mapping for the continuous contract symbol_map = client.symbology.resolve( dataset=dataset, symbols=symbol, stype_in="continuous", stype_out="instrument_id", start_date="2026-04-12", ) # Get instrument ID for the resolved futures contract fut_id = int(symbol_map["result"][symbol][0]["s"]) # Get all option definitions def_df = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="definition", start="2026-04-12", ).to_df() # Filter for options with the continuous contract as an underlying opt_def_df = def_df[ (def_df["user_defined_instrument"] == db.UserDefinedInstrument.NO) & (def_df["instrument_class"].isin((db.InstrumentClass.CALL, db.InstrumentClass.PUT))) & (def_df["underlying_id"] == fut_id) ] opt_def_df = opt_def_df[ [ "ts_event", "expiration", "raw_symbol", "instrument_id", "asset", "strike_price", "underlying", "underlying_id", "instrument_class", ] ] # Get trades data for the filtered options opt_trades_df = ( client.timeseries.get_range( dataset=dataset, schema="trades", symbols=[f"{x}.OPT" for x in opt_def_df["asset"].unique()], stype_in="parent", start=start, end=end, ) .to_df() .rename(columns={"size": "trade_size", "price": "trade_price", "side": "trade_side"}) ) # Join options with their definitions opt_df = ( opt_trades_df.reset_index() .merge( opt_def_df.reset_index(), on="instrument_id", how="inner", suffixes=("", "_def"), ) .set_index("ts_recv") .sort_index() ) fut_mbp_df = client.timeseries.get_range( dataset=dataset, schema="mbp-1", symbols=fut_id, stype_in="instrument_id", start=start, end=end, ).to_df() fut_mbp_df = fut_mbp_df.rename( columns={ "bid_px_00": "underlying_bid", "ask_px_00": "underlying_ask", }, )[["underlying_bid", "underlying_ask"]] # Join most recent underlying bid/ask with options trades df = pd.merge_asof( opt_df, fut_mbp_df, left_index=True, right_index=True, direction="backward", ) print( df[ [ "ts_event", "instrument_id", "raw_symbol", "trade_price", "trade_size", "trade_side", "strike_price", "instrument_class", "expiration", "underlying_bid", "underlying_ask", ] ], ) # 42963414 is the `instrument_id` for NL2J6 P10000 opt_id = 42963414 opt_bbo_df = get_bbo_data(dataset, opt_id, start, end) opt_bbo_df = ( opt_bbo_df.reset_index().merge(opt_def_df, on="instrument_id", how="left").set_index("ts_recv") ) fut_bbo_df = get_bbo_data(dataset, fut_id, start, end) plot_option_with_underlying(opt_bbo_df, fut_bbo_df) ``` ## Result We can see all trades for options that have the volume-based continuous contract as an underlying. ```plaintext ts_event instrument_id raw_symbol trade_price trade_size ... strike_price instrument_class expiration underlying_bid underlying_ask ts_recv ... 2026-04-12 22:00:00.356333832+00:00 2026-04-12 22:00:00+00:00 42157669 LOK6 C9900 6.01 1 ... 99.00 C 2026-04-16 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.356370138+00:00 2026-04-12 22:00:00+00:00 42180553 LOK6 P8200 0.39 1 ... 82.00 P 2026-04-16 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.357402871+00:00 2026-04-12 22:00:00+00:00 42511805 ML2J6 C10000 2.40 2 ... 100.00 C 2026-04-13 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.357526014+00:00 2026-04-12 22:00:00+00:00 42639465 ML2J6 P9100 0.74 1 ... 91.00 P 2026-04-13 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.357609934+00:00 2026-04-12 22:00:00+00:00 42963412 NL2J6 P9600 0.99 1 ... 96.00 P 2026-04-14 18:30:00+00:00 NaN NaN ... ... ... ... ... ... ... ... ... ... ... ... 2026-04-13 20:57:18.223377118+00:00 2026-04-13 20:57:18.222967937+00:00 42308894 LOK6 P8500 0.27 1 ... 85.00 P 2026-04-16 18:30:00+00:00 97.96 97.98 2026-04-13 20:57:18.223891862+00:00 2026-04-13 20:57:18.223423419+00:00 42308894 LOK6 P8500 0.27 1 ... 85.00 P 2026-04-16 18:30:00+00:00 97.96 97.98 2026-04-13 20:57:19.267462126+00:00 2026-04-13 20:57:19.267057293+00:00 42631933 WL3J6 C9775 3.03 1 ... 97.75 C 2026-04-15 18:30:00+00:00 97.97 97.99 2026-04-13 20:58:04.991697756+00:00 2026-04-13 20:58:04.983728493+00:00 42958534 NL2J6 C10100 0.91 1 ... 101.00 C 2026-04-14 18:30:00+00:00 97.94 97.96 2026-04-13 20:58:38.292464827+00:00 2026-04-13 20:58:38.292053561+00:00 379004 LOK6 P7000 0.02 1 ... 70.00 P 2026-04-16 18:30:00+00:00 97.97 97.99 [17262 rows x 11 columns] ``` Now we'll plot the midprice of the underlying future and a single option for the duration of a full session. In this example, the option `NL2J6 P10000` expires on 2026-04-14. ![Futures and options midprice](https://databento.com/docs/options-and-futures-0.svg.md) --- Source: https://databento.com/docs/examples/options/options-venues?lang=python # US equity options volume by venue ## Overview This example will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve data from the entire US equity options market and compare volume across execution venues. We'll use the [OHLCV-1m](https://databento.com/docs/schemas/ohlcv.md) schema and look at data over a 30-minute window. This example could be modified to look at the entire session by changing the `start` and `end` variables, or by using the OHLCV-1d schema. The OPRA datafeed provides consolidated options data for the 20 US equity options venues. The `publisher_id` field indicates which venue the record is associated with. Using this information, we will create a table and plot similar to the data found in [Cboe's Market Volume Summary](https://www.cboe.com/us/options/market_statistics/) report. > **See also** > > For a full list of the OPRA venues, see the [OPRA documentation](https://databento.com/docs/datasets/opra-pillar.md#venues). ## Example ```python import databento as db import matplotlib.pyplot as plt from matplotlib.patches import Patch # Set parameters dataset = "OPRA.PILLAR" start = "2025-03-26T14:00" end = "2025-03-26T14:30" # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Request OHLCV-1m data for ALL_SYMBOLS and convert to DataFrame df = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1m", symbols="ALL_SYMBOLS", start=start, end=end, ).to_df() # Aggregate volume by venue df = df.groupby("publisher_id", as_index=False)["volume"].sum() df["percentage"] = df["volume"] / df["volume"].sum() * 100 # Rename publisher_id and columns for readability publishers = client.metadata.list_publishers() venue_name_map = { x["publisher_id"]: db.Publisher.from_int(x["publisher_id"]).venue.description for x in publishers } df["publisher_id"] = df["publisher_id"].map(venue_name_map) df = df.rename(columns={"publisher_id": "Venue", "volume": "Volume", "percentage": "% of Total"}) # Calculate volume by group and sort df["Group"] = df["Venue"].str.split(" ").str[0] df["Group Volume"] = df.groupby("Group")["Volume"].transform("sum") df = df.sort_values(by=["Group Volume", "Volume"], ascending=[False, False]) df = df[["Group", "Venue", "Volume", "% of Total"]] print(df.set_index(["Group", "Venue"]).to_string(float_format="%0.2f")) # Now, let's plot this data fig, ax = plt.subplots(figsize=(14, 6)) group_handles = [] # Legend handles for group_idx, group in enumerate(df["Group"].unique()): group_data = df[df["Group"] == group] # Create handle for plot legend group_pct = group_data["% of Total"].sum() group_handles.append(Patch(facecolor=f"C{group_idx}", label=f"{group}: {group_pct:.2f}%")) last_venue_pct = 0 for i, (_, venue_data) in enumerate(group_data.iterrows()): venue_pct = venue_data["% of Total"] # Used as height for each venue section # Add section for venue bar = ax.bar( x=group, height=venue_pct, bottom=last_venue_pct, color=f"C{group_idx}", alpha=1.0 - i * 0.15, ) # Add the venue label ax.text( x=group_idx, y=last_venue_pct + venue_pct / 2, s=" ".join(venue_data["Venue"].split(" ")[:2]), ha="center", va="center", ) last_venue_pct += venue_pct ax.legend(handles=group_handles) ax.set_title("U.S. Options Market Volume") ax.set_ylabel("% of Total Volume") plt.tight_layout() plt.show() ``` ```plaintext Volume % of Total Group Venue Cboe Cboe Options 954511 14.47 Cboe EDGX Options 381901 5.79 Cboe BZX Options 261275 3.96 Cboe C2 Options 152628 2.31 Nasdaq Nasdaq PHLX 435190 6.60 Nasdaq ISE 381951 5.79 Nasdaq Options 327166 4.96 Nasdaq GEMX 272005 4.12 Nasdaq MRX 182480 2.77 Nasdaq Texas Options 105090 1.59 NYSE NYSE American Options 865360 13.12 NYSE Arca Options 664865 10.08 MIAX MIAX Options 447819 6.79 MIAX Emerald 230339 3.49 MIAX Pearl 201093 3.05 MIAX Sapphire 124162 1.88 BOX BOX Options 386105 5.85 MEMX MEMX Options 222973 3.38 ``` ![Options venues volume](https://databento.com/docs/options-venues-0.svg.md) --- Source: https://databento.com/docs/examples/options/nbbo-resampling?lang=python # Resample US equity options NBBO ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve the regional and consolidated best bid and offer (BBO) for the US equities market from the OPRA dataset. From this, we'll filter out the NBBO and then downsample at a 1-minute interval. To do this we will use pandas, specifically the `DataFrame.groupby` method. Note that sometimes the BBO is referred to as a *quote*, and *ask* and *offer* are synonyms referring to the sell price. ## MBP-1 schema We'll use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) in this example, which contains the price and size for both the bid and the ask. For datasets such as OPRA that cover multiple execution venues (exchanges), the `publisher_id` field indicates which venue the record is associated with. There will be records for each individual venue to indicate the *regional* BBO, and then an additional record to represent the *consolidated* NBBO. For the OPRA dataset, which is the SIP for US equity options, the consolidated `publisher_id` is `30` (venue `OPRA`). Trades, which are included in the MBP-1 schema as records with `action=T`, also use `publisher_id` to indicate which venue they occurred on. > **See also** > > [API reference - Historical - metadata.list\_publishers](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md) ## Example ```python import databento as db import pandas as pd # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, request MBP-1 for the SPIKE options chain (a volatility index) df = client.timeseries.get_range( dataset="OPRA.PILLAR", start="2023-07-03", schema="mbp-1", stype_in="parent", symbols="SPIKE.OPT", limit=20_000, ).to_df() # Only include NBBO records (`publisher_id == 30`) df = df[df["publisher_id"] == db.Publisher.OPRA_PILLAR_OPRA.to_int()] # Resample the data at a fixed one minute interval, per symbol df_1m = df.groupby([pd.Grouper(freq="1T"), "symbol"]).first() # Print the NBBO print(df_1m[["bid_px_00", "ask_px_00"]]) ``` ## Result ```plaintext bid_px_00 ask_px_00 ts_recv symbol 2023-07-03 13:30:00+00:00 SPIKE 230719C00010000 5.25 5.40 SPIKE 230719C00010500 4.75 4.90 SPIKE 230719C00011000 4.25 4.40 SPIKE 230719C00011500 3.75 3.90 SPIKE 230719C00012000 3.25 3.40 ... ... ... 2023-07-03 13:34:00+00:00 SPIKE 231018C00047500 0.16 0.70 SPIKE 231018C00050000 0.12 0.66 SPIKE 231018C00055000 0.06 0.59 SPIKE 231018P00014000 0.15 0.69 SPIKE 231115C00050000 0.20 0.75 [1657 rows x 2 columns] ``` --- Source: https://databento.com/docs/examples/options/estimating-implied-volatility?lang=python # Estimate implied volatility ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to process instrument definition and MBP-1 data to graph implied volatility by strike price for the front-month [E-mini S&P 500 Futures (ES)](https://www.cmegroup.com/markets/equities/sp/e-mini-sandp500.html) contract. To do this we will request top-of-book data for the front-month ES futures contract using our [continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous) and the entire ES options chain using our [parent symbology](https://databento.com/docs/conventions/symbology.md#parent). We will then filter the options by their underlying, to only consider options for the front-month contract. Finally, we will extract prices for these instruments and use the [Black-76 model](https://en.wikipedia.org/wiki/Black_model) to graph implied volatility by strike price for the nearest quarterly E-mini S&P 500 (ES) options expiration. ## Implied volatility Implied Volatility (IV) is a key concept in the field of options trading and financial markets. It represents the market's forecast of a likely movement in a security's price. Rather than reflecting historical price changes, implied volatility looks forward into the market's expectations and is often used as an indicator of market sentiment for options trading. Naturally, a higher implied volatility indicates larger price movements are expected. ## Black-76 model The most famous model for calculating implied volatility is the [Black-Scholes model](https://en.wikipedia.org/wiki/Black%E2%80%93Scholes_model). This model is primarily used for pricing equities options. In this example, we will be calculating the implied volatility using options on futures, so we will use an alternative model called the [Black-76 model](https://en.wikipedia.org/wiki/Black_model). This model is more appropriate for options on futures, bonds, and other interest rate products. Typically, the implied volatility is assumed and is used to calculate the price of an option. In this exercise we will invert this operation, and use historical prices to solve for implied volatility, usually denoted as σ (sigma). We cannot solve for this analytically, so we will use the [scipy](https://pypi.org/project/scipy/) module to find sigma numerically using [scipy.optimize.root\_scalar](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.root_scalar.html). ## Dependencies This example will use the [scipy](https://pypi.org/project/scipy/) module to perform the implied volatility calculation. The [matplotlib](https://pypi.org/project/matplotlib/) package will be used for charting. These dependencies can be installed with the following: ```bash $ pip install scipy matplotlib ``` ## Example ```python from collections.abc import Iterable import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd from matplotlib.ticker import PercentFormatter from scipy.optimize import root_scalar from scipy.stats import norm dataset = "GLBX.MDP3" def black_76( S: float, K: float, T: float, sigma: float, r: float = 0.05, is_call_option: bool = True, ) -> float: """ Calculate an option price from the Black-76 model. Parameters ---------- S : float Current underlying price. K : float Option strike price. T : float Time to expiration in years. sigma : float Implied volatility, annualized. r : float, default 0.05 The risk free interest rate, annualized. is_call_option : bool, default True Flag to indicate the option is a call or put. Returns ------- float """ d1 = (np.log(S / K) + (sigma**2 / 2) * T) / (sigma * np.sqrt(T)) d2 = d1 - sigma * np.sqrt(T) discount_factor = np.exp(-r * T) cp = 1 if is_call_option else -1 return discount_factor * cp * (S * norm.cdf(cp * d1) - K * norm.cdf(cp * d2)) def get_front_month_symbol( parent: str, start: pd.Timestamp, ) -> str: """ Get the symbol of the front month futures contract of a given parent symbol. Parameters ---------- parent : str The parent symbol, such as ES. start : pd.Timestamp The date to obtain the front month symbol for. Returns ------- str """ front_def = client.timeseries.get_range( dataset=db.Dataset.GLBX_MDP3, schema=db.Schema.DEFINITION, symbols=[f"{parent}.c.0"], stype_in=db.SType.CONTINUOUS, start=start.date(), end=(start + pd.Timedelta(days=1)).date(), ).to_df() return front_def.iloc[0]["raw_symbol"] def get_options_chain( parent: str, underlying: str, start: pd.Timestamp, ) -> pd.DataFrame: """ Retrieve the definitions of all options contract of the given parent symbol with for the underlying. Parameters ---------- parent : str The parent symbol, such as ES. underlying : str The underlying contract for the option. start : pd.Timestamp The date to obtain the definitions for. Returns ------- pd.DataFrame """ options_def = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=f"{parent}.OPT", stype_in="parent", start=start.date(), ) df = options_def.to_df() df = df[df["underlying"] == underlying] df = df[df["instrument_class"].isin(("C", "P"))] df["years_to_expiration"] = (df["expiration"] - start).dt.days / 365 return df.sort_values("strike_price") def get_midprice( symbols: Iterable[str], start: pd.Timestamp, end: pd.Timestamp, ) -> pd.Series: """ Get the last top-of-book midprice for one or more symbols between the `start` and `end` times. Parameters ---------- symbols : Iterable[str] A collection of symbols to retrieve the midprices for. start : pd.Timestamp The start time. end : pd.Timestamp The end time (exclusive). Returns ------- pd.Series """ price_df = client.timeseries.get_range( dataset=dataset, schema="mbp-1", symbols=symbols, start=start, end=end, ).to_df() price_df = price_df.groupby("symbol").last() price_df["midprice"] = np.mean(price_df[["bid_px_00", "ask_px_00"]], axis=1) return price_df["midprice"] def find_sigma( row: pd.Series, ) -> float: """ Find the roots of the Black-76 model by varying sigma, implied volatility. This function is for use with `pandas.Dataframe.apply`. Each row should contain a column for "strike_price", "years_to_expiration", "instrument_class", "midprice", and "underlying_price", If the optimization fails, `numpy.nan` is returned. Parameters ---------- row : pd.Series A series of data to process. Returns ------- float | numpy.nan """ def f(sigma: float) -> float: return row["midprice"] - black_76( S=row["underlying_price"], K=row["strike_price"], T=row["years_to_expiration"], sigma=sigma, is_call_option=row["instrument_class"] == "C", ) result = root_scalar(f, x0=0.1, x1=0.5) if result.converged: return result.root print( f"Could not find sigma for {row['raw_symbol']} with midprice {row['midprice']}", ) return np.nan # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Now, define some parameters for the calculation parent_symbol = "ES" start = pd.Timestamp("2024-04-04T17:00:00", tz="US/Central") end = pd.Timestamp("2024-04-04T18:00:00", tz="US/Central") # Get the front month futures contract for the parent symbol front_month_symbol = get_front_month_symbol( parent=parent_symbol, start=start, ) # Get the definitions for all options with the front-month future as the underlying options_chain = get_options_chain( parent=parent_symbol, underlying=front_month_symbol, start=start, ) # Retrieve the midprices for the options and front-month future midprices = get_midprice( symbols=[front_month_symbol, *options_chain["raw_symbol"]], start=start, end=end, ) # Then, join the options definitions with the midprices underlying_price = midprices.loc[front_month_symbol] options_chain = options_chain.join(midprices, on="symbol").dropna(subset="midprice") options_chain["underlying_price"] = underlying_price # Remove options with strike prices far from the underlying price options_chain = options_chain[ abs(options_chain["strike_price"] - underlying_price) <= underlying_price * 0.1 ] # Next, find the implied volatility for each option options_chain["sigma"] = options_chain.apply( find_sigma, axis=1, ) # Separate options into calls and puts call_options = options_chain[options_chain["instrument_class"] == "C"] put_options = options_chain[options_chain["instrument_class"] == "P"] # Finally, plot each curve of implied volatility by strike price plt.axvline( underlying_price, label="underlying price", linestyle=":", ) plt.plot( "strike_price", "sigma", data=call_options, label="call", ) plt.plot( "strike_price", "sigma", data=put_options, label="put", ) plt.title(f"{front_month_symbol} Implied Volatility by Strike - {end}") plt.ylabel("Implied Volatility ($\\sigma$)") plt.xlabel("Strike Price") plt.gca().yaxis.set_major_formatter(PercentFormatter(1)) plt.legend() plt.show() ``` ## Results ![Results](https://databento.com/docs/estimating-implied-volatility-0.svg.md) --- Source: https://databento.com/docs/examples/options/zero-dte-options?lang=python # Get symbols for 0DTE options ## Overview In this example we will show how to get symbols for 0DTE options for a product. 0DTE (Zero **D**ays **T**o **E**xpiration) options are contracts that expire at the end of the current session. ## Definition schema The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) contains instrument definitions and properties. In these examples, we are primarily interested in the `raw_symbol` and `expiration` fields. We will retrieve definition data for all options that trade under a parent symbol. We will then filter for 0DTE contracts based on `expiration`. We can then use `raw_symbol` to pass a list of symbols into another data query. ## Historical example First, we'll take a look at how to do this with the Historical client. We'll use a pandas DataFrame to filter for this example. ```python import datetime import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "OPRA.PILLAR" parent_symbol = "SPXW" # SPX Weekly Options date = datetime.date(2025, 9, 10) # Download definition data for a parent symbol def_data = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=f"{parent_symbol}.OPT", stype_in="parent", start=date, ) # Convert to DataFrame and filter for 0DTE def_df = def_data.to_df() def_df = def_df[def_df["expiration"].dt.date == date] # Print out raw_symbols, expiration, and strike price print(def_df[["raw_symbol", "expiration", "strike_price"]]) # Download some TCBBO data with these symbols tcbbo_data = client.timeseries.get_range( dataset=dataset, schema="tcbbo", symbols=def_df["raw_symbol"].to_list(), start=date, limit=10000, ) # Convert to DataFrame tcbbo_df = tcbbo_data.to_df() # Print TCBBO DataFrame print(tcbbo_df) ``` ## Result ```plaintext raw_symbol expiration strike_price ts_recv 2025-09-10 10:30:02.624717419+00:00 SPXW 250910P04000000 2025-09-10 00:00:00+00:00 4000.0 2025-09-10 10:30:02.634731457+00:00 SPXW 250910P06700000 2025-09-10 00:00:00+00:00 6700.0 2025-09-10 10:30:02.644740147+00:00 SPXW 250910P06200000 2025-09-10 00:00:00+00:00 6200.0 2025-09-10 10:30:02.644740147+00:00 SPXW 250910C05775000 2025-09-10 00:00:00+00:00 5775.0 2025-09-10 10:30:02.662006482+00:00 SPXW 250910P06450000 2025-09-10 00:00:00+00:00 6450.0 ... ... ... ... 2025-09-10 10:30:29.833577055+00:00 SPXW 250910C05970000 2025-09-10 00:00:00+00:00 5970.0 2025-09-10 10:30:29.844558808+00:00 SPXW 250910C06145000 2025-09-10 00:00:00+00:00 6145.0 2025-09-10 10:30:29.869477754+00:00 SPXW 250910C06065000 2025-09-10 00:00:00+00:00 6065.0 2025-09-10 10:30:29.883667299+00:00 SPXW 250910C05960000 2025-09-10 00:00:00+00:00 5960.0 2025-09-10 10:30:29.994986746+00:00 SPXW 250910P06710000 2025-09-10 00:00:00+00:00 6710.0 [406 rows x 3 columns] ``` ```plaintext ts_event rtype publisher_id instrument_id side ... bid_sz_00 ask_sz_00 bid_pb_00 ask_pb_00 symbol ts_recv ... 2025-09-10 13:30:02.764440257+00:00 2025-09-10 13:30:02.764236273+00:00 194 22 1241523583 N ... 0 0 0 0 SPXW 250910P06350000 2025-09-10 13:30:02.764443670+00:00 2025-09-10 13:30:02.764240780+00:00 194 22 1241523583 N ... 0 0 0 0 SPXW 250910P06350000 2025-09-10 13:30:02.764725051+00:00 2025-09-10 13:30:02.764522162+00:00 194 22 1241515705 N ... 0 0 0 0 SPXW 250910P06400000 2025-09-10 13:30:02.764725051+00:00 2025-09-10 13:30:02.764522162+00:00 194 22 1241515705 N ... 0 0 0 0 SPXW 250910P06400000 2025-09-10 13:30:02.764734633+00:00 2025-09-10 13:30:02.764531350+00:00 194 22 1241515705 N ... 0 0 0 0 SPXW 250910P06400000 ... ... ... ... ... ... ... ... ... ... ... ... 2025-09-10 13:31:50.928232746+00:00 2025-09-10 13:31:50.928030554+00:00 194 22 1241531922 N ... 385 364 22 22 SPXW 250910C06610000 2025-09-10 13:31:50.956014355+00:00 2025-09-10 13:31:50.955812019+00:00 194 22 1241520104 N ... 134 200 22 22 SPXW 250910P06495000 2025-09-10 13:31:50.961115681+00:00 2025-09-10 13:31:50.960913601+00:00 194 22 1241528118 N ... 234 200 22 22 SPXW 250910C06555000 2025-09-10 13:31:50.962087120+00:00 2025-09-10 13:31:50.961883488+00:00 194 22 1241516275 N ... 136 60 22 22 SPXW 250910C06545000 2025-09-10 13:31:50.962087120+00:00 2025-09-10 13:31:50.961883488+00:00 194 22 1241519790 N ... 54 188 22 22 SPXW 250910C06550000 [10000 rows x 15 columns] ``` ## Live example Next, we'll take a look at how to do this with the Live client. We'll use the `pretty_expiration` attribute which converts `expiration` into a datetime object to filter. > **Info** > > You will need a live license for `OPRA.PILLAR` to run this example. See our [plans and live data page]($PLATFORM_HOSTNAME/docs/portal/live-data) for more information. ```python import datetime import databento as db # Create a live client live_client = db.Live("$YOUR_API_KEY", compression=db.Compression.ZSTD) # Set parameters dataset = "OPRA.PILLAR" parent_symbol = "SPXW" # SPX Weekly Options date = datetime.datetime.now(tz=datetime.timezone.utc).date() # Subscribe to the definition schema for the parent symbol for this session live_client.subscribe( dataset=dataset, schema="definition", symbols=f"{parent_symbol}.OPT", stype_in="parent", start=0, ) # Create a list of 0DTE symbols symbols = [] def append_symbol_list(msg): if isinstance(msg, db.InstrumentDefMsg): # Filter for 0DTE if msg.pretty_expiration.date() == date: symbols.append(msg.raw_symbol) # Add the callback and start the stream live_client.add_callback(append_symbol_list) live_client.start() # Listen to the stream for 15 seconds to get all messages live_client.block_for_close(timeout=15) # Print out the symbol count and the first 5 symbols print(f"Total symbol count for {dataset} = {len(symbols)}\nFirst 5 symbols...{symbols[:5]}") # Re-subscribe to the TCBBO schema with 0DTE symbols live_client.subscribe( dataset=dataset, schema="tcbbo", symbols=symbols, ) # Add a print callback and start the stream live_client.add_callback(print) live_client.start() # Listen to the stream for 10 seconds live_client.block_for_close(timeout=10) ``` ## Result ```plaintext Total symbol count for OPRA.PILLAR = 139 First 5 symbols...['SPXW 260122P02800000', 'SPXW 260122P08200000', 'SPXW 260122C05800000', 'SPXW 260122C04800000', 'SPXW 260122C03400000'] ``` ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=1174406426, ts_event=1769047672270047890, pretty_ts_event='2026-01-22T02:07:52.270047890Z', stype_in=None, stype_in_symbol='SPXW 260122P02800000', stype_out=None, stype_out_symbol='SPXW 260122P02800000', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') ... Cmbp1Msg(ts_recv=1769027025118777730, pretty_ts_recv='2026-01-21T20:23:45.118777730Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410419, ts_event=1769027025118577558, pretty_ts_event='2026-01-21T20:23:45.118577558Z', action='T', side='N', price=2420000000, pretty_price=2.42, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=2400000000, pretty_bid_px_00=2.4, ask_px_00=2450000000, pretty_ask_px_00=2.45, bid_sz_00=24, ask_sz_00=33, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027025118782732, pretty_ts_recv='2026-01-21T20:23:45.118782732Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410419, ts_event=1769027025118582557, pretty_ts_event='2026-01-21T20:23:45.118582557Z', action='T', side='N', price=2420000000, pretty_price=2.42, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=2400000000, pretty_bid_px_00=2.4, ask_px_00=2450000000, pretty_ask_px_00=2.45, bid_sz_00=24, ask_sz_00=33, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027026311010903, pretty_ts_recv='2026-01-21T20:23:46.311010903Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174415604, ts_event=1769027026310810688, pretty_ts_event='2026-01-21T20:23:46.310810688Z', action='T', side='N', price=19140000000, pretty_price=19.14, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=19000000000, pretty_bid_px_00=19, ask_px_00=19300000000, pretty_ask_px_00=19.3, bid_sz_00=19, ask_sz_00=28, bid_pb_00=22, ask_pb_00=22) ... Cmbp1Msg(ts_recv=1769027042926074875, pretty_ts_recv='2026-01-21T20:24:02.926074875Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410087, ts_event=1769027042925867805, pretty_ts_event='2026-01-21T20:24:02.925867805Z', action='T', side='N', price=10720000000, pretty_price=10.72, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=10600000000, pretty_bid_px_00=10.6, ask_px_00=10800000000, pretty_ask_px_00=10.8, bid_sz_00=13, ask_sz_00=11, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027043433405443, pretty_ts_recv='2026-01-21T20:24:03.433405443Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410087, ts_event=1769027043433197884, pretty_ts_event='2026-01-21T20:24:03.433197884Z', action='T', side='N', price=10700000000, pretty_price=10.7, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=10600000000, pretty_bid_px_00=10.6, ask_px_00=10800000000, pretty_ask_px_00=10.8, bid_sz_00=14, ask_sz_00=26, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027043433405682, pretty_ts_recv='2026-01-21T20:24:03.433405682Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410087, ts_event=1769027043433202613, pretty_ts_event='2026-01-21T20:24:03.433202613Z', action='T', side='N', price=10700000000, pretty_price=10.7, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=10600000000, pretty_bid_px_00=10.6, ask_px_00=10800000000, pretty_ask_px_00=10.8, bid_sz_00=14, ask_sz_00=26, bid_pb_00=22, ask_pb_00=22) ``` --- Source: https://databento.com/docs/examples/options/equity-open-interest?lang=python # Daily statistics for equity options ## Overview In this example, we'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate volume and open interest statistics for an equity options chain. First, we'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties, to get information such as `expiration`, `strike_price`, and `instrument_class`. Next, we'll use the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md) to get the total volume for each option. Finally, we'll use the [statistics schema](https://databento.com/docs/schemas/statistics.md) to request open interest statistics. OPRA publishes start-of-day open interest statistics before the regular trading hours (RTH) open at 9:30 ET. ## Example ```python import datetime as dt from zoneinfo import ZoneInfo import databento as db import matplotlib.pyplot as plt import pandas as pd from matplotlib.patches import Patch from matplotlib.ticker import MaxNLocator def get_volume_data( client: db.Historical, dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """Get volume data for all options for a given symbol.""" volume_df = client.timeseries.get_range( dataset=dataset, symbols=f"{symbol}.OPT", schema="ohlcv-1d", stype_in="parent", start=date, ).to_df() return volume_df.groupby("symbol")["volume"].sum().reset_index() def get_oi_data( client: db.Historical, dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """Get open interest data for all options for a given symbol.""" # Start-of-day open interest is published before the RTH open at 9:30 ET end_time = dt.time(9, 30, 0, tzinfo=ZoneInfo("America/New_York")) stats_df = client.timeseries.get_range( dataset=dataset, symbols=f"{symbol}.OPT", schema="statistics", stype_in="parent", start=date, end=dt.datetime.combine(date, end_time), ).to_df() stats_df = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST] stats_df = stats_df.drop_duplicates("symbol", keep="last") stats_df["open_interest"] = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST][ "quantity" ] return stats_df[["open_interest", "symbol"]] def get_definition_data( client: db.Historical, dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """Get definition data for all options for a given symbol.""" def_df = client.timeseries.get_range( dataset=dataset, symbols=f"{symbol}.OPT", schema="definition", stype_in="parent", start=date, ).to_df() def_df["days_to_expiration"] = (def_df["expiration"] - def_df.index.normalize()).dt.days def_df["expiration"] = def_df["expiration"].dt.normalize().dt.date return def_df[ ["symbol", "strike_price", "instrument_class", "expiration", "days_to_expiration"] ] def plot_oi_by_strike( df: pd.DataFrame, expiration_date: dt.date, ) -> None: df = df[df["expiration"] == expiration_date] trade_date = df["trade_date"].iloc[0] days_to_expiration = df["days_to_expiration"].iloc[0] df_strikes = df.groupby(["strike_price", "instrument_class"])["open_interest"].sum().unstack() # Plot bar chart _, ax = plt.subplots(figsize=(14, 6)) df_strikes.plot( ax=ax, kind="bar", xlabel="Strike price", ylabel="Open interest", ) # Adjust X ticks and labels ax.xaxis.set_major_locator(MaxNLocator(15)) ax.set_xticklabels(ax.get_xticklabels(), rotation=45, ha="right") # Set title and legend ax.set_title(f"{symbol} open interest \n {trade_date} ({days_to_expiration} DTE)") ax.legend(handles=[Patch(facecolor="C0", label="Call"), Patch(facecolor="C1", label="Put")]) plt.tight_layout() plt.show() # Set parameters dataset = "OPRA.PILLAR" symbol = "NVDA" start = dt.date(2025, 8, 19) # Create a historical client client = db.Historical("$YOUR_API_KEY") # Get strike price, expiration, and instrument class in the definition schema def_df = get_definition_data(client, dataset, symbol, start) # Volume from the OHLCV-1d schema volume_df = get_volume_data(client, dataset, symbol, start) # Start-of-day open interest is published in the morning in the statistics schema stats_df = get_oi_data(client, dataset, symbol, start) # Merge different schemas together df = def_df.merge(volume_df, on="symbol", how="left") df = df.merge(stats_df, on="symbol", how="left") df["trade_date"] = start df["volume"] = df["volume"].fillna(0).astype(int) df["open_interest"] = df["open_interest"].fillna(0).astype(int) # Print out top 10 contracts by open interest print(df.nlargest(10, "open_interest").to_string(index=False)) # Plot open interest by expiration date plot_oi_by_strike(df, dt.date(2025, 8, 22)) ``` ## Result You could use the resulting data to filter for the contracts with the highest open interest. Additionally, you could plot the open interest distribution for a given expiration. ```plaintext symbol strike_price instrument_class expiration days_to_expiration volume open_interest trade_date NVDA 250919C00160000 160.0 C 2025-09-19 31 52206 173864 2025-08-19 NVDA 260618P00075000 75.0 P 2026-06-18 303 1 109796 2025-08-19 NVDA 250919C00170000 170.0 C 2025-09-19 31 4636 106760 2025-08-19 NVDA 250822C00185000 185.0 C 2025-08-22 3 158486 102111 2025-08-19 NVDA 250919C00185000 185.0 C 2025-09-19 31 61860 101843 2025-08-19 NVDA 250919C00190000 190.0 C 2025-09-19 31 59766 92379 2025-08-19 NVDA 250822C00190000 190.0 C 2025-08-22 3 46489 91666 2025-08-19 NVDA 250919C00165000 165.0 C 2025-09-19 31 759 90851 2025-08-19 NVDA 250919C00175000 175.0 C 2025-09-19 31 5862 85548 2025-08-19 NVDA 260116C00100000 100.0 C 2026-01-16 150 161 85414 2025-08-19 ``` ![Options venues volume](https://databento.com/docs/equity-open-interest-0.svg.md) --- Source: https://databento.com/docs/examples/options/option-spreads?lang=python # Get end of day option spreads ## Overview In this example, we'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to get the end of day spread for an options chain. First, we'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties, to get information such as `expiration`, `strike_price`, and `instrument_class`. We'll also use the [CBBO-1m schema](https://databento.com/docs/schemas/bbo.md#consolidated-bbo-on-interval-cbbo) to get the bid/ask spread for each option. We'll plot the end-of-day midprice and spread in basis-points (BPS) across all strikes for multiple expirations. ## Example ```python import datetime as dt from zoneinfo import ZoneInfo import databento as db import matplotlib.pyplot as plt import pandas as pd def get_options_data( client: db.Historical, symbol: str, schema: str, start: dt.datetime, end: dt.datetime | None = None, ) -> pd.DataFrame: """Fetch options data between timestamps""" return client.timeseries.get_range( dataset="OPRA.PILLAR", schema=schema, stype_in="parent", symbols=f"{symbol}.OPT", start=start, end=end, ).to_df() def combine_data( trade_date: dt.date, df_def: pd.DataFrame, df_close: pd.DataFrame, ) -> pd.DataFrame: """Combine open and close data with proper column naming""" df_close = df_close.groupby("symbol")[["bid_px_00", "ask_px_00"]].last() df_def = df_def[["symbol", "instrument_class", "expiration", "strike_price"]] df_def["dte"] = (df_def["expiration"].dt.tz_localize(None) - pd.Timestamp(trade_date)).dt.days df_def = df_def.set_index("symbol") df = df_close.join(df_def, on="symbol", how="left").reset_index().sort_values("symbol") # Calculate spreads in basis points df["mid_px"] = (df["bid_px_00"] + df["ask_px_00"]) / 2 df["spread"] = df["ask_px_00"] - df["bid_px_00"] df["spread_bps"] = (df["spread"] / df["mid_px"]) / 1e-4 df["trade_date"] = trade_date return df def plot_spreads( symbol: str, df: pd.DataFrame, target_dtes: tuple[int], ) -> None: """Plot bid-ask spreads for selected expirations""" fig, axes = plt.subplots(len(target_dtes), 1, figsize=(12, 12)) selected_expirations = [ df.loc[(df["dte"] - t).abs().idxmin(), "expiration"] for t in target_dtes ] subset = df[df["expiration"].isin(selected_expirations)] y_min = subset["mid_px"].min() y_max = subset["mid_px"].max() y_range = y_max - y_min x_min = subset["strike_price"].min() x_max = subset["strike_price"].max() x_padding = (x_max - x_min) * 0.02 y_padding = y_range * 0.1 # 10% of y_range = 1000 bps y_spread_scale = (y_range * 0.10) / 1000 for idx, (ax, exp) in enumerate(zip(axes, selected_expirations)): df_exp = df[df["expiration"] == exp].copy() # Scale spreads for visualization df_exp["scaled_spread"] = df_exp["spread_bps"].apply(lambda x: x * y_spread_scale) def plot_options( instrument_class: db.InstrumentClass, color: str, ) -> None: data = df_exp[df_exp["instrument_class"] == instrument_class].sort_values( "strike_price", ) if len(data) > 0: half = data["scaled_spread"] / 2 ax.vlines( x=data["strike_price"], ymin=data["mid_px"] - half, ymax=data["mid_px"] + half, colors=color, linewidth=2, alpha=0.7, label=instrument_class.name.capitalize(), ) for y_offset in (half, -half): ax.hlines( y=data["mid_px"] + y_offset, xmin=data["strike_price"] - 0.5, xmax=data["strike_price"] + 0.5, colors=color, linewidth=1.5, alpha=0.7, ) ax.plot(data["strike_price"], data["mid_px"], "s", color=color, markersize=3) plot_options(db.InstrumentClass.CALL, "C0") plot_options(db.InstrumentClass.PUT, "C1") ref_x = x_max - (x_max - x_min) * 0.05 ref_y_base = y_min * 0.3 s1000 = y_spread_scale * 1000 ax.plot([ref_x, ref_x], [ref_y_base, ref_y_base + s1000], "k-", linewidth=2) for y_off, label in [(0, "0 bps"), (s1000, "1,000 bps")]: ax.hlines( y=ref_y_base + y_off, xmin=ref_x - 1, xmax=ref_x + 1, colors="black", linewidth=1.5, ) ax.text( ref_x - 3, ref_y_base + y_off, label, fontsize=8, va="center", ha="right", ) ax.annotate( f"DTE = {df_exp['dte'].iloc[0]} days", xy=(0.02, 0.98), xycoords="axes fraction", fontsize=11, ha="left", va="top", bbox=dict(boxstyle="round,pad=0.3", fc="white", ec="gray", alpha=0.8), ) ax.set_xlim(x_min - x_padding, x_max + x_padding) ax.set_ylim(y_min - y_padding, y_max + y_padding) ax.set_ylabel("Option Price ($)", fontsize=11) ax.grid(True, alpha=0.3) ax.legend(loc="upper right") fig.suptitle(f"{symbol} Options End of Day NBBO ({df['trade_date'].iloc[0]})") fig.supxlabel("Strike Price ($)", fontsize=11) plt.tight_layout() plt.show() symbol = "AAPL" start_date = dt.date(2025, 11, 19) close_time = dt.time(16, 0, 0, tzinfo=ZoneInfo("US/Eastern")) close_dt = dt.datetime.combine(start_date, close_time) client = db.Historical("$YOUR_API_KEY") df_close = get_options_data( client, symbol, "cbbo-1m", close_dt - dt.timedelta(minutes=15), close_dt, ) df_definition = get_options_data(client, symbol, "definition", start_date) df = combine_data(start_date, df_definition, df_close) print(df) # Plot spreads for targeted DTEs plot_spreads(symbol, df, target_dtes=(0, 30, 60)) ``` ## Result ```plaintext symbol bid_px_00 ask_px_00 instrument_class expiration ... dte mid_px spread spread_bps trade_date 0 AAPL 251121C00100000 167.30 168.55 C 2025-11-21 00:00:00+00:00 ... 2 167.925 1.25 74.437993 2025-11-19 1 AAPL 251121C00105000 162.30 165.25 C 2025-11-21 00:00:00+00:00 ... 2 163.775 2.95 180.125172 2025-11-19 2 AAPL 251121C00110000 157.25 160.25 C 2025-11-21 00:00:00+00:00 ... 2 158.750 3.00 188.976378 2025-11-19 3 AAPL 251121C00115000 152.25 155.25 C 2025-11-21 00:00:00+00:00 ... 2 153.750 3.00 195.121951 2025-11-19 4 AAPL 251121C00120000 147.25 150.25 C 2025-11-21 00:00:00+00:00 ... 2 148.750 3.00 201.680672 2025-11-19 ... ... ... ... ... ... ... ... ... ... ... ... 2690 AAPL 280121P00490000 219.00 224.00 P 2028-01-21 00:00:00+00:00 ... 793 221.500 5.00 225.733634 2025-11-19 2691 AAPL 280121P00500000 229.00 234.00 P 2028-01-21 00:00:00+00:00 ... 793 231.500 5.00 215.982721 2025-11-19 2692 AAPL 280121P00510000 239.00 244.00 P 2028-01-21 00:00:00+00:00 ... 793 241.500 5.00 207.039337 2025-11-19 2693 AAPL 280121P00520000 249.00 254.00 P 2028-01-21 00:00:00+00:00 ... 793 251.500 5.00 198.807157 2025-11-19 2694 AAPL 280121P00530000 259.00 264.00 P 2028-01-21 00:00:00+00:00 ... 793 261.500 5.00 191.204589 2025-11-19 [2695 rows x 11 columns] ``` ![Options venues volume](https://databento.com/docs/option-spreads-0.svg.md) --- Source: https://databento.com/docs/examples/basics-live?lang=python ## Live data | Title | Subtitle | Supported languages | | [Handle multiple record types](https://databento.com/docs/examples/basics-live/live-dispatch.md) | Handle different record types using callbacks. | Python, C++, Rust, Live | | [Stream live data to a file](https://databento.com/docs/examples/basics-live/live-stream-to-file.md) | Write and save live data to a file. | Python, C++, Rust, Live | | [Estimate Databento feed latency](https://databento.com/docs/examples/basics-live/live-latency.md) | Get time between our servers and your real-time application. | Python, C++, Rust, Live | | [Calculate TICK and TRIN indicators](https://databento.com/docs/examples/basics-live/live-tick-trin.md) | Compute market TICK and TRading INdex on real-time trades. | Python, C++, Rust, Live | | [Subscribe to MBO snapshot](https://databento.com/docs/examples/basics-live/live-snapshot.md) | Get the latest order book state without replaying the entire trading session. | Python, C++, Rust, Live | | [Compare on-exchange and off-exchange trade volume](https://databento.com/docs/examples/basics-live/live-nasdaq-trf.md) | Compare the volumes of the different venues of Nasdaq Basic with NLS Plus. | Python, Live | --- Source: https://databento.com/docs/examples/basics-live/live-dispatch?lang=python # Handle multiple record types ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to process order book data using [a callback](https://databento.com/docs/api-live/client/add-callback.md) function. We will use Python's [functools.singledispatch](https://docs.python.org/3/library/functools.html#functools.singledispatch) to make handling different record types easier. ## MBO schema We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) in this example but any schema's records can be dispatched this way. ## Example ```python from functools import singledispatch import databento as db # First, we will create our callback function using singledispatch @singledispatch def dbn_handler(_: db.DBNRecord): pass @dbn_handler.register def _(sym: db.SymbolMappingMsg): instrument_id = sym.instrument_id symbol = sym.stype_in_symbol print(f"{symbol} has an instrument ID of {instrument_id}") @dbn_handler.register def _(mbo: db.MBOMsg): order_id = f"{mbo.order_id:13d}" side = mbo.side size = mbo.size price = f"${mbo.pretty_price:,.06f}" print(order_id, side, size, price, sep="\t") @dbn_handler.register def _(sys: db.SystemMsg): print(f"{sys.msg}") @dbn_handler.register def _(error: db.ErrorMsg): print(f"ERROR: {error.err}") # Then, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the MBO schema live_client.subscribe( dataset="GLBX.MDP3", schema="mbo", stype_in="continuous", symbols="ZN.c.0", ) # Now, add our callback function and start the streaming session live_client.add_callback(dbn_handler) live_client.start() # Finally, we will wait for 5 seconds before closing the connection live_client.block_for_close(timeout=5) ``` ## Result ```plaintext Subscription request 1 for mbo data succeeded ZN.c.0 has an instrument ID of 42066706 8416556628429 A 4 $112.062500 8416556628472 A 3 $112.046875 8416556625196 B 1 $112.046875 8416556625202 B 1 $112.046875 8416556625204 B 1 $112.046875 8416556625196 B 1 $112.046875 8416556625202 B 1 $112.046875 8416556625204 B 1 $112.046875 8416556628476 A 1 $112.062500 8416556628477 A 1 $112.062500 8416556628479 A 1 $112.062500 8416556628482 A 1 $112.125000 8416556628483 A 1 $112.125000 8416556628484 B 1 $112.046875 8416556628128 B 5 $112.046875 8416556628379 B 4 $112.046875 ... ``` --- Source: https://databento.com/docs/examples/basics-live/live-stream-to-file?lang=python # Stream live data to a file ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to save live data to a file using an [output stream](https://databento.com/docs/api-live/client/add-stream.md). We will then open this file as a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md). This allows for live data to be archived for replay and testing purposes. ## Databento Binary Encoding (DBN) The binary data will be written to the stream in the [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md) format. This is a zero-copy format that is extremely fast and efficient on bandwidth. The easiest way to read data encoded in DBN is through our client libraries or the [dbn-cli](https://crates.io/crates/dbn-cli) tool. ## Example ```python import databento as db # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the OHLCV-1s schema for a few symbols live_client.subscribe( dataset="GLBX.MDP3", schema="ohlcv-1s", stype_in="continuous", symbols=["ES.c.0", "ES.c.1", "CL.c.0", "CL.c.1"], ) # Now, we will open a file for writing and start streaming live_client.add_stream("example.dbn") live_client.start() # We will listen for 10 seconds before closing the connection # Any ohlcv-1s bars received during this time will be saved live_client.block_for_close(timeout=10) # Finally, we will open the DBN file dbn_store = db.DBNStore.from_file("example.dbn") print(dbn_store.to_df(schema="ohlcv-1s")) ``` ## Result ```plaintext rtype publisher_id instrument_id open high low close volume symbol ts_event 2023-12-18 14:40:19+00:00 32 1 17077 4785.75 4785.75 4785.75 4785.75 143 ESH4 2023-12-18 14:40:19+00:00 32 1 686071 74.19 74.19 74.19 74.19 6 CLG4 2023-12-18 14:40:20+00:00 32 1 17077 4785.75 4785.75 4785.50 4785.75 8 ESH4 2023-12-18 14:40:20+00:00 32 1 686071 74.19 74.20 74.19 74.20 6 CLG4 2023-12-18 14:40:21+00:00 32 1 17077 4785.75 4785.75 4785.75 4785.75 105 ESH4 2023-12-18 14:40:21+00:00 32 1 686071 74.20 74.21 74.20 74.21 8 CLG4 ``` --- Source: https://databento.com/docs/examples/basics-live/live-latency?lang=python # Estimate Databento feed latency ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to calculate latency of incoming records. We will use the `ts_recv` field to make a measurement of the time between Databento's servers receiving a message from the exchange, to the message being received by the client. To do this, we will request [MBP-1](https://databento.com/docs/schemas/mbp-1.md) data for a [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) and calculate the latency for each message. You can read our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md) to learn more about our architecture. Latency results will depend upon the quality of your system clock and internet connection. Your individual results might be very large or even negative given these factors. To obtain the most accurate results, we suggest you sync your clocks to our [NTP service](https://databento.com/docs/architecture/ntp-service.md). ## Example ```python import time import databento as db import matplotlib.pyplot as plt import numpy as np # Set parameters dataset = "GLBX.MDP3" product = "ES" num_samples = 5000 percentiles = [50, 90, 95, 99] num_bins = 100 # Create a live client live_client = db.Live(key="$YOUR_API_KEY") # Subscribe to MBP-1 data live_client.subscribe( dataset="GLBX.MDP3", schema="mbp-1", symbols=f"{product}.FUT", stype_in="parent", ) # Iterate over the live client and calculate latency latency_ms = [] for record in live_client: if isinstance(record, db.MBP1Msg): delta = time.time_ns() - record.ts_recv latency_ms.append(delta / 1e6) if len(latency_ms) >= num_samples: break fig, ax = plt.subplots(figsize=(10, 6)) x_min = min(latency_ms) * 0.95 x_max = max(latency_ms) * 1.05 # Create histogram with log bins log_bins = np.logspace(np.log10(x_min), np.log10(x_max), num_bins) n, bins, patches = ax.hist(latency_ms, bins=log_bins, rwidth=0.85, color="C3") y_max = max(n) * 1.25 percentile_handles = [] percentile_labels = [] # Create percentile lines for p in percentiles: p_value = np.percentile(latency_ms, p) line = ax.axvline(p_value, color="C0", linestyle="--", linewidth=1) ax.text(p_value, y_max * 0.9, f"{p}th", color="C0", rotation=-90) percentile_handles.append(line) percentile_labels.append(f"p{p} - {round(p_value, 2)} ms") # Set log scale and labels tick_locations = np.logspace(np.log10(x_min), np.log10(x_max), 10) ax.set_xscale("log") ax.set_xticks(tick_locations) ax.set_xticklabels([str(round(x)) for x in tick_locations]) ax.tick_params(axis="x", which="minor", bottom=False, labelbottom=False) # Set limits and labels ax.set_xlim(x_min * 0.95, x_max * 1.05) ax.set_ylim(0, y_max) ax.set_xlabel("Latency (ms)", fontsize=12) ax.set_ylabel("Frequency", fontsize=12) ax.set_title("Latency Distribution", fontsize=14) # Add legend and display ax.legend(handles=percentile_handles, labels=percentile_labels) plt.tight_layout() plt.show() ``` ![Latency plot](https://databento.com/docs/live-latency-0.svg.md) --- Source: https://databento.com/docs/examples/basics-live/live-tick-trin?lang=python # Calculate TICK and TRIN indicators ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to calculate two technical indicators: market tick (TICK) and the traders index (TRIN). ## Trades schema These indicators are updated on every execution for all symbols on the venue. To get this, we will use the [trades schema](https://databento.com/docs/schemas/trades.md), which provides records for every trade execution. ## Example We can pass in `symbols="ALL_SYMBOLS"` to request all symbols of a dataset at once. ```python import databento as db import numpy as np # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to trades for all symbols live_client.subscribe( dataset="GLBX.MDP3", schema="trades", symbols="ALL_SYMBOLS", ) # Then, we setup our initial values for TICK/TRIN uptick = downtick = upvol = downvol = 0 # We will consume 20 records and then close for this example trades = 0 for record in live_client: if isinstance(record, db.TradeMsg): side = record.side if side == "A": downtick += 1 downvol += record.size elif side == "B": uptick += 1 upvol += record.size tick = uptick / downtick if downtick != 0 else np.nan trin_base = upvol / downvol if downvol != 0 else np.nan trin = tick / trin_base if trin_base != 0 else np.nan print(f"{tick:.4f}", f"{trin:.4f}") trades += 1 if trades >= 20: break elif isinstance(record, db.ErrorMsg): print(f"Error: {record.err}") ``` ## Result ```plaintext 1.0000 1.0000 0.5000 1.0000 0.3333 1.0000 0.2500 1.0000 0.2000 2.2000 0.1667 2.0000 0.1429 1.8571 0.1250 1.8750 0.1111 2.1111 0.2222 2.1111 0.3333 1.5833 0.4444 1.0556 0.4000 1.0000 0.5000 1.0000 0.6000 1.0909 0.5455 1.1405 0.5000 1.0909 0.4615 1.0909 0.4286 1.0909 ``` --- Source: https://databento.com/docs/examples/basics-live/live-snapshot?lang=python # Subscribe to MBO snapshot ## Overview This example shows how to use the [Live client](https://databento.com/docs/api-live/client/live.md) to request a snapshot for [MBO schema](https://databento.com/docs/schemas/mbo.md), which can be used to reconstruct the [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) without replaying the entire trading session. More details about live snapshot can be found in this [article](https://databento.com/docs/conventions/mbo-snapshot.md). ## Requesting live snapshot To request a subscription with snapshot, we provide `snapshot=True` and leave `start` default to `None`. Once the session has started, it streams all the snapshot records (with the `F_SNAPSHOT` [flag](https://databento.com/docs/conventions/common-fields-enums-types.md#flags) set), followed by real-time records. We stop iterating over the records when encountering a record with the last flag set, to make sure the book is in a valid state. ## Example ```python import databento as db # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the MBO snapshot live_client.subscribe( dataset="GLBX.MDP3", schema="mbo", stype_in="continuous", symbols="ES.c.0", snapshot=True, ) # Finally, we will process all snapshot records, and stop at the first record # with F_LAST flag, which indicates that the snapshot is complete and the # order book is in a valid state for record in live_client: if isinstance(record, db.MBOMsg): if record.flags & db.RecordFlags.F_SNAPSHOT: print(f"Snapshot: {record}") else: print(f"Live: {record}") if record.flags & db.RecordFlags.F_LAST: print("Snapshot is complete") break elif isinstance(record, db.ErrorMsg): print(f"Error {record.err}") break ``` ## Result ```plaintext Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1769043341317912993, pretty_ts_event='2026-01-22T00:55:41.317912993Z', action='R', side='N', price=9223372036854775807, pretty_price=NaN, size=0, channel_id=0, order_id=0, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=0) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6811250000000, pretty_price=6811.25, size=1, channel_id=0, order_id=6416171004846, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=761) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6810000000000, pretty_price=6810, size=2, channel_id=0, order_id=6416138613427, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6805000000000, pretty_price=6805, size=2, channel_id=0, order_id=6416108605777, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=756) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=50, channel_id=0, order_id=6416091817328, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=754) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=10, channel_id=0, order_id=6416102672407, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=755) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=5, channel_id=0, order_id=6416117277784, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=756) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=20, channel_id=0, order_id=6416141452066, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=758) ... Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6795000000000, pretty_price=6795, size=25, channel_id=0, order_id=6416122261431, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6791000000000, pretty_price=6791, size=1, channel_id=0, order_id=6416120339416, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6780500000000, pretty_price=6780.5, size=1, channel_id=0, order_id=6416121479238, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6775000000000, pretty_price=6775, size=1, channel_id=0, order_id=6416127400070, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) ``` --- Source: https://databento.com/docs/examples/basics-live/live-nasdaq-trf?lang=python # Compare on-exchange and off-exchange trade volume ## Overview In this example, we will stream live data and compare the volume of trades on the different venues that compose the [Nasdaq Basic with NLS Plus](https://databento.com/docs/datasets/xnas-basic.md) (`XNAS.BASIC`) dataset. The following venues are included in this dataset: - Nasdaq (XNAS) - Nasdaq Texas (XBOS) - Nasdaq PSX (XPSX) - FINRA/Nasdaq TRF Carteret (FINN) - FINRA/Nasdaq TRF Chicago (FINC) The [FINRA/Nasdaq TRFs](https://www.nasdaq.com/solutions/post-trade-reporting) aggregate data from most of the 30 ATSs and account for approximately 45% to 49% of the average daily volume (ADV) in all exchange-listed securities. As of January 2025, `XNAS.BASIC` represented about 63% of ADV, which includes both on-exchange and off-exchange trades. ## Trades schema We will use the [trades schema](https://databento.com/docs/schemas/trades.md) for this example. [NLS Plus](https://data.nasdaq.com/databases/NLS) reports the size, price, and exchange/TRF for every trade. The trade aggressor side is not available on this dataset, so the `side` field will always be set to `N`. The `side` field is populated on our depth-of-book feeds, such as [Nasdaq TotalView-ITCH](https://databento.com/docs/datasets/xnas-itch.md) (`XNAS.ITCH`). > **Info** > > Real-time `XNAS.BASIC` data is included with a Plus or Unlimited subscription through our Databento US Equities service. > Visit our [pricing page]($PLATFORM_HOSTNAME/pricing/) for more information. ## Example ```python import databento as db # First, create the respective clients historical_client = db.Historical(key="$YOUR_API_KEY") live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the trades schema for the designated symbol symbol = "TSLA" live_client.subscribe( dataset="XNAS.BASIC", schema="trades", symbols=symbol, ) # We will stream live data to file for further use live_client.add_stream(f"{symbol}_trades.dbn") live_client.start() # Let the connection run for 15 seconds. All trades in this time will be saved # to the file above live_client.block_for_close(timeout=15) # Now we will open the file for our analysis and create a Pandas DataFrame dbn_store = db.DBNStore.from_file(f"{symbol}_trades.dbn") df = dbn_store.to_df(schema="trades") # Calculate statistics by venue venue_df = df.groupby("publisher_id", as_index=False)["size"].agg(["sum", "mean"]) venue_df["mean"] = venue_df["mean"].round().astype(int) total_volume = venue_df["sum"].sum() venue_df["percentage"] = (venue_df["sum"] / total_volume * 100).round(2) # Clean up for readability publishers = historical_client.metadata.list_publishers() publisher_venues = {x["publisher_id"]: x["description"].partition("-")[2] for x in publishers} venue_df["publisher_id"] = venue_df["publisher_id"].map(publisher_venues) venue_df = venue_df.sort_values("sum", ascending=False) venue_df.columns = ["Venue", "Total Volume", "Average Size", "% of Dataset Total"] print(venue_df) ``` ## Result > **Info** > > If you do not see any output, it could be because the markets are closed. See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) for using [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext Venue Total Volume Average Size % of Dataset Total 1 FINRA/Nasdaq TRF Carteret 86591 56 68.93 0 Nasdaq 36218 48 28.83 2 FINRA/Nasdaq TRF Chicago 2408 51 1.92 3 Nasdaq BX 348 58 0.28 4 Nasdaq PSX 57 11 0.05 ``` --- Source: https://databento.com/docs/examples/basics-historical?lang=python ## Historical data | Title | Subtitle | Supported languages | | [Programmatic batch downloads](https://databento.com/docs/examples/basics-historical/programmatic-batch-download.md) | Request and download batch files programmatically. | Python, C++, Rust, Historical | | [Best bid, best offer, and midprice](https://databento.com/docs/examples/basics-historical/midprice.md) | Calculate midprice using BBO from MBP-1 or TBBO data. | Python, C++, Rust, Historical | | [Custom OHLCV bars from trades](https://databento.com/docs/examples/basics-historical/custom-ohlcv.md) | Resample trades data to construct custom OHLCV bars. | Python, Rust, Historical | | [Join schemas on instrument ID](https://databento.com/docs/examples/basics-historical/joining-schemas.md) | Augment price data with instrument definitions by joining on instrument ID. | Python, Historical | | [Plot a candlestick chart](https://databento.com/docs/examples/basics-historical/candlestick-plot.md) | Visualize OHLCV data on a candlestick chart. | Python, Historical | | [Calculate VWAP and RSI](https://databento.com/docs/examples/basics-historical/technical-indicators.md) | Compute the volume-weighted average price and relative strength indicator. | Python, Historical | | [End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md) | Get mark-to-market valuation of a portfolio at end of day. | Python, C++, Rust, Historical | | [Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md) | Calculate metrics like risk-adjusted return and Sharpe ratio of a portfolio. | Python, Historical | | [Market halts, volatility interrupts, and price bands](https://databento.com/docs/examples/basics-historical/halts.md) | Handle trading halts using status messages. | Python, Historical | | [Resample OHLCV from 1-minute to 5-minute](https://databento.com/docs/examples/basics-historical/ohlcv-resampling.md) | Forward-fill gaps from intervals with no trades, then resample to 5-minute bars. | Python, Historical | | [Convert DBN to other encoding formats](https://databento.com/docs/examples/basics-historical/encodings.md) | Request data in DBN format and convert to CSV, JSON, or Parquet. | Python, Historical | | [Request a large number of symbols](https://databento.com/docs/examples/basics-historical/requesting.md) | Get multiple symbols or a firehose of all symbols traded on a venue. | Python, Rust, Historical | --- Source: https://databento.com/docs/examples/basics-historical/programmatic-batch-download?lang=python # Programmatic batch downloads ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to submit a batch request and download files programmatically. To do this we will request five days of trade data, wait for our request to process, and then download the files locally and load them into a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. > **See also** > > [Streaming vs. batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md). ## Example ```python import pathlib import time import databento as db # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will submit a batch job new_job = client.batch.submit_job( dataset="GLBX.MDP3", start="2022-12-12T00:00:00", end="2022-12-17T00:00:00", symbols="OZN.OPT", schema="trades", split_duration="day", stype_in="parent", ) # Retrieve the new job ID new_job_id: str = new_job["id"] # Now, we have to wait for our batch job to complete while True: job_details = client.batch.get_job_details(new_job_id) if job_details["state"] == "done": break # Exit the loop to continue time.sleep(1.0) # Once complete, we will download the files downloaded_files = client.batch.download( job_id=new_job_id, output_dir=pathlib.Path.cwd(), ) # Finally, we can load the data into a DBNStore for analysis for file in sorted(downloaded_files): if file.name.endswith(".dbn.zst"): data = db.DBNStore.from_file(file) # Convert the data to a pandas.DataFrame df = data.to_df() print(f"{file.name} contains {len(df):,d} records") ``` ## Result ```plaintext glbx-mdp3-20221212.trades.dbn.zst contains 2,055 records glbx-mdp3-20221213.trades.dbn.zst contains 4,112 records glbx-mdp3-20221214.trades.dbn.zst contains 2,936 records glbx-mdp3-20221215.trades.dbn.zst contains 3,034 records glbx-mdp3-20221216.trades.dbn.zst contains 2,540 records ``` --- Source: https://databento.com/docs/examples/basics-historical/midprice?lang=python # Best bid, best offer, and midprice ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve the best bid and offer (BBO) and use these data to calculate the midprice. To do this we will request a subset of the entire order book data and then average the bid and ask prices to calculate the midprice. Note that _ask_ and _offer_ are synonyms referring to the sell price. ## TBBO schema We'll use the [TBBO schema](https://databento.com/docs/schemas/tbbo.md) to demonstrate this. This schema contains every trade execution event together with the best bid and offer immediately before the trade. ## Example ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request TBBO data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2022-10-28T20:30:00", end="2022-10-28T21:00:00", symbols="ESH3", stype_in="raw_symbol", schema="tbbo", ) # Convert to DataFrame tbbo_data = data.to_df() # Calculate the midprice for each row tbbo_data["mid_px"] = tbbo_data[["ask_px_00", "bid_px_00"]].mean(axis=1) # We now have a DataFrame that contains execution price, BBO, and midprice print(tbbo_data[["symbol", "side", "price", "ask_px_00", "mid_px", "bid_px_00"]]) ``` ## Result ```plaintext symbol side price ask_px_00 mid_px bid_px_00 ts_recv 2022-10-28 20:30:59.047138053+00:00 ESH3 B 3955.25 3955.25 3955.000 3954.75 2022-10-28 20:37:53.112494436+00:00 ESH3 A 3955.00 3955.75 3955.375 3955.00 2022-10-28 20:39:05.152071481+00:00 ESH3 A 3955.00 3955.75 3955.375 3955.00 2022-10-28 20:51:37.650473254+00:00 ESH3 B 3955.50 3955.50 3955.125 3954.75 2022-10-28 20:52:36.842346377+00:00 ESH3 A 3956.00 3956.25 3956.125 3956.00 2022-10-28 20:52:57.780797186+00:00 ESH3 B 3955.00 3955.00 3954.750 3954.50 2022-10-28 20:57:28.474236098+00:00 ESH3 A 3954.75 3956.25 3955.500 3954.75 2022-10-28 20:59:15.075191111+00:00 ESH3 A 3953.75 3956.00 3954.875 3953.75 2022-10-28 20:59:34.607239899+00:00 ESH3 A 3954.50 3956.00 3955.250 3954.50 ``` --- Source: https://databento.com/docs/examples/basics-historical/custom-ohlcv?lang=python # Custom OHLCV bars from trades ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to resample trade data at a fixed interval to construct 1-minute OHLCV bars. Our OHLCV schemas are directly derived from the [Trades schema](https://databento.com/docs/schemas/trades.md). This example will repeat this derivation to demonstrate that trades data can be resampled to construct the bars in the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). This will be done using the [pandas](https://pypi.org/project/pandas/) package, specifically the [DataFrame.resample](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html) method. ## Example ```python import databento as db import pandas as pd # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" start = "2022-06-06T20:50:00" end = "2022-06-06T21:00:00" symbols = ["ESM2", "ESU2"] # Request trades data for two instruments and convert to DataFrame trades_df = client.timeseries.get_range( dataset=dataset, start=start, end=end, symbols=symbols, schema="trades", ).to_df() # Groupby symbol and resample to 1-minute bars resampled_df = ( trades_df.groupby("symbol") .resample("1min", include_groups=False) .agg( open=("price", "first"), high=("price", "max"), low=("price", "min"), close=("price", "last"), volume=("size", "sum"), ) .dropna() .reset_index() .rename(columns={"ts_recv": "ts_event"}) .sort_values(by=["ts_event", "symbol"]) .set_index("ts_event") ) # Print out resampled data print(resampled_df) # We will validate our resampled data against the OHLCV-1m schema ohlcv_data = client.timeseries.get_range( dataset=dataset, start=start, end=end, symbols=symbols, schema="ohlcv-1m", ) # Sort the DataFrame rows by ts_event and symbol ohlcv_df = ( ohlcv_data.to_df().reset_index().sort_values(by=["ts_event", "symbol"]).set_index("ts_event") ) # Validate these two DataFrames are equal pd.testing.assert_frame_equal( resampled_df, ohlcv_df[["symbol", "open", "high", "low", "close", "volume"]], check_dtype=False, # ignore dtypes ) ``` ## Result ```plaintext symbol open high low close volume ts_event 2022-06-06 20:50:00+00:00 ESM2 4120.50 4122.25 4120.50 4122.25 428 2022-06-06 20:50:00+00:00 ESU2 4123.25 4124.50 4123.25 4124.50 9 2022-06-06 20:51:00+00:00 ESM2 4122.25 4122.25 4121.50 4122.00 254 2022-06-06 20:51:00+00:00 ESU2 4124.00 4124.00 4124.00 4124.00 2 2022-06-06 20:52:00+00:00 ESM2 4122.00 4122.25 4122.00 4122.25 118 2022-06-06 20:52:00+00:00 ESU2 4124.25 4124.25 4124.25 4124.25 1 2022-06-06 20:53:00+00:00 ESM2 4122.25 4122.50 4121.75 4121.75 94 2022-06-06 20:54:00+00:00 ESM2 4121.75 4122.50 4121.75 4122.25 125 2022-06-06 20:55:00+00:00 ESM2 4122.00 4122.50 4122.00 4122.00 172 2022-06-06 20:56:00+00:00 ESM2 4122.25 4122.50 4122.00 4122.25 34 2022-06-06 20:56:00+00:00 ESU2 4124.75 4124.75 4124.75 4124.75 6 2022-06-06 20:57:00+00:00 ESM2 4122.25 4122.50 4122.00 4122.00 121 2022-06-06 20:57:00+00:00 ESU2 4124.75 4124.75 4124.75 4124.75 1 2022-06-06 20:58:00+00:00 ESM2 4122.00 4122.00 4121.25 4121.50 131 2022-06-06 20:58:00+00:00 ESU2 4124.00 4124.25 4124.00 4124.25 2 2022-06-06 20:59:00+00:00 ESM2 4121.50 4121.75 4120.50 4120.75 313 2022-06-06 20:59:00+00:00 ESU2 4123.75 4123.75 4122.75 4122.75 9 ``` --- Source: https://databento.com/docs/examples/basics-historical/joining-schemas?lang=python # Join schemas on instrument ID ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to get trades and definitions data for Henry Hub natural gas futures, excluding spreads. ## Example ```python import asyncio import databento as db async def fetch_data(): client = db.Historical(key="$YOUR_API_KEY") mbp_task = client.timeseries.get_range_async( dataset="GLBX.MDP3", schema="trades", symbols=["NG.FUT"], start="2023-04-02T00:00:00", end="2023-04-05T00:00:00", stype_in="parent", ) def_task = client.timeseries.get_range_async( dataset="GLBX.MDP3", schema="definition", symbols=["NG.FUT"], start="2023-04-02T00:00:00", end="2023-04-05T00:00:00", stype_in="parent", ) return await asyncio.gather(mbp_task, def_task) # Concurrently fetch definitions and trades mbp_store, def_store = asyncio.run(fetch_data()) mbp_df = mbp_store.to_df() def_df = def_store.to_df() # Join futures with their definitions on instrument ID combined_df = mbp_df.merge( def_df, on="instrument_id", how="inner", suffixes=("", "_def"), ).set_index("ts_event") # Remove spreads combined_df = combined_df.loc[combined_df["instrument_class"] == "F", :] print(combined_df[["raw_symbol", "expiration", "price"]].head()) ``` ## Result ```plaintext raw_symbol expiration price ts_event 2023-04-02 22:00:00+00:00 NGK3 2023-04-26 18:30:00+00:00 2.085 2023-04-02 22:00:00+00:00 NGK3 2023-04-26 18:30:00+00:00 2.085 2023-04-02 22:00:00+00:00 NGK3 2023-04-26 18:30:00+00:00 2.085 2023-04-02 22:00:00+00:00 NGM3 2023-05-26 18:30:00+00:00 2.345 2023-04-02 22:00:00+00:00 NGM3 2023-05-26 18:30:00+00:00 2.345 ``` > **See also** > > [List of fields by schema](https://databento.com/docs/schemas/whats-a-schema.md). --- Source: https://databento.com/docs/examples/basics-historical/candlestick-plot?lang=python # Plot a candlestick chart ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve AAPL pre-market one-minute OHLCV bars and plot a candlestick chart. A candlestick chart graphs the opening, high, low, and closing price at regular intervals. This is a common method for graphing price movements. ## OHLCV-1m schema We'll demonstrate this example using the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. We'll use an interval of one minute, a typical charting frequency, which is specified by the suffix `-1m`. ## Dependencies This example will use the [mplfinance](https://pypi.org/project/mplfinance/) package to plot a candlestick chart. This dependency can be installed with the following: ```bash $ pip install mplfinance ``` ## Example ```python import databento as db import mplfinance as mpf # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, we will request OHLCV-1m data for AAPL pre-market data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="ohlcv-1m", symbols=["AAPL"], start="2023-08-28T08:00:00-4", end="2023-08-28T09:30:00-4", ) # Then, convert to a pandas DataFrame df = data.to_df() # Convert the index to the US/Eastern timezone df.index = df.index.tz_convert("US/Eastern") # Finally, plot the candlestick chart mpf.plot( df, type="candle", volume=True, title="Nasdaq TotalView AAPL pre-market", ylabel="OHLCV-1M Candles", ylabel_lower="Volume", xlabel="Time", ) ``` ## Result ![AAPL pre-market candlestick](https://databento.com/docs/candlestick-plot-0.png.md) --- Source: https://databento.com/docs/examples/basics-historical/technical-indicators?lang=python # Calculate VWAP and RSI ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate two technical indicators: the volume weighted average price (VWAP) and relative strength indicator (RSI). To do this we will request the open, high, low, and close prices in one minute intervals along with aggregated volume data to calculate these indicators at one minute intervals. ## OHLCV-1m schema We'll demonstrate this example using the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. We'll use an interval of one minute, a typical charting frequency, which is specified by the suffix `-1m`. ## Example ```python import databento as db # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request the OHLCV-1m data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2022-09-20", symbols="CLV2", stype_in="raw_symbol", schema="ohlcv-1m", ) # Then, convert the DBNStore to a DataFrame ohlcv_data = data.to_df() # Now, we will calculate the volume weighted average price (VWAP) ohlcv_data["pvt"] = ohlcv_data[["open", "high", "close"]].mean(axis=1) * ohlcv_data["volume"] ohlcv_data["vwap"] = ohlcv_data["pvt"].cumsum() / ohlcv_data["volume"].cumsum() # Finally, we will calculate the relative strength indicator (RSI) ohlcv_data["up"] = (ohlcv_data["close"] - ohlcv_data["open"]).clip(lower=0) ohlcv_data["down"] = (ohlcv_data["open"] - ohlcv_data["close"]).clip(lower=0) # We will use a simple moving average for this example ohlcv_data["rs"] = ohlcv_data["up"].rolling(5).mean() / ohlcv_data["down"].rolling(5).mean() ohlcv_data["rsi"] = 100 - (100 / (1 + ohlcv_data["rs"])) # We now have a DataFrame containing our technical indicators print(ohlcv_data[["symbol", "open", "close", "vwap", "rsi"]]) ``` ## Result ```plaintext symbol open close vwap rsi ts_event 2022-09-20 00:00:00+00:00 CLV2 85.85 85.85 85.850000 NaN 2022-09-20 00:08:00+00:00 CLV2 86.05 86.01 85.938889 NaN 2022-09-20 00:10:00+00:00 CLV2 86.03 86.03 85.950278 NaN 2022-09-20 00:11:00+00:00 CLV2 86.03 85.98 85.964516 NaN 2022-09-20 00:12:00+00:00 CLV2 86.01 86.01 85.965938 0.000000 ... ... ... ... ... ... 2022-09-20 18:25:00+00:00 CLV2 84.27 84.40 84.532655 53.488372 2022-09-20 18:26:00+00:00 CLV2 84.41 84.32 84.529908 51.111111 2022-09-20 18:27:00+00:00 CLV2 84.32 84.33 84.526147 64.864865 2022-09-20 18:28:00+00:00 CLV2 84.33 84.40 84.523789 77.500000 2022-09-20 18:29:00+00:00 CLV2 84.30 84.19 84.523212 51.219512 [176 rows x 5 columns] ``` --- Source: https://databento.com/docs/examples/basics-historical/eod?lang=python # End-of-day pricing and portfolio valuation ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate the end-of-day (EOD) value of a hypothetical portfolio. To do this we will request the closing price for each symbol on a given date and use this price to determine the value of our portfolio. ## OHLCV-1d schema We'll demonstrate this example using the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. Since we are interested in end-of-day evaluation, we'll use an interval of one day, which is specified by the suffix `-1d`. Many users will prefer to use the official daily settlement prices found in the `statistics` schema for this purpose. ## Example ```python import databento as db # A hypothetical portfolio mapping symbols to a quantity of shares portfolio = { "AAPL": 200, "AMZN": 200, "GOOG": 100, "META": 100, "NFLX": 114, } # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1d data data = client.timeseries.get_range( dataset="XNAS.ITCH", start="2022-09-19", symbols=list(portfolio.keys()), stype_in="raw_symbol", schema="ohlcv-1d", ) # Convert to DataFrame eod_data = data.to_df() eod_data = eod_data.set_index("symbol") # Sum the products of the close prices and portfolio quantities eod_evaluation = sum( eod_data.at[symbol, "close"] * quantity for symbol, quantity in portfolio.items() ) print(f"${eod_evaluation:,.2f}") ``` ## Result ```plaintext $109,235.90 ``` --- Source: https://databento.com/docs/examples/basics-historical/benchmarks?lang=python # Benchmark portfolio performance > **Info** > > This example builds upon concepts introduced in the [Resampling data](https://databento.com/docs/examples/basics-historical/custom-ohlcv.md) > and [End-of-day pricing and portfolio evaluation](https://databento.com/docs/examples/basics-historical/eod.md) > examples. ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to benchmark a hypothetical portfolio. To do this, we will request daily open and close price data. We will then calculate the daily returns of our portfolio for one year and benchmark against a risk-free rate of 2.5% per year, and calculate the Sharpe ratio. ## OHLCV-1d schema We'll demonstrate this example using the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. Since we are interested in end-of-day evaluation, we'll use an interval of one day, which is specified by the suffix `-1d`. Many users will prefer to use the official daily settlement prices found in the `statistics` schema for this purpose. ## Example ```python import databento as db # A hypothetical portfolio mapping symbols to a quantity of shares portfolio = { "DVN": 300, "MRO": 200, "FTNT": 100, } # Assume a risk-free rate of 2.5% per year annual_benchmark_return = 0.025 daily_benchmark_return = ((1 + annual_benchmark_return) ** (1 / 252)) - 1 # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request the ohlcv-1d data data = client.timeseries.get_range( dataset="XNAS.ITCH", start="2020-12-31T00:00:00", end="2022-01-01T00:00:00", symbols=list(portfolio.keys()), stype_in="raw_symbol", schema="ohlcv-1d", ) # Then, convert the DBNStore to a pandas DataFrame daily_data = data.to_df() # Reshape the data with one column per symbol, propagating the previous price forward for dates without trades. # Note: this does not account for corporate actions such as dividends or splits. close_prices = daily_data.pivot(columns="symbol", values="close").fillna(method="ffill") portfolio_value = sum(close_prices[sym] * qty for sym, qty in portfolio.items()) # Calculate some simple properties of the strategy cost_basis = portfolio_value.iloc[0] final_value = portfolio_value.iloc[-1] total_return = (final_value - cost_basis) / cost_basis # Calculate the arithmetic mean and standard deviation of the daily returns daily_returns = portfolio_value.pct_change().dropna() avg_return = daily_returns.mean() std_return = daily_returns.std() # Finally, we calculate the annualized ex-post Sharpe ratio sharpe_ratio = (252**0.5) * (avg_return - daily_benchmark_return) / std_return print(f"Cost basis: ${cost_basis:,.02f}") print(f"Final portfolio value: ${final_value:,.02f}") print(f"Total return: {total_return:.1%}") print(f"Average of daily returns: {avg_return:.4%}") print(f"Stdev of daily returns: {std_return:.4%}") print(f"Annualized Sharpe ratio: {sharpe_ratio:.2f}") ``` ## Result ```plaintext Cost basis: $20,989.00 Final portfolio value: $52,326.00 Total return: 149.3% Average of daily returns: 0.3830% Stdev of daily returns: 2.0029% Annualized Sharpe ratio: 2.96 ``` --- Source: https://databento.com/docs/examples/basics-historical/halts?lang=python # Market halts, volatility interrupts, and price bands Venues implement different types of market integrity controls to ensure a fair and efficient market. Some of these controls will limit how far price can move in one session, as well as how much price can move in a short period of time. When these events are triggered, it can result in a temporary halt in trading or quoting. ## Velocity Logic CME Globex has [Velocity Logic](https://www.cmegroup.com/education/demos-and-tutorials/understanding-velocity-logic.html), which prevents price from moving too far, too fast. During Velocity Logic, an instrument will transition to a "Pre-Open" state. While this state is normally seen before the full session open for CME Globex, it is also seen during these events. At the end of the event, the instrument will transition back to an "Open" state. ## Overview On 2025-04-09, a tariff pause announcement was made that led to significantly increased volatility on CME Globex. Multiple Velocity Logic events were triggered in Equity Index futures following the announcement. In this example, we'll take a look at just one of the Velocity Logic events after the announcement. We'll use the [status schema](https://databento.com/docs/schemas/status.md) to show the changes in market state during this event. Next, we'll plot the BBO and trades from the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) to show how quoting and trading were affected. Additionally, we'll use the [statistics schema](https://databento.com/docs/schemas/statistics.md) to show some exchange generated statistics. ```python import databento as db import matplotlib.pyplot as plt # Create historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" product = "NQ" start = "2025-04-09T17:18:40" end = "2025-04-09T17:18:55" # Request status data and convert to DataFrame status_data = client.timeseries.get_range( dataset=dataset, schema="status", start=start, end=end, symbols=f"{product}.v.0", stype_in="continuous", ) status_df = status_data.to_df() status_df = status_df[status_df["action"].isin([db.StatusAction.PRE_OPEN, db.StatusAction.TRADING])] status_df["action"] = status_df["action"].map( { db.StatusAction.PRE_OPEN: "Pre-Open", db.StatusAction.TRADING: "Open", }, ) def add_status_vlines(ax): for status_change in status_df.iterrows(): ax.axvline(status_change[0], color="dimgray", linestyle=":", linewidth=1.5) ax.text( status_change[0], ax.get_ylim()[1], status_change[1]["action"], color="dimgray", ha="center", ) # Request MBP-1 data and convert to DataFrame mbp1_data = client.timeseries.get_range( dataset=dataset, schema="mbp-1", start=start, end=end, symbols=f"{product}.v.0", stype_in="continuous", ) mbp1_df = mbp1_data.to_df() mbp1_df = mbp1_df.rename(columns={"bid_px_00": "Bid", "ask_px_00": "Ask"}) # Request statistics data and convert to DataFrame stats_data = client.timeseries.get_range( dataset=dataset, schema="statistics", start=start, end=end, symbols=f"{product}.v.0", stype_in="continuous", ) stats_df = stats_data.to_df() stats_df = stats_df[stats_df["stat_type"] == db.StatType.INDICATIVE_OPENING_PRICE] fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10.9, 10.9), sharey=True) # Plot trades trades_df = mbp1_df[mbp1_df["action"] == db.Action.TRADE] trades_df["price"].plot( ax=ax1, style=".", markersize=5, color="C4", xlabel="Time (UTC)", ylabel="Price", label="Trades", title=f"{product} Trades", ) add_status_vlines(ax1) ax1.legend() # Plot top of book data with Indicative Opening Price mbp1_df[["Bid", "Ask"]].plot(ax=ax2, drawstyle="steps-post") add_status_vlines(ax2) stats_df["price"].plot( ax=ax2, style=".", markersize=10, color="C2", xlabel="Time (UTC)", ylabel="Price", label="IOP", title=f"{product} Top of Book", ) ax2.legend() plt.tight_layout() plt.show() ``` In this first plot, you'll see how trading activity will temporarily halt when the market transitions to a Pre-Open state during Velocity Logic. The second plot takes a look at the BBO. While quoting is still allowed during these events, the bid-ask spread can become crossed. During these Pre-Open states, the exchange publishes a statistic called the [Indicative Opening Price](https://www.cmegroup.com/education/indicative-opening-price-overview.html), also known as the "IOP". This statistic, which can be updated multiple times during a Pre-Open state, provides the most probable price that trading will resume at when the market transitions back to an Open state. When the market transitions back to an Open state, the bid-ask spread will uncross, and trading will resume. ![Velocity Logic 1](https://databento.com/docs/halts-0.svg.md) --- Source: https://databento.com/docs/examples/basics-historical/ohlcv-resampling?lang=python # Resample OHLCV from 1-minute to 5-minute Databento follows a convention in the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md) to only publish a record when a trade occurs in the interval. This approach is adopted by most data vendors for two key reasons: - Multiple interpolation strategies exist and the optimal choice depends on the specific use case. Client-side interpolation keeps the strategy transparent. - This reduces storage and bandwidth requirements, especially for markets with many illiquid instruments like options. This example demonstrates one way to interpolate OHLCV-1m data to ensure exactly one row per minute. The interpolation strategy used in this example does the following: - Forward-fills the close price from the last known value. - Sets open, high, and low equal to the forward-filled close price. - Sets volume to 0 for interpolated periods. ## Overview In this example, we'll use the [historical client](https://databento.com/docs/api-historical/client.md) to request data from the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). First, we'll highlight how to interpolate missing rows in the 1-minute data. Next, we'll show how to resample the data to 5-minute bars. Note that while this example is showing these methods being used individually, you can also chain these methods together. ## Example ```python import databento as db import pandas as pd def interpolate_ohlcv( df: pd.DataFrame, start: pd.Timestamp, end: pd.Timestamp, interp_interval: str, ) -> pd.DataFrame: """ Interpolate OHLCV records between `start` and `end`, since Databento only sends an OHLCV record if a trade happens in that interval. """ def _interpolate_group(group): """Interpolate OHLCV records for each group""" # Reindex with a complete index using specified start/end times group = group.reindex( pd.date_range( start=start, end=end, freq=interp_interval, inclusive="left", ).rename(group.index.name), ) # Forward fill close prices (may remain NaN if no prior data exists) group["close"] = group["close"].ffill() # For intervals with no trades, set open/high/low equal to the close and volume to 0 group = group.fillna( { **{col: group["close"] for col in ["open", "high", "low"]}, "volume": 0, }, ) group["volume"] = group["volume"].astype(int) group = group.drop(columns=["rtype", "instrument_id"], errors="ignore") return group df = ( df.groupby(["publisher_id", "symbol"]) .apply(_interpolate_group, include_groups=False) .reset_index(["publisher_id", "symbol"]) .sort_values(["ts_event", "publisher_id", "symbol"]) ) return df def resample_ohlcv( df: pd.DataFrame, resample_interval: str, ) -> pd.DataFrame: """Resample OHLCV bars to the specified interval""" resampled_df = ( df.groupby(["publisher_id", "symbol"]) .resample(resample_interval) .agg( { "open": "first", "high": "max", "low": "min", "close": "last", "volume": "sum", }, ) .reset_index(["publisher_id", "symbol"]) .sort_values(["ts_event", "publisher_id", "symbol"]) ) return resampled_df # Set parameters start = pd.Timestamp("2024-12-17T09:30:00", tz="US/Eastern") end = pd.Timestamp("2024-12-17T16:00:00", tz="US/Eastern") # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1m data for all AAPL options and convert to DataFrame df = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="ohlcv-1m", symbols="AAPL.OPT", stype_in="parent", start=start, end=end, ).to_df(tz=start.tzinfo) # Interpolate missing rows. df1 = interpolate_ohlcv(df, start, end, "1min") print(df1) # Resample to 5-minute bars df2 = resample_ohlcv(df, "5min") print(df2) ``` ## Result ```plaintext publisher_id symbol open high low close volume ts_event 2024-12-17 09:30:00-05:00 20 AAPL 241220C00115000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00155000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00170000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00195000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00200000 NaN NaN NaN NaN 0 ... ... ... ... ... ... ... ... 2024-12-17 15:59:00-05:00 61 AAPL 261218C00240000 50.70 50.70 50.70 50.70 0 2024-12-17 15:59:00-05:00 61 AAPL 261218P00050000 0.29 0.29 0.29 0.29 0 2024-12-17 15:59:00-05:00 61 AAPL 270115C00240000 50.14 50.14 50.14 50.14 0 2024-12-17 15:59:00-05:00 61 AAPL 270115C00250000 44.34 44.34 44.34 44.34 0 2024-12-17 15:59:00-05:00 61 AAPL 270115C00440000 3.50 3.50 3.50 3.50 0 [3129360 rows x 7 columns] publisher_id symbol open high low close volume ts_event 2024-12-17 09:30:00-05:00 20 AAPL 241220C00220000 30.70 30.70 30.70 30.70 1 2024-12-17 09:30:00-05:00 20 AAPL 241220C00225000 25.95 25.95 25.95 25.95 2 2024-12-17 09:30:00-05:00 20 AAPL 241220C00242500 8.20 8.20 8.20 8.20 1 2024-12-17 09:30:00-05:00 20 AAPL 241220C00245000 6.30 6.30 6.17 6.17 4 2024-12-17 09:30:00-05:00 20 AAPL 241220C00247500 3.70 3.85 3.70 3.85 2 ... ... ... ... ... ... ... ... 2024-12-17 15:55:00-05:00 61 AAPL 250124C00255000 5.95 5.95 5.95 5.95 1 2024-12-17 15:55:00-05:00 61 AAPL 250124C00285000 0.30 0.30 0.30 0.30 47 2024-12-17 15:55:00-05:00 61 AAPL 250124P00235000 1.01 1.01 1.01 1.01 1 2024-12-17 15:55:00-05:00 61 AAPL 250321C00225000 33.90 33.90 33.90 33.90 1 2024-12-17 15:55:00-05:00 61 AAPL 250919P00105000 0.33 0.33 0.33 0.33 2 [231713 rows x 7 columns] ``` --- Source: https://databento.com/docs/examples/basics-historical/encodings?lang=python # Convert DBN to other encoding formats In this example, we'll walk through how to convert DBN to CSV, JSON, or Parquet encoding formats. ## DBN [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md) is an extremely fast message encoding and storage format for normalized market data. DBN is the default encoding used across our API and client libraries. Historical data can be requested with [timeseries.get_range()](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md). This method returns a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md). > **See also** > > Historical data can also be requested in various encoding formats with batch downloads. > See the [programmatic batch downloads example](https://databento.com/docs/examples/basics-historical/programmatic-batch-download.md) for more information. ```python import databento as db client = db.Historical("$YOUR_API_KEY") dbn_store = client.timeseries.get_range( dataset="EQUS.SUMMARY", symbols=["AAPL", "NVDA", "NFLX", "META", "GOOGL"], schema="ohlcv-1d", start="2025-09-30", ) ``` You can save this data to a DBN file with [DBNStore.to_file()](https://databento.com/docs/api-historical/helpers/dbn-store-to-file.md), then read it back into a DBNStore with [DBNStore.from_file()](https://databento.com/docs/api-historical/helpers/dbn-store-from-file.md). ```python dbn_store.to_file("demo_data.dbn") dbn_store = db.DBNStore.from_file("demo_data.dbn") ``` You can inspect the symbology mappings for a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) with the `symbology` attribute. ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") print(dbn_store.symbology) ``` ```stdout {"symbols": ["AAPL", "NVDA", "NFLX", "META", "GOOGL"], "stype_in": "raw_symbol", "stype_out": "instrument_id", "start_date": "2025-09-30", "end_date": "2025-10-01", "partial": [], "not_found": [], "mappings": {"GOOGL": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "7152"}], "NVDA": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "11667"}], "AAPL": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "38"}], "NFLX": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "11275"}], "META": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "10451"}]}} ``` You can [iterate](https://databento.com/docs/api-historical/helpers/dbn-store-iter.md) over a DBNStore, which will yield DBN records. ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") for record in dbn_store: print(record) ``` ```stdout OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=38, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=254855000000, pretty_open=254.855, high=255919000000, pretty_high=255.919, low=253110000000, pretty_low=253.11, close=254630000000, pretty_close=254.63, volume=37704259) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=7152, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=242810000000, pretty_open=242.81, high=243290000000, pretty_high=243.29, low=239245000000, pretty_low=239.245, close=243100000000, pretty_close=243.1, volume=34724346) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=10451, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=742250000000, pretty_open=742.25, high=742970000000, pretty_high=742.97, low=726300000000, pretty_low=726.3, close=734380000000, pretty_close=734.38, volume=16226750) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=11275, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=1206410000000, pretty_open=1206.41, high=1208500000000, pretty_high=1208.5, low=1178000000000, pretty_low=1178, close=1198920000000, pretty_close=1198.92, volume=3830304) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=11667, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=182080000000, pretty_open=182.08, high=187350000000, pretty_high=187.35, low=181480000000, pretty_low=181.48, close=186580000000, pretty_close=186.58, volume=236981032) ``` You can convert a DBNStore into a Pandas DataFrame with [DBNStore.to_df()](https://databento.com/docs/api-historical/helpers/dbn-store-to-df.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") df = dbn_store.to_df() print(df) ``` ```stdout rtype publisher_id instrument_id open high low close volume symbol ts_event 2025-09-30 00:00:00+00:00 35 90 38 254.855 255.919 253.110 254.63 37704259 AAPL 2025-09-30 00:00:00+00:00 35 90 7152 242.810 243.290 239.245 243.10 34724346 GOOGL 2025-09-30 00:00:00+00:00 35 90 10451 742.250 742.970 726.300 734.38 16226750 META 2025-09-30 00:00:00+00:00 35 90 11275 1206.410 1208.500 1178.000 1198.92 3830304 NFLX 2025-09-30 00:00:00+00:00 35 90 11667 182.080 187.350 181.480 186.58 236981032 NVDA ``` ## CSV > **See also** > > Data can also be directly requested in CSV format by setting `encoding="csv"` in [batch.submit_job()](https://databento.com/docs/api-historical/batch/batch-submit-job.md). You can save a DBNStore to a CSV file with [DBNStore.to_csv()](https://databento.com/docs/api-historical/helpers/dbn-store-to-csv.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") dbn_store.to_csv("demo_data.csv") with open("demo_data.csv", "r") as f: print(f.read()) ``` ```stdout ts_event,rtype,publisher_id,instrument_id,open,high,low,close,volume,symbol 2025-09-30T00:00:00.000000000Z,35,90,38,254.855000000,255.919000000,253.110000000,254.630000000,37704259,AAPL 2025-09-30T00:00:00.000000000Z,35,90,7152,242.810000000,243.290000000,239.245000000,243.100000000,34724346,GOOGL 2025-09-30T00:00:00.000000000Z,35,90,10451,742.250000000,742.970000000,726.300000000,734.380000000,16226750,META 2025-09-30T00:00:00.000000000Z,35,90,11275,1206.410000000,1208.500000000,1178.000000000,1198.920000000,3830304,NFLX 2025-09-30T00:00:00.000000000Z,35,90,11667,182.080000000,187.350000000,181.480000000,186.580000000,236981032,NVDA ``` ## JSON > **See also** > > Data can also be directly requested in JSON format by setting `encoding="json"` in [batch.submit_job()](https://databento.com/docs/api-historical/batch/batch-submit-job.md). You can save a DBNStore to a JSONL file with [DBNStore.to_json()](https://databento.com/docs/api-historical/helpers/dbn-store-to-json.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") dbn_store.to_json("demo_data.json") with open("demo_data.json", "r") as f: print(f.read()) ``` ```stdout {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":38},"open":"254.855000000","high":"255.919000000","low":"253.110000000","close":"254.630000000","volume":"37704259","symbol":"AAPL"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":7152},"open":"242.810000000","high":"243.290000000","low":"239.245000000","close":"243.100000000","volume":"34724346","symbol":"GOOGL"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":10451},"open":"742.250000000","high":"742.970000000","low":"726.300000000","close":"734.380000000","volume":"16226750","symbol":"META"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":11275},"open":"1206.410000000","high":"1208.500000000","low":"1178.000000000","close":"1198.920000000","volume":"3830304","symbol":"NFLX"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":11667},"open":"182.080000000","high":"187.350000000","low":"181.480000000","close":"186.580000000","volume":"236981032","symbol":"NVDA"} ``` ## Parquet You can save a DBNStore to a parquet file with [DBNStore.to_parquet()](https://databento.com/docs/api-historical/helpers/dbn-store-to-parquet.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") dbn_store.to_parquet("demo_data.parquet") ``` --- Source: https://databento.com/docs/examples/basics-historical/requesting?lang=python # Request a large number of symbols > **Info** > > If you're new to Databento, see the [Quickstart guide](https://databento.com/docs/quickstart.md) for a walkthrough on setting up our live and historical APIs. > Databento supports several ways for you to get multiple symbols in a single request. If you're interested in a large universe of symbols, we recommend you use any of these supported approaches and filter the symbols on client side as opposed to requesting one symbol at a time. This minimizes the number of requests, which will usually give you better performance. ## Getting all symbols with `"ALL_SYMBOLS"` The simplest way to do this is to get all symbols of a given dataset. Passing in `"ALL_SYMBOLS"` as the `symbols` argument will give you every listed symbol. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="trades", symbols="ALL_SYMBOLS", start="2024-05-21T15:00", end="2024-05-21T15:01", ) df = data.to_df() print(f"{len(df):,d} trade(s) for {len(df['instrument_id'].unique()):,d} instrument(s)") ``` ```plaintext 31,405 trade(s) for 2,854 instrument(s) ``` This works for live subscriptions as well, which gives you a firehose subscription to all symbols: ```python import databento as db client = db.Live("$YOUR_API_KEY") client.subscribe( dataset="XNAS.ITCH", schema="ohlcv-1s", symbols="ALL_SYMBOLS", ) ``` Our live data gateways are capable of delivering every book event on most venues--like a direct feed, except over internet. You'll usually run into bandwidth limitations long before our gateway runs into other practical limits. If your application requires real-time data of all symbols at the granularity of the `trades` schema or more, we recommend you to look into our supported options for [dedicated connectivity](https://databento.com/docs/architecture/dedicated-connectivity-guide.md). ## Getting multiple, specific symbols If you're only interested in a subset of specific symbols, you can pass in a list of symbols--this works for all symbology types. We support a maximum of 2,000 symbols in a single request. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="trades", symbols=["AAPL", "MSFT", "NVDA"], start="2024-05-21T15:00", end="2024-05-21T15:01", ) ``` Likewise, this works for live subscriptions as well: ```python import databento as db client = db.Live("$YOUR_API_KEY") client.subscribe( dataset="XNAS.ITCH", schema="ohlcv-1s", symbols=["AAPL", "MSFT", "NVDA"], ) ``` If you need more than 2,000 symbols in a single request, you can split your symbol list and dispatch multiple requests, but in such situations we usually encourage you to fetch all symbols and discard unwanted symbols on client side, even if it may cost more. There's a practical reason for this recommendation and why we limit to the number of symbols you can specify in this manner. While it may seem unintuitive, selecting specific symbols is usually a much slower operation than requesting all symbols, because our servers have to validate your symbol list, resolve any parent symbols into their child instruments, and inspect each event if it qualifies your selection. ## Getting all child instruments using parent symbology For futures and options data, there's often a need to get all symbols with the same underlying or all symbols that share the same product root. We allow you to reference such a product or cluster of symbols with the same underlying using a **parent symbol**. To specify a parent symbol for futures, you can pass in `"parent"` as the input symbology argument, `stype_in`. You must append the `.FUT` suffix to the symbol: > **See also** > > For more on parent symbology, see the [Symbology](https://databento.com/docs/conventions/symbology.md#parent) guide. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols=["ES.FUT", "NQ.FUT"], start="2024-05-21T15:00", end="2024-05-21T15:01", ) df = data.to_df() print(f"{len(df):,d} trade(s) for {len(df['instrument_id'].unique()):,d} instrument(s)") ``` ```plaintext 1,101 trade(s) for 4 instrument(s) ``` We can also use parent symbology for options with the `.OPT` suffix: ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="trades", stype_in="parent", symbols=["QQQ.OPT", "VIX.OPT"], start="2024-05-21T15:00", end="2024-05-21T15:01", ) df = data.to_df() print(f"{len(df):,d} trade(s) for {len(df['instrument_id'].unique()):,d} instrument(s)") ``` ```plaintext 1,021 trade(s) for 146 instrument(s) ``` Common use cases of parent symbology include fetching [option chains](https://databento.com/docs/examples/options/equity-options-introduction.md#using-parent-symbology-to-fetch-an-option-chain), all outrights and expirations of a futures product, or all maturities of a fixed income product. > **See also** > > See the [0DTE symbols](https://databento.com/docs/examples/options/zero-dte-options.md) example on how to filter instruments based on `expiration`. --- Source: https://databento.com/docs/examples/symbology?lang=python ## Symbology | Title | Subtitle | Supported languages | | [Continuous contracts](https://databento.com/docs/examples/symbology/continuous.md) | Get symbols by lead month and handle rollovers using continuous symbology. | Python, Rust, Futures, Historical | | [Parent symbology](https://databento.com/docs/examples/symbology/parent-symbology.md) | Get all symbols with the same product root or option chains using parent symbology. | Python, C++, Rust, Historical | | [Symbology mapping for live data](https://databento.com/docs/examples/symbology/live-symbol-mapping.md) | Convert between instrument IDs and raw symbols. | Python, C++, Rust, Live | | [Dataset symbols](https://databento.com/docs/examples/symbology/all-dataset-symbols.md) | Get all symbols available in a dataset. | Python, Historical, Live | --- Source: https://databento.com/docs/examples/symbology/continuous?lang=python # Continuous contracts > **Info** > Our continuous contract symbology is a notation that maps to an actual, tradable instrument on any given date. > The continuous contract prices provided are the original, unadjusted prices. Unlike some vendor implementations that > back-adjust prices to remove jumps during rollovers, our approach maintains the original properties of the data. When dealing with far-reaching historical data on expiring products (like the [E-mini S&P 500 Futures](https://databento.com/portal/datasets/GLBX.MDP3/Futures/ES)), users have to be concerned with expiration cycles and what is defined as the [front month](https://www.cftc.gov/LearnAndProtect/AdvisoriesAndArticles/CFTCGlossary/index.htm#frontmonth) or first expiry, second expiry, etc. When requesting (E-mini S&P 500 Futures) data, we can specify the particular product we want. Take for example, `ESH5`, the March 2025 expiring E-Mini S&P 500 futures contract. Since this instrument has a March 2025 expiry, it will not exist after March 2025. Since March 2025 is the front month (i.e. the nearest expiration date for the contract) only from December 2024 to March 2025, the instrument's data dating before December 2024 could be considered irrelevant. Continuous contracts symbology provides a way to navigate this complexity. You can easily access recent contracts: front month, second expiry, and so on. ## Symbology To access a specific contract in a continuous contract product, query for that contract using this symbol structure. `[ticker].v.[expiry]` ## Expiry In place of the exchange's proprietary code for specifying the month of expiration (e.g. `H5` for `ESH5` in CME data), we use an index to refer to which expiry in the current cycle we want. | Index | Expiry | | 0 | front month | | 1 | second expiry | | 2 | third expiry | | n | nth expiry | ## Algorithm To request the front month for the `ES` product based on volume, use `ES.v.0`. In this example, the v indicates the calendar roll rule. The zero corresponds to the front month. Learn more about continuous contract symbology in our [continuous symbology](https://databento.com/docs/conventions/symbology.md#continuous) guide. ## Example ```python import databento as db # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2025-03-16" end = "2025-03-23" # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1d data for the continuous contract data = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=f"{product}.v.0", stype_in="continuous", start=start, end=end, ) # Convert to DataFrame df = data.to_df() print(df) ``` The data received is now related to the volume-based front month for each day of the requested time range. If we are interested in the second month then we can just use `ES.v.1` instead. The results below are from the week of expiration for `ES`. As you can see, the contract that maps to the volume-based front month, as shown by the `instrument_id` field, changes mid-week as the volume shifts to the new contract. ```plaintext rtype publisher_id instrument_id open high low close volume symbol ts_event 2025-03-16 00:00:00+00:00 35 1 5002 5626.75 5632.00 5602.50 5608.50 30078 ES.v.0 2025-03-17 00:00:00+00:00 35 1 5002 5608.75 5707.50 5599.00 5679.50 723022 ES.v.0 2025-03-18 00:00:00+00:00 35 1 5002 5679.75 5683.25 5600.00 5621.25 412431 ES.v.0 2025-03-19 00:00:00+00:00 35 1 4916 5672.50 5770.50 5657.50 5747.75 1331926 ES.v.0 2025-03-20 00:00:00+00:00 35 1 4916 5747.50 5765.25 5682.50 5714.75 1503542 ES.v.0 2025-03-21 00:00:00+00:00 35 1 4916 5714.25 5723.75 5651.25 5720.00 1454908 ES.v.0 ``` --- Source: https://databento.com/docs/examples/symbology/parent-symbology?lang=python # Parent symbology ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve a large number of instrument definitions using [parent symbology](https://databento.com/docs/conventions/symbology.md#parent). When requesting data with a parent symbol (`ES.FUT`), all contracts that trade under the parent name will be returned. This includes outright futures (`ESM5`), as well as calendar spreads (`ESM5-ESU5`). > **See also** > > If you are only interested in the front month contract for a futures product, see our [continuous symbology](https://databento.com/docs/examples/symbology/continuous.md) example. We'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties such as `expiration`, `raw_symbol`, and `instrument_class`. `instrument_class` will indicate the type of instrument, such as future or future spread. A full list of variants can be found in the [instrument class documentation](https://databento.com/docs/schemas/instrument-definitions.md#instrument-class). ## Example ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2025-05-19", symbols="ES.FUT", stype_in="parent", schema="definition", ) # Convert to DataFrame df = data.to_df() # Filter out spreads and sort by expiration df = df[df["instrument_class"] == db.InstrumentClass.FUTURE] df = df.set_index("expiration").sort_index() print(df[["instrument_id", "raw_symbol"]]) ``` ## Result ```plaintext instrument_id raw_symbol expiration 2025-06-20 13:30:00+00:00 4916 ESM5 2025-09-19 13:30:00+00:00 14160 ESU5 2025-12-19 14:30:00+00:00 294973 ESZ5 2026-03-20 13:30:00+00:00 42140878 ESH6 2026-06-18 13:30:00+00:00 42140864 ESM6 ... 2026-09-18 13:30:00+00:00 42140870 ESU6 2026-12-18 14:30:00+00:00 10252 ESZ6 2027-03-19 13:30:00+00:00 42140860 ESH7 2027-06-17 13:30:00+00:00 42140856 ESM7 2027-09-17 13:30:00+00:00 42140874 ESU7 2027-12-17 14:30:00+00:00 17740 ESZ7 2028-03-17 13:30:00+00:00 42140879 ESH8 ``` --- Source: https://databento.com/docs/examples/symbology/live-symbol-mapping?lang=python # Symbology mapping for live data ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to create symbology mappings using the `SymbolMappingMsg`. This message is sent at the start of every session and when a subscribed symbol mapping is updated. These messages allow us to map between the instrument ID on every record and more human-readable STypes like raw and continuous contract symbologies. ## Example ```python import databento as db # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to a few symbols live_client.subscribe( dataset="GLBX.MDP3", schema="ohlcv-1s", symbols=["ES.c.0", "ES.c.1", "ES.c.2", "ES.c.3"], stype_in="continuous", ) # Finally, we will iterate over the incoming records. # The SymbolMappingMsgs will be sent at the start. for record in live_client: if isinstance(record, db.SymbolMappingMsg): instrument_id = record.instrument_id # The stype_in_symbol contains the continuous symbology we requested continuous_symbol = record.stype_in_symbol # The stype_out_symbol contains the raw symbol for smart symbology raw_symbol = record.stype_out_symbol print( f"{continuous_symbol} ({raw_symbol}) has an instrument ID of {instrument_id}", ) # Close the connection # We will still process all received data live_client.stop() ``` ## Result ```plaintext ES.c.0 (ESM4) has an instrument ID of 5602 ES.c.1 (ESU4) has an instrument ID of 118 ES.c.2 (ESZ4) has an instrument ID of 183748 ES.c.3 (ESH5) has an instrument ID of 5002 ``` > **See also** > > [Symbology](https://databento.com/docs/api-live/basics/symbology.md) for further details. --- Source: https://databento.com/docs/examples/symbology/all-dataset-symbols?lang=python # Dataset symbols ## Overview In this example we will show how to get all symbols for a dataset. We will use the [Historical client](https://databento.com/docs/api-historical/client.md) to get all symbols available on a specific date. We will also use the [Live client](https://databento.com/docs/api-live/client/live.md) to get all symbols active for the current session. ## Definition schema We will use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). We will create a map of `raw_symbol` to `InstrumentDefMsgs`. An `InstrumentDefMsg` contains fields such as: - `raw_symbol` - The instrument name (symbol) provided by the publisher - `instrument_class` - The classification of the instrument - `expiration` - The expiration timestamp for options/futures contracts - `min_price_increment` - The minimum constant trade tick for an instrument ## Historical example First, we'll take a look at how to do this with the Historical client. ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") dataset = "EQUS.SUMMARY" # Download definition schema for all symbols data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", start="2025-02-06", end="2025-02-07", schema="definition", ) # Create a map of raw_symbol -> InstrumentDefMsgs symbol_map = {msg.raw_symbol: msg for msg in data} # Get all symbols symbols = sorted(symbol_map.keys()) # Print out the symbol count and the first 5 symbols print(f"Total symbol count for {dataset} = {len(symbols)}\nFirst 5 symbols...{symbols[:5]}") ``` ```plaintext Total symbol count for EQUS.SUMMARY = 11198 First 5 symbols...['A', 'AA', 'AAA', 'AAAU', 'AACBU'] ``` ## Live example Next, we'll take a look at how to do this with the Live client. > **Info** > > You will need a live license for the dataset to run this example. See our [plans and live data page]($PLATFORM_HOSTNAME/docs/portal/live-data) for more information. ```python import databento as db # Create a live client live_client = db.Live("$YOUR_API_KEY") dataset = "GLBX.MDP3" # Subscribe to the definition schema for all symbols for this session live_client.subscribe( dataset=dataset, schema="definition", symbols="ALL_SYMBOLS", start=0, ) # Create a map of raw_symbol -> InstrumentDefMsgs symbol_map = {} def append_symbol_map(msg): if isinstance(msg, db.InstrumentDefMsg): symbol_map[msg.raw_symbol] = msg # Add the callback and start the stream live_client.add_callback(append_symbol_map) live_client.start() # Listen to the stream for 10 seconds to get all messages live_client.block_for_close(timeout=10) # Get all symbols symbols = sorted(symbol_map.keys()) # Print out the symbol count and the first 5 symbols print(f"Total symbol count for {dataset} = {len(symbols)}\nFirst 5 symbols...{symbols[:5]}") ``` ```plaintext Total symbol count for GLBX.MDP3 = 784038 First 5 symbols...['00CH5', '00CJ5', '00CK5', '00CM5', '00CN5'] ``` --- Source: https://databento.com/docs/examples/instrument-definitions?lang=python ## Instrument definitions | Title | Subtitle | Supported languages | | [Find most liquid instruments](https://databento.com/docs/examples/instrument-definitions/liquid-universe.md) | Find most active instruments using instrument definitions and OHLCV data. | Python, Historical | | [Handling tick sizes](https://databento.com/docs/examples/instrument-definitions/tick-sizes.md) | Convert prices and spread size to tick units. | Python, Historical | | [Find ICE intercommodity spreads](https://databento.com/docs/examples/instrument-definitions/ice-crack-spreads.md) | Find ICE intercommodity futures spreads from leg definitions. | Python, Historical | | [Displaying fractional prices](https://databento.com/docs/examples/instrument-definitions/fractional-prices.md) | Render prices in the venue's own fractional convention, such as 108'185. | Python, Historical | | [Computing tick value and contract notional](https://databento.com/docs/examples/instrument-definitions/contract-notional.md) | Derive the value of one tick and the notional value of a contract. | Python, Historical | --- Source: https://databento.com/docs/examples/instrument-definitions/liquid-universe?lang=python # Finding liquid instruments ## What is a liquid instrument? A liquid instrument is an instrument that is readily traded. Identifying liquid instruments is important for a few reasons: - **Reducing transaction costs.** A liquid instrument generally has tight bid-ask spreads and adequate size on the bid and ask. This reduces the amount of slippage incurred from using aggressive order types. - **Signal generation.** Consistent trading activity allows for signal generation in order flow strategies. - **Price efficiency.** Increased participation results in more efficient price discovery. This results in more reliable price valuations for risk management. Check out the [Databento Microstructure Guide](https://databento.com/microstructure/liquidity) for more information about liquidity. ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to find liquid futures instruments. We'll use the following schemas: - The [statistics schema](https://databento.com/docs/schemas/statistics.md), which contains exchange-published statistics such as cleared volume and open interest. - The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties such as `raw_symbol` and `asset`. The `asset` field refers to the parent product for an instrument. - The [BBO schema](https://databento.com/docs/schemas/bbo.md), which contains the best bid and offer, subsampled at 1-second or 1-minute intervals. We'll request statistics and definition data for all symbols. Next, we'll filter for the top 25 instruments by volume. After we filter, we'll get BBO data for these instruments to find the median bid and ask size over a full day. ## Example ```python import databento as db # First, create a Historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" start_date = "2025-03-07" top_instruments_count = 25 # First, download definition data for all symbols def_data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="definition", start=start_date, ) # Convert to DataFrame. Filter for outright futures def_df = def_data.to_df() def_df = def_df[def_df["instrument_class"] == db.InstrumentClass.FUTURE] def_df = def_df[["raw_symbol", "instrument_id", "asset", "min_price_increment"]] # Next, download statistics data for all symbols stats_data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="statistics", start=start_date, ) # Convert to DataFrame stats_df = stats_data.to_df() # Get cleared volume records volume_df = stats_df[stats_df["stat_type"] == db.StatType.CLEARED_VOLUME] volume_df = volume_df.drop_duplicates("instrument_id", keep="last") volume_df = volume_df.rename(columns={"quantity": "volume"}) volume_df = volume_df[["instrument_id", "volume"]] # Get open interest records open_interest_df = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST] open_interest_df = open_interest_df.drop_duplicates("instrument_id", keep="last") open_interest_df = open_interest_df.rename(columns={"quantity": "open_interest"}) open_interest_df = open_interest_df[["instrument_id", "open_interest"]] # Merge volume and open interest data stats_df = volume_df.merge(open_interest_df, on="instrument_id", how="inner") # Merge definition data with statistics data stats_df = stats_df.merge(def_df, on="instrument_id", how="inner") # Sort by volume, keeping one instrument per product stats_df = stats_df.sort_values("volume", ascending=False) stats_df = stats_df.drop_duplicates("asset") # Get instrument IDs for highest volume instruments top_instruments = stats_df["instrument_id"].to_list()[:top_instruments_count] # Download BBO-1s data for highest volume instruments bbo_data = client.timeseries.get_range( dataset=dataset, symbols=top_instruments, stype_in="instrument_id", schema="bbo-1s", start=start_date, ) # Convert to DataFrame bbo_df = bbo_data.to_df() # Merge DataFrames df = bbo_df.merge(stats_df, on="instrument_id", how="inner") df["spread_ticks"] = (df["ask_px_00"] - df["bid_px_00"]) / df["min_price_increment"] # Calculate aggregated values and sort by volume df = ( df.groupby(by="instrument_id") .agg( product=("asset", "first"), symbol=("raw_symbol", "first"), volume=("volume", "first"), open_interest=("open_interest", "first"), median_bid_size=("bid_sz_00", lambda x: int(x.median())), median_ask_size=("ask_sz_00", lambda x: int(x.median())), median_tick_spread=("spread_ticks", lambda x: int(x.median().round())), ) .sort_values("volume", ascending=False) ) print(df) ``` ## Result ```plaintext product symbol volume open_interest median_bid_size median_ask_size median_tick_spread instrument_id 42004113 ZN ZNM5 3259518 4675273 903 946 1 5002 ES ESH5 2553819 2115168 9 9 1 42003617 MNQ MNQH5 2419630 154823 2 2 2 42325990 ZF ZFM5 2039292 6111849 483 492 1 42005347 MES MESH5 1931181 210864 12 12 1 42325992 ZT ZTM5 1320314 3854308 443 455 1 42002878 TN TNM5 962958 2206335 278 285 1 254274 SR3 SR3Z5 894673 979670 2864 2898 1 42288528 NQ NQH5 763150 273526 1 1 4 42004255 ZB ZBM5 708447 1770658 228 236 1 42001682 UB UBM5 474088 1757409 99 102 1 42272 6E 6EH5 392404 601400 15 14 1 625061 CL CLJ5 341632 239110 6 6 1 680969 ZC ZCK5 313858 756635 83 74 1 42001620 RTY RTYH5 258273 454862 2 3 2 57969 6J 6JH5 242466 278473 23 21 1 42011026 MYM MYMH5 216632 23321 3 3 2 892 NG NGJ5 202475 208149 3 3 2 457556 ZS ZSK5 192141 382968 16 15 1 42002868 YM YMH5 185559 70792 3 3 2 29307 6B 6BH5 178634 192142 39 40 1 19604 GC GCJ5 166364 329381 2 2 2 713217 ZQ ZQJ5 158838 497521 41383 11147 1 45908 6C 6CH5 137522 303329 27 28 1 52126 6A 6AH5 131315 180048 29 31 1 ``` --- Source: https://databento.com/docs/examples/instrument-definitions/tick-sizes?lang=python # Handling tick sizes ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to retrieve instrument definitions and top-of-book data for two futures contracts during non-US equity cash session hours. We will then calculate the mean spread and normalize the values to a discrete number of price increments or "ticks" using the `min_price_increment` field from the definition schema. Finally, we will plot the changes in mean spread. The futures we will use for this example are [British Pound Futures (6B)](https://databento.com/portal/datasets/GLBX.MDP3/Futures/6B) and [Euro FX Futures (6E)](https://databento.com/portal/datasets/GLBX.MDP3/Futures/6E). We will use [continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous) to request the second highest contracts by open interest, which should have a more dynamic mean spread. ## Definition schema The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) contains instrument definitions and properties. In this example, we are interested in the `min_price_increment` field, which will allow us to normalize a spread to a discrete number of increments. ## Mean spread The mean spread is the average difference between the bid and ask price of an instrument. The mean spread is a metric which helps traders understand the liquidity and transaction costs associated with a particular trade. Illiquid and volatile instruments tend to have a larger mean spread. Such instruments are more likely to experience price slippage, where the actual price of a trade moves or "slips" away from the expected price. ## Dependencies For plotting, this example will use the [matplotlib](https://pypi.org/project/matplotlib/) package. This dependency can be installed with the following: ```bash $ pip install matplotlib ``` ## Example ```python import databento as db import pandas as pd from matplotlib import pyplot as plt # Get symbols for 6B and 6E futures with the second highest open interest symbols = ["6B.n.1", "6E.n.1"] # Define a start and end time for this example dataset = "GLBX.MDP3" start = pd.Timestamp("2024-04-07T21:00:00", tz="US/Eastern") end = pd.Timestamp("2024-04-08T04:00:00", tz="US/Eastern") # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, retrieve the instrument definitions for our symbols definitions_data = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=symbols, stype_in="continuous", start=start.date(), ) # And extract the min_price_increment field definitions_df = definitions_data.to_df() tick_sizes = definitions_df[["raw_symbol", "instrument_id", "min_price_increment"]].set_index( "instrument_id", ) # Print the min_price_increment for each symbol print(tick_sizes) # Then, request top-of-book updates for our symbols overnight mbp_data = client.timeseries.get_range( dataset=dataset, schema="MBP-1", symbols=symbols, stype_in="continuous", start=start, end=end, ) # And convert to a DataFrame and join with our tick size mbp_df = mbp_data.to_df(tz="US/Eastern") mbp_df = mbp_df.join(tick_sizes, on="instrument_id") mbp_df = mbp_df[["raw_symbol", "ask_px_00", "bid_px_00", "min_price_increment"]] # Now, calculate the spread in number of ticks mbp_df["spread"] = (mbp_df["ask_px_00"] - mbp_df["bid_px_00"]) / mbp_df["min_price_increment"] # Finally, plot the mean spread for each symbol for symbol, data in mbp_df.groupby("raw_symbol"): # Average this data using a 5-minute rolling window mean_spread = ( data["spread"] .ewm( times=data.index, halflife=pd.Timedelta(minutes=5), ) .mean() .reset_index() ) plt.plot( "ts_recv", "spread", data=mean_spread, label=symbol, ) plt.legend() plt.title(f"Overnight Mean Spread - {start.date()}") plt.ylabel("Mean Spread (ticks)") plt.xlabel("Time (UTC)") plt.show() ``` ## Result ```plaintext raw_symbol min_price_increment instrument_id 200251 6BU4 0.00010 201330 6EU4 0.00005 ``` ![Handling tick sizes](https://databento.com/docs/tick-sizes-0.svg.md) --- Source: https://databento.com/docs/examples/instrument-definitions/ice-crack-spreads?lang=python # Finding ICE intercommodity spreads ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to find ICE intercommodity futures spreads, where the legs do not all belong to the same asset. Crack spreads are a specific type of intercommodity spread. ICE's [Strategy Code Reference Manual](https://www.ice.com/publicdocs/technology/ICE_Strategy_Code_Reference_Manual.pdf) describes `CRACK` as an exchange-defined spread between two products where one leg is a refined product and the other is an unrefined product. ICE also lists concrete crack spread contracts such as [RBOB Gasoline/Brent Crack Spread](https://www.ice.com/products/2634472/RBOB-Gasoline-Brent-Crack-Spread). We'll request the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) for all symbols in `IFEU.IMPACT`, filter to instruments with `instrument_class` equal to `FUTURE_SPREAD`, and use the leg definition fields to inspect each spread's legs. For each leg, we'll join `leg_instrument_id` back to the definitions to recover the leg instrument's `asset`. The example prints the `raw_symbol` of every futures spread where at least one leg has the target asset `BRN`, and the set of leg assets contains more than one asset. ## Example ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "IFEU.IMPACT" target_asset = "BRN" date = "2026-05-26" # Download definition data for all symbols data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="definition", start=date, ) # Convert to a DataFrame df = data.to_df() # Map each instrument_id to its asset. This lets us look up the asset for each # leg_instrument_id in a spread definition. latest_defs = df.drop_duplicates("instrument_id", keep="last") asset_by_instrument_id = latest_defs.set_index("instrument_id")["asset"] # In the definition schema, each spread has one row for each leg. The primary # fields identify the spread, while fields beginning with leg_ identify the leg. spread_legs = df[ (df["instrument_class"] == db.InstrumentClass.FUTURE_SPREAD) & (df["leg_count"] > 0) ].copy() spread_legs = spread_legs.drop_duplicates( ["instrument_id", "leg_index"], keep="last", ) spread_legs["leg_asset"] = spread_legs["leg_instrument_id"].map( asset_by_instrument_id, ) spread_legs = spread_legs.dropna(subset=["leg_asset"]) def matches_asset_and_is_intercommodity(legs): leg_assets = set(legs["leg_asset"]) return target_asset in leg_assets and len(leg_assets) > 1 matches = spread_legs.groupby("instrument_id").filter( matches_asset_and_is_intercommodity, ) raw_symbols = matches.drop_duplicates("instrument_id")["raw_symbol"].sort_values() print(raw_symbols) ``` ## Result ```plaintext ts_recv 2026-05-26 00:00:00+00:00 ADM FMF0027-BRN FMF0027 2026-05-26 00:00:00+00:00 ADM FMG0027-BRN FMG0027 2026-05-26 00:00:00+00:00 ADM FMH0027-BRN FMH0027 2026-05-26 00:00:00+00:00 ADM FMJ0027-BRN FMJ0027 2026-05-26 00:00:00+00:00 ADM FMK0027-BRN FMK0027 ... 2026-05-26 00:00:00+00:00 WBS FYF0028.Z0028-BRN FYF0028.Z0028 2026-05-26 00:00:00+00:00 WBS FYF0029.Z0029-BRN FYF0029.Z0029 2026-05-26 00:00:00+00:00 WBS FYF0030.Z0030-BRN FYF0030.Z0030 2026-05-26 00:00:00+00:00 WBS FYF0031.Z0031-BRN FYF0031.Z0031 2026-05-26 00:00:00+00:00 WBS FYF0032.Z0032-BRN FYF0032.Z0032 Name: raw_symbol, Length: 436, dtype: str ``` --- Source: https://databento.com/docs/examples/instrument-definitions/fractional-prices?lang=python # Displaying fractional prices ## Overview CME quotes its grain and treasury products in fractions rather than decimals. A 10-year treasury note trading at 108.578125 is quoted as `108'185`, i.e. 108 37⁄64, and corn trading at 498.25 is quoted as `498'2`, i.e. 498 2⁄8. In this example we will use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to retrieve instrument definitions and daily bars for several of these products, then render each close price the way the exchange does. ## Definition schema The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) describes the convention with three fields, which CME documents under [fractional pricing](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457092591/Fractional+Pricing). | Field | Description | | `main_fraction` | The denominator of the main fraction. 255 when unused. | | `sub_fraction` | The denominator of a fraction of one unit of the main fraction. 255 when unused. | | `price_display_format` | The number of digits printed to the right of the tick mark. | `sub_fraction` subdivides one unit of the main fraction, so the product of the two fields give the finest fraction the venue displays. For example, a `main_fraction` of 32 with a `sub_fraction` of 2 displays halves of a 32nd, or 64ths. When `sub_fraction` is unset, the main fraction is the finest unit. | Product | `main_fraction` | `sub_fraction` | `price_display_format` | Decimal price | Quoted | Interpretation | | Corn (`ZCZ6`) | 8 | unset | 1 | 498.25 | `498'2` | 498 2⁄8 | | 30-year bond (`ZBZ6`) | 32 | unset | 2 | 109.5625 | `109'18` | 109 18⁄32 | | 10-year note (`ZNZ6`) | 32 | 2 | 3 | 108.578125 | `108'185` | 108 18⁄32 plus half a 32nd, or 108 37⁄64 | | 2-year note (`ZTZ6`) | 32 | 8 | 3 | 102.9453125 | `102'302` | 102 30⁄32 plus a quarter of a 32nd, or 102 242⁄256 | `price_display_format` is the total number of digits printed after the tick mark ('), including one digit for the subdivision when `sub_fraction` is set. The last digit is a *decimal* rendering of the subdivision rather than its numerator, which is why half of a 32nd prints as `5` in `108'185`. That digit is truncated rather than rounded, so a price of 102.9609375, which is 102 and 30 6⁄8 32nds, renders as `102'307` and not `102'308`. The finest displayed fraction can be finer than the instrument's tick. Corn futures have a `min_price_increment` of 0.25, or two eighths, so only even digits appear in their final position. Products quoted in decimals, such as E-mini S&P 500 futures, leave all three fields unset. > **Info** > > These three fields carry venue-specific meanings. > This example applies to GLBX.MDP3, and other datasets may populate them differently, so check the [venue documentation](https://databento.com/docs/datasets.md) before reusing this logic elsewhere. ## Example ```python import databento as db # The instruments to display. ESU6 is included to show a decimal-quoted product symbols = ["ZCZ6", "ZBZ6", "ZNZ6", "ZTZ6", "ESU6"] dataset = "GLBX.MDP3" date = "2026-08-19" # Value when unset or null UNSET = 255 def display_price(price, main_fraction, sub_fraction, price_display_format): """Render a price in the venue's own quoting convention, such as 108'185.""" if main_fraction == UNSET or price_display_format == UNSET: return f"{price:g}" whole = int(price) # The number of main fractions, such as 32nds, in the remainder. Fractional # prices land on a power-of-two grid, so this arithmetic is exact ticks = (price - whole) * main_fraction main = int(ticks) if sub_fraction == UNSET: return f"{whole}'{main:0{price_display_format}d}" # The last digit is the single-digit decimal of the leftover fraction of one main # fraction, so a half prints as 5 and three quarters as 7 digits = max(price_display_format - 1, 0) leftover = int((ticks - main) * 10) return f"{whole}'{main:0{digits}d}{leftover}" # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, retrieve the instrument definitions definitions_data = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=symbols, start=date, ) # And extract the three fields that describe the display convention definitions_df = definitions_data.to_df() definitions_df = definitions_df.loc[ definitions_df["instrument_class"] == db.InstrumentClass.FUTURE, :, ] conventions = definitions_df.drop_duplicates("raw_symbol").set_index("instrument_id")[ ["raw_symbol", "main_fraction", "sub_fraction", "price_display_format"] ] # Then, request daily bars ohlcv_data = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbols, start=date, ) # And join the bars with the display convention of each instrument df = ohlcv_data.to_df().join(conventions, on="instrument_id").sort_values("raw_symbol") # Finally, render each closing price in the venue's convention df["decimal_close"] = df["close"] df["display_close"] = df.apply( lambda row: display_price( row["close"], row["main_fraction"], row["sub_fraction"], row["price_display_format"], ), axis=1, ) print( df[ [ "raw_symbol", "decimal_close", "main_fraction", "sub_fraction", "price_display_format", "display_close", ] ], ) ``` ## Result ```plaintext raw_symbol decimal_close main_fraction sub_fraction price_display_format display_close ts_event 2026-08-19 00:00:00+00:00 ESU6 7740.000000 255 255 255 7740 2026-08-19 00:00:00+00:00 ZBZ6 109.562500 32 255 2 109'18 2026-08-19 00:00:00+00:00 ZCZ6 498.250000 8 255 1 498'2 2026-08-19 00:00:00+00:00 ZNZ6 108.578125 32 2 3 108'185 2026-08-19 00:00:00+00:00 ZTZ6 102.945312 32 8 3 102'302 ``` --- Source: https://databento.com/docs/examples/instrument-definitions/contract-notional?lang=python # Computing tick value and contract notional ## Overview The value of a tick and the notional value of a contract can be derived from the contract and tick size in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). E-mini S&P 500 futures are quoted in index points, crude oil in dollars per barrel, lean hogs in cents per pound, and corn in cents per bushel. In this example we will use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to derive this information from the definition schema and convert daily closes into contract values. ## Definition schema | Field | Description | | `unit_of_measure` | The unit the contract is measured in, such as `BBL` for barrels or `LBS` for pounds. | | `unit_of_measure_qty` | The contract size, in the `unit_of_measure`. | | `min_price_increment` | The minimum tick size, in price units. | | `min_price_increment_amount` | The value of one tick, with the display factor applied. | | `display_factor` | The multiplier from the venue's display price to the conventional price. | > **Warning** > > CME lists tag 1146, which carries `min_price_increment_amount`, as under development and permits a null value. > Treat the tick value and notional below as a first pass and confirm them against the contract specification for instruments you trade. Three steps take you from these fields to a tick value and a contract value. ``` scale = (min_price_increment_amount / display_factor) / (min_price_increment * unit_of_measure_qty) tick_value = min_price_increment * unit_of_measure_qty * scale notional = close * unit_of_measure_qty * scale ``` `scale` is the multiplier that turns `close * unit_of_measure_qty` into the currency named by `currency`. A value of 1 means the price is already in that currency's major unit per unit of measure, as for E-mini S&P 500 futures at 7740.00 index points with a contract size of 50, giving a notional of 387,000 USD. A value of 0.01 means the price is in hundredths, as for lean hogs at 81.50 cents per pound with a contract size of 40,000 pounds, giving 32,600 USD. The formula for `scale` reconciles CME's published tick value against the contract terms. When the two agree it gives a power of ten. > **Info** > > A `scale` of 0.01 reads as cents when `unit_of_measure` is a physical unit, and as percent of par when it is a currency, as it is for treasuries. ## Fractionally quoted products CME does not publish a usable `min_price_increment_amount` for the products it quotes in fractions. Corn reports 1.0 against a real tick value of 12.50 USD, and the 10-year note reports 0.001 against 15.625 USD. These products have [fractional pricing](https://databento.com/docs/examples/instrument-definitions/fractional-prices.md), and they are quoted in hundredths: grains are cents per bushel and treasuries are percent of par, so a `scale` of 0.01 applies to both. Taking that value gives the correct tick value and notional for both. All fractionally-priced products in CME are listed on CBOT (`exchange=XCBT`), but there are many non-fractionally priced future assets on CBOT. ## Products with a contradictory tick value For some products, CME publishes a `min_price_increment_amount` that contradicts the contract size it accompanies. Silver futures (`SI.FUT`) report 5.00 USD, the tick value of the 1,000-ounce contract (`SIL.FUT`), against a `unit_of_measure_qty` of 5,000 ounces. `min_price_increment` and `unit_of_measure_qty` are correct here. They give a tick value of 25.00 and a notional of 339,275 USD at a close of 67.855. > **Info** > > A `scale` reached this way is an inference. > Exact reconciliation shows that the amount and the contract terms agree, not that either is right. > Check tick values against the venue's contract specifications. ## Validating the result `price_scale` returns `None` when the tick amount is missing, or too far from the contract terms to reconcile. The example covers outright futures, and options and spreads need separate handling. A tick size can change during the day. Other datasets populate the field differently, so check the [venue documentation](https://databento.com/docs/datasets.md) before reusing this logic. ## Example ```python import math import databento as db # A mix of products quoted in the currency's major unit and in cents symbols = [ "ESU6", "CLV6", "GCZ6", "6EU6", "HEV6", "ZLZ6", "MZCZ6", "ZCZ6", "ZNZ6", "SIZ6", ] dataset = "GLBX.MDP3" date = "2026-08-19" # Value when unset or null UNSET = 255 # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, retrieve the instrument definitions definitions = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=symbols, start=date, ).to_df() # And extract the fields that describe the contract size and the tick definitions = definitions.loc[definitions["instrument_class"] == db.InstrumentClass.FUTURE] contracts = definitions.drop_duplicates("raw_symbol").set_index("instrument_id")[ [ "raw_symbol", "unit_of_measure", "unit_of_measure_qty", "main_fraction", "min_price_increment", "min_price_increment_amount", "display_factor", ] ] # Then, request daily bars bars = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbols, start=date, ).to_df() # And join the bars with the contract terms of each instrument df = bars.join(contracts, on="instrument_id").sort_values("raw_symbol") # Prices arrive in the currency's major unit or in hundredths of it CONVENTIONS = (1.0, 0.01) def nearest_convention(scale): """Return the convention nearest scale, or None when it is over ten times off.""" convention = min(CONVENTIONS, key=lambda c: abs(math.log10(scale / c))) if abs(math.log10(scale / convention)) >= 1: return None return convention def price_scale( main_fraction, min_price_increment, min_price_increment_amount, display_factor, unit_of_measure_qty, ): """Return the multiplier from price * unit_of_measure_qty to the currency, or None.""" # Fractionally quoted products are priced in hundredths if main_fraction != UNSET: return 0.01 # A scale needs a positive tick, contract size, and tick amount terms = (min_price_increment, unit_of_measure_qty, min_price_increment_amount) if not all(term > 0.0 for term in terms): return None tick_value = min_price_increment_amount / display_factor contract_tick = min_price_increment * unit_of_measure_qty # One tick moves the contract by one tick value scale = tick_value / contract_tick exponent = round(math.log10(scale)) if math.isclose(scale, 10.0**exponent, rel_tol=1e-9): return 10.0**exponent # Otherwise the amount contradicts the contract size, as it does for silver return nearest_convention(scale) # Now, derive the price scale of each contract df["scale"] = df.apply( lambda row: price_scale( row["main_fraction"], row["min_price_increment"], row["min_price_increment_amount"], row["display_factor"], row["unit_of_measure_qty"], ), axis=1, ) # Finally, convert one tick and each close price into currency df["tick_value"] = df["min_price_increment"] * df["unit_of_measure_qty"] * df["scale"] df["notional"] = df["close"] * df["unit_of_measure_qty"] * df["scale"] print( df[ [ "raw_symbol", "unit_of_measure", "unit_of_measure_qty", "close", "tick_value", "scale", "notional", ] ], ) ``` ## Result ```plaintext raw_symbol unit_of_measure unit_of_measure_qty close tick_value scale notional ts_event 2026-08-19 00:00:00+00:00 6EU6 EUR 125000.0 1.168450 6.250 1.00 146056.250 2026-08-19 00:00:00+00:00 CLV6 BBL 1000.0 84.300000 10.000 1.00 84300.000 2026-08-19 00:00:00+00:00 ESU6 IPNT 50.0 7740.000000 12.500 1.00 387000.000 2026-08-19 00:00:00+00:00 GCZ6 TRYOZ 100.0 4569.200000 10.000 1.00 456920.000 2026-08-19 00:00:00+00:00 HEV6 LBS 40000.0 81.500000 10.000 0.01 32600.000 2026-08-19 00:00:00+00:00 MZCZ6 BU 500.0 498.000000 2.500 0.01 2490.000 2026-08-19 00:00:00+00:00 SIZ6 TRYOZ 5000.0 67.855000 25.000 1.00 339275.000 2026-08-19 00:00:00+00:00 ZCZ6 BU 5000.0 498.250000 12.500 0.01 24912.500 2026-08-19 00:00:00+00:00 ZLZ6 LBS 60000.0 69.860000 6.000 0.01 41916.000 2026-08-19 00:00:00+00:00 ZNZ6 USD 100000.0 108.578125 15.625 0.01 108578.125 ``` --- Source: https://databento.com/docs/examples/order-book?lang=python ## Order book | Title | Subtitle | Image | Supported languages | | [Types of order book events](https://databento.com/docs/examples/order-book/order-actions.md) | Learn how Databento normalizes MBO data from an order's life cycle (e.g., **A**dd, **F**ill). | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, C++, Rust, Historical | | [State management of resting orders](https://databento.com/docs/examples/order-book/order-tracking.md) | Manage state of all orders to calculate BBO. Precursor to order book construction. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, C++, Rust, Historical | | [Limit order book construction](https://databento.com/docs/examples/order-book/limit-order-book.md) | A complete order book implementation. Use MBO data to build the limit order book. | ![example1](https://databento.com/docs/limit-order-book.svg.md) | Python, C++, Rust, Historical | | [Microprice and book imbalance](https://databento.com/docs/examples/order-book/microprice.md) | Common predictors using order book data. | ![example2](https://databento.com/docs/microprice-thumbnail.svg.md) | Python, Historical | | [Queue position of an order](https://databento.com/docs/examples/order-book/queue-position.md) | Use MBO data to get an accurate position of any order within a price level. | ![Image placeholder](https://databento.com/docs/queue-position-thumbnail.svg.md) | Python, Historical | --- Source: https://databento.com/docs/examples/order-book/order-actions?lang=python # Types of order book events ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve individual order events and process all events for a single order. This will allow us to track the life cycle of an order, such as when it was added to the order book and when it was removed. ## MBO schemas We'll use the [MBO schema](https://databento.com/docs/schemas/mbo.md) for this example. This is the most granular schema which contains every individual order event keyed by its order ID. ## Example ```python import databento as db # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request a minute of MBO data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2022-08-26T10:59:00", end="2022-08-26T11:00:00", symbols="ES.v.0", stype_in="continuous", schema="mbo", ) df = data.to_df() # Now, we will inspect the history of a particular order ORDER_ID = 6410543150678 order_df = df.loc[df["order_id"] == ORDER_ID] # Finally, we will process each event for this order print(f"events for order_id: {ORDER_ID}") for index, row in order_df.iterrows(): event = (row["price"], row["size"]) action = row["action"] if action == "A": # Add action print(index, *event, "[A]dded", sep="\t") elif action == "C": # Cancel action print(index, *event, "[C]anceled", sep="\t") elif action == "M": # Modify action print(index, *event, "[M]odified", sep="\t") elif action == "R": # Book clear action print(index, *event, "clea[R]ed", sep="\t") elif action == "T": # Trade action print(index, *event, "[T]raded", sep="\t") elif action == "F": # Fill action print(index, *event, "[F]illed", sep="\t") elif action == "N": # None action print(index, *event, "[N]one", sep="\t") ``` ## Result ```plaintext events for order_id: 6410543150678 2022-08-26 10:59:37.217088184+00:00 4184.75 14 [A]dded 2022-08-26 10:59:50.544746948+00:00 4184.75 7 [F]illed 2022-08-26 10:59:50.544805832+00:00 4184.75 7 [M]odified 2022-08-26 10:59:50.544805832+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544805832+00:00 4184.75 6 [M]odified 2022-08-26 10:59:50.544880884+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544880884+00:00 4184.75 5 [M]odified 2022-08-26 10:59:50.544927788+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544927788+00:00 4184.75 4 [M]odified 2022-08-26 10:59:50.544927788+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544927788+00:00 4184.75 3 [M]odified 2022-08-26 10:59:50.545075499+00:00 4184.75 3 [F]illed 2022-08-26 10:59:50.545093643+00:00 4184.75 3 [C]anceled ``` --- Source: https://databento.com/docs/examples/order-book/order-tracking?lang=python # State management of resting orders ## Overview This example shows how to track orders and update them according to the different MBO actions. Order tracking is simpler to implement than a full [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md): it keeps track of all resting orders, but not the book levels. The book levels such as the top of book (BBO) can be constructed from a map of resting orders, but it requires inefficiently iterating over all the orders. > **Info** > > This example works for a single instrument. > To use it with multiple instruments, you will need to extend it to keep separate books for each instrument. ## Order events | Name | Value | Action | | Add | `A` | Insert a new order into the book. | | Modify | `M` | Change an order's price and/or size. | | Cancel | `C` | Fully or partially cancel an order from the book. | | Clear | `R` | Remove all resting orders for the instrument. | | Trade | `T` | An aggressing order traded. Does not affect the book. | | Fill | `F` | A resting order was filled. Does not affect the book. | | None | `N` | No action: does not affect the book, but may carry `flags` or other information. | Trade and Fill actions do not affect the book because all fills will be accompanied by cancel actions that do update the book. ## Handling F_LAST A single event from a publisher can be normalized into multiple records. Databento sets the flag `F_LAST` to denote the last record for an instrument in an event. The book state should only be examined _after_ a record with `F_LAST` set. ## Example ```python from dataclasses import dataclass, field import databento as db OrderId = int UnixTimestamp = int @dataclass(slots=True) class Order: side: str price: int size: int ts_event: UnixTimestamp @dataclass(slots=True) class PriceLevel: price: int | None = None size: int = 0 count: int = 0 @dataclass(slots=True) class Book: orders: dict[OrderId, Order] = field(default_factory=dict) def bbo(self) -> tuple[PriceLevel, PriceLevel]: best_ask = PriceLevel() best_bid = PriceLevel() for order in self.orders.values(): if order.side == "A": if best_ask.price is None or best_ask.price > order.price: best_ask = PriceLevel( price=order.price, size=order.size, count=1, ) elif best_ask.price == order.price: best_ask.size += order.size best_ask.count += 1 elif order.side == "B": if best_bid.price is None or best_bid.price < order.price: best_bid = PriceLevel( price=order.price, size=order.size, count=1, ) elif best_bid.price == order.price: best_bid.size += order.size best_bid.count += 1 return best_bid, best_ask def apply( self, ts_event: UnixTimestamp, action: str, side: str, order_id: int, price: int, size: int, flags: db.RecordFlags, ) -> None: # Trade, Fill, or None: no change if action in ("T", "F", "N"): return # Clear book: remove all resting orders if action == "R": self.orders.clear() # Add: insert a new order elif action == "A": # For top-of-book publishers, remove previous order associated with this side if flags & db.RecordFlags.F_TOB: self.orders = {i: o for i, o in self.orders.items() if o.side != side} # UNDEF_PRICE indicates the price level was removed. There's no new level to add if price == db.UNDEF_PRICE: return self.orders[order_id] = Order(side, price, size, ts_event) # Cancel: partially or fully cancel some size from a resting order elif action == "C": existing_order = self.orders[order_id] assert existing_order.size >= size existing_order.size -= size # If the full size is cancelled, remove the order from the book if existing_order.size == 0: self.orders.pop(order_id) # Modify: change the price and/or size of a resting order elif action == "M": existing_order = self.orders[order_id] # The order loses its priority if the price changes or the size increases if existing_order.price != price or existing_order.size < size: existing_order.ts_event = ts_event existing_order.size = size existing_order.price = price # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request MBO data starting from the beginning of pre-market trading hours data = client.timeseries.get_range( dataset="XNAS.ITCH", start="2022-08-26T08:00:00", end="2022-08-26T14:30:00", symbols="MSFT", schema="mbo", ) # Then, we replay the data, updating the book with each record book = Book() book_is_ready = False for mbo in data: book.apply(mbo.ts_event, mbo.action, mbo.side, mbo.order_id, mbo.price, mbo.size, mbo.flags) book_is_ready = mbo.flags & db.RecordFlags.F_LAST # Finally we inspect the final BBO if book_is_ready: best_bid, best_ask = book.bbo() print(f"Best ask {float(best_ask.price) / db.FIXED_PRICE_SCALE} × {best_ask.size}") print(f"Best bid {float(best_bid.price) / db.FIXED_PRICE_SCALE} × {best_bid.size}") ``` ## Result ```plaintext Best ask 276.07 × 62 Best bid 276.01 × 19 ``` --- Source: https://databento.com/docs/examples/order-book/limit-order-book?lang=python # Limit order book construction ## Overview This example demonstrates a complete limit order book implementation for Databento's MBO normalization schema. It handles multiple instruments, here `GOOG` and `GOOGL`, across multiple venues that make up Databento Equities Basic: - Investors Exchange (`IEXG`) - NYSE Texas (`XCHI`) - NYSE National (`XCIS`) We use `publisher_id` to disambiguate between venues. While this example uses the Historical API, the same implementation can be used with live data by forwarding every `MBOMsg` to `market.apply`. > **See also** > > It's recommended to first start with the [Order tracking example](https://databento.com/docs/examples/order-book/order-tracking.md#order-events), which explains each event more in depth. ## Design Orders are stored in a `dict` for quick retrieval. Each side of the book is a `SortedDict` to support both random access by price to add, modify, or cancel an order, as well as retrieving the lowest or highest price level for determining the BBO. The values in the `SortedDict` represent one level and are lists of orders sorted by priority. Investors Exchange and NYSE National only provide top-of-book information meaning their MBO data is [normalized differently](https://databento.com/docs/conventions/common-fields-enums-types.md#top-of-book-datasets). Each **A**dd message indicates the new top-of-book and replaces all levels on that side. Add messages with an `UNDEF_PRICE` price indicate there's no depth for that side of the book. The final special case comes when calculating the number of orders for a level since top-of-book updates don't correspond with individual orders. ## Dependencies This example takes advantage of the [sortedcontainers](https://grantjenks.com/docs/sortedcontainers/) package for performant lists and dicts. The `SortedDict` hybrid data structures balances the performance of random access to handle order book events with the typical pattern of checking the top of book. ## Example ```python from __future__ import annotations import os from collections import defaultdict from dataclasses import dataclass, field from itertools import takewhile import databento as db from sortedcontainers import SortedDict @dataclass(slots=True) class PriceLevel: price: int size: int = 0 count: int = 0 def __str__(self) -> str: price = self.price / db.FIXED_PRICE_SCALE return f"{self.size:4} @ {price:6.2f} | {self.count:2} order(s)" @dataclass(slots=True) class LevelOrders: price: int orders: list[db.MBOMsg] = field(default_factory=list, compare=False) def __bool__(self) -> bool: return bool(self.orders) @property def level(self) -> PriceLevel: return PriceLevel( price=self.price, count=sum(1 for o in self.orders if not o.flags & db.RecordFlags.F_TOB), size=sum(o.size for o in self.orders), ) @dataclass(slots=True) class Book: orders_by_id: dict[int, db.MBOMsg] = field(default_factory=dict) offers: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) bids: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) def bbo(self) -> tuple[PriceLevel | None, PriceLevel | None]: return self.get_bid_level(), self.get_ask_level() def get_bid_level(self, idx: int = 0) -> PriceLevel | None: if self.bids and len(self.bids) > idx: # Reverse for bids to get highest prices first return self.bids.peekitem(-(idx + 1))[1].level return None def get_ask_level(self, idx: int = 0) -> PriceLevel | None: if self.offers and len(self.offers) > idx: return self.offers.peekitem(idx)[1].level return None def get_bid_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "B").level except KeyError: return None def get_ask_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "A").level except KeyError: return None def get_order(self, id: int) -> db.MBOMsg | None: return self.orders_by_id.get(id) def get_queue_pos(self, id: int) -> int | None: order = self.get_order(id) if not order: return None level = self._get_level(order.price, order.side) return sum(order.size for order in takewhile(lambda o: o.order_id != id, level.orders)) def get_snapshot(self, level_count: int = 1) -> list[db.BidAskPair]: snapshots = [] for level in range(level_count): ba_pair = db.BidAskPair() bid = self.get_bid_level(level) if bid: ba_pair.bid_px = bid.price ba_pair.bid_sz = bid.size ba_pair.bid_ct = bid.count ask = self.get_ask_level(level) if ask: ba_pair.ask_px = ask.price ba_pair.ask_sz = ask.size ba_pair.ask_ct = ask.count snapshots.append(ba_pair) return snapshots def apply(self, mbo: db.MBOMsg) -> None: # Trade, Fill, or None: no change if mbo.action in ("T", "F", "N"): return # Clear book: remove all resting orders if mbo.action == "R": self._clear() return # side=N is only valid with Trade, Fill, and Clear actions assert mbo.side in ("A", "B") # UNDEF_PRICE indicates the book level should be removed if mbo.price == db.UNDEF_PRICE and mbo.flags & db.RecordFlags.F_TOB: self._side_levels(mbo.side).clear() return # Add: insert a new order if mbo.action == "A": self._add(mbo) # Cancel: partially or fully cancel some size from a resting order elif mbo.action == "C": self._cancel(mbo) # Modify: change the price and/or size of a resting order elif mbo.action == "M": self._modify(mbo) else: raise ValueError(f"Unknown action={mbo.action}") def _clear(self) -> None: self.orders_by_id.clear() self.offers.clear() self.bids.clear() def _add(self, mbo: db.MBOMsg) -> None: if mbo.flags & db.RecordFlags.F_TOB: levels = self._side_levels(mbo.side) levels.clear() levels[mbo.price] = LevelOrders(price=mbo.price, orders=[mbo]) else: level = self._get_or_insert_level(mbo.price, mbo.side) assert mbo.order_id not in self.orders_by_id self.orders_by_id[mbo.order_id] = mbo level.orders.append(mbo) def _cancel(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id[mbo.order_id] level = self._get_level(mbo.price, mbo.side) assert order.size >= mbo.size order.size -= mbo.size # If the full size is cancelled, remove the order from the book if order.size == 0: self.orders_by_id.pop(mbo.order_id) level.orders.remove(order) # If the level is now empty, remove it from the book if not level: self._remove_level(mbo.price, mbo.side) def _modify(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id.get(mbo.order_id) if order is None: # If order not found, treat it as an add self._add(mbo) return assert order.side == mbo.side, f"Order {order} changed side to {mbo.side}" level = self._get_level(order.price, order.side) if order.price != mbo.price: # Changing price loses priority level.orders.remove(order) if not level: self._remove_level(order.price, mbo.side) level = self._get_or_insert_level(mbo.price, mbo.side) level.orders.append(mbo) elif order.size < mbo.size: # Increasing size loses priority level.orders.remove(order) level.orders.append(mbo) else: # Update in place level.orders[level.orders.index(order)] = mbo self.orders_by_id[mbo.order_id] = mbo def _side_levels(self, side: str) -> SortedDict: if side == "A": return self.offers if side == "B": return self.bids raise ValueError(f"Invalid {side =}") def _get_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price not in levels: raise KeyError(f"No price level found for {price =} and {side =}") return levels[price] def _get_or_insert_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price in levels: return levels[price] level = LevelOrders(price=price) levels[price] = level return level def _remove_level(self, price: int, side: str) -> None: levels = self._side_levels(side) levels.pop(price) @dataclass(slots=True) class Market: books: defaultdict[int, defaultdict[int, Book]] = field( default_factory=lambda: defaultdict(lambda: defaultdict(Book)), ) def get_books_by_pub(self, instrument_id: int) -> defaultdict[int, Book]: return self.books[instrument_id] def get_book(self, instrument_id: int, publisher_id: int) -> Book: return self.books[instrument_id][publisher_id] def bbo( self, instrument_id: int, publisher_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: return self.books[instrument_id][publisher_id].bbo() def aggregated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: agg_bbo: list[PriceLevel | None] = [None, None] # max for bids, min for asks for idx, reducer in [(0, max), (1, min)]: all_best = [b.bbo()[idx] for b in self.books[instrument_id].values()] all_best = [b for b in all_best if b] if not all_best: continue best_price = reducer(b.price for b in all_best) best = [b for b in all_best if b.price == best_price] agg_bbo[idx] = PriceLevel( price=best_price, size=sum(b.size for b in best), count=sum(b.count for b in best), ) return tuple(agg_bbo) def apply(self, mbo: db.MBOMsg) -> None: book = self.books[mbo.instrument_id][mbo.publisher_id] book.apply(mbo) if __name__ == "__main__": # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request MBO data starting from the beginning of pre-market trading hours # or load the file if we've already downloaded it. data_path = "dbeq-basic-20240403.mbo.dbn.zst" if os.path.exists(data_path): data = db.DBNStore.from_file(data_path) else: data = client.timeseries.get_range( dataset="DBEQ.BASIC", start="2024-04-03T08:00:00", end="2024-04-03T14:00:00", symbols=["GOOG", "GOOGL"], schema="mbo", path=data_path, ) # Then we parse the symbology into a more usable format instrument_map = db.common.symbology.InstrumentMap() instrument_map.insert_metadata(data.metadata) # Finally, we iterate over each book update market = Market() for mbo in data: # And apply it market.apply(mbo) # If it's the last update in an event, print the state of the aggregated book if mbo.flags & db.RecordFlags.F_LAST: symbol = instrument_map.resolve(mbo.instrument_id, mbo.pretty_ts_recv.date()) or "" print(f"{symbol} Aggregated BBO | {mbo.pretty_ts_recv}") best_bid, best_offer = market.aggregated_bbo(mbo.instrument_id) print(f" {best_offer}") print(f" {best_bid}") ``` ## Results ```plaintext GOOGL Aggregated BBO | 2024-04-03 11:00:00.903881953+00:00 None 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.903887026+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.914731635+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.925739296+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.925744379+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:03:55.585836619+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:03:55.585841782+00:00 200 @ 155.40 | 1 order(s) 200 @ 152.80 | 1 order(s) ... ``` --- Source: https://databento.com/docs/examples/order-book/microprice?lang=python # Microprice and book imbalance In this example, we'll explore different ways to estimate the fair price of a financial instrument using market data. Specifically, we'll examine microprice and book imbalance to better understand order book dynamics. ## Overview We'll use the [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md) ("L2" data) which shows all trades, as well as all updates to the top 10 levels of the order book. We'll plot various ways you can use microprice or book imbalance when analyzing price action for an instrument. The standard formula for microprice, which only uses the top-of-book, is shown below: \[\text{Microprice} = \frac{s_b p_a + s_a p_b}{s_b + s_a}\] where - \(p_b\) and \(p_a\) are the best bid and ask prices - \(s_b\) and \(s_a\) are the sizes at the best bid and ask price levels To calculate book imbalance over N levels, we will use the following formula: \[Imb_N = \frac{\sum_{i=1}^{N} s_{b,i} - \sum_{i=1}^{N} s_{a,i}}{\sum_{i=1}^{N} s_{b,i} + \sum_{i=1}^{N} s_{a,i}}\] where - \(s_{b,i}\) is the size at the i-th best bid price level - \(s_{a,i}\) is the size at the i-th best ask price level - \(N\) is the number of levels to aggregate (e.g., N=5 for top 5 levels) - \(Imb_N\) ranges from -1 to 1 ## Example ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd def calc_microprice( df_mbp10: pd.DataFrame, ) -> pd.DataFrame: """ Calculate top-of-book microprice. """ numerator = (df_mbp10["bid_sz_00"] * df_mbp10["ask_px_00"]) + ( df_mbp10["ask_sz_00"] * df_mbp10["bid_px_00"] ) denominator = df_mbp10["bid_sz_00"] + df_mbp10["ask_sz_00"] df_mbp10["Microprice"] = numerator / denominator return df_mbp10 def calc_book_imbalance( df_mbp10: pd.DataFrame, n: int, ) -> pd.DataFrame: """ Calculate book imbalance over N levels of the order book. If a positive value, there is more size on the bid than the offer. If a negative value, there is more size on the offer than the bid. If the size is equal, or there is no size on either the bid or the ask, the value will be 0. """ total_bid_size = df_mbp10[[f"bid_sz_{i:02}" for i in range(n)]].sum(axis=1).astype("int64") total_ask_size = df_mbp10[[f"ask_sz_{i:02}" for i in range(n)]].sum(axis=1).astype("int64") s = ((total_bid_size - total_ask_size) / (total_bid_size + total_ask_size)).fillna(0) df_mbp10[f"Book imbalance ({n})"] = s return df_mbp10 def plot_microprice( df_mbp10: pd.DataFrame, symbol: str, ) -> None: """ Plot microprice against midprice and trades. """ df_trades = df_mbp10[df_mbp10["action"] == db.Action.TRADE] df_mbp10["Midprice"] = (df_mbp10["bid_px_00"] + df_mbp10["ask_px_00"]) / 2 df_mbp10 = calc_microprice(df_mbp10) y_tick_range = np.arange( min(df_mbp10["bid_px_00"].min(), df_trades["price"].min()), max(df_mbp10["ask_px_00"].max() + tick_size, df_trades["price"].max() + tick_size), tick_size, ) # Plot df_mbp10.plot( y=["Midprice", "Microprice"], title=f"{symbol} Midprice vs. Microprice", xlabel="Time (UTC)", ylabel="Price", drawstyle="steps-post", ) # Plot trades plt.scatter( df_trades.index, df_trades["price"], color="C2", marker=".", s=100, label="Trades", ) plt.yticks(y_tick_range) plt.legend() plt.show() def plot_book_imbalance( df_mbp10: pd.DataFrame, symbol: str, n: int = 10, ) -> None: """ Plot book imbalance over N levels of the book, along with trades. """ df_trades = df_mbp10[df_mbp10["action"] == db.Action.TRADE] df_mbp10 = calc_book_imbalance(df_mbp10, n) # Plot ax1 = df_mbp10[f"Book imbalance ({n})"].plot( drawstyle="steps-post", title=f"{symbol} Book Imbalance ({n} {'level' if n == 1 else 'levels'})", xlabel="Time (UTC)", ylabel="Imbalance", ) ax1.set_ylim(-1, 1) # Plot trades df_trades["is_buy"] = np.where(df_trades["side"] == "B", 1, -1) ax2 = ax1.twinx() ax2.scatter( df_trades.index, df_trades["is_buy"], color="C2", marker=".", s=150, label="Trades", ) lines1, labels1 = ax1.get_legend_handles_labels() lines2, labels2 = ax2.get_legend_handles_labels() ax2.legend(lines1 + lines2, labels1 + labels2) ax2.grid(False) ax2.set_yticks([-1, 1]) ax2.set_yticklabels(["Sell", "Buy"]) ax2.set_ylabel("Trade Side") plt.show() if __name__ == "__main__": # Set parameters dataset = "XNAS.ITCH" symbol = "TSLA" tick_size = 0.01 start = "2025-09-10T14:00:00" end = "2025-09-10T14:00:05" # Create historical client client = db.Historical("$YOUR_API_KEY") # Download MBP-10 data and convert to a DataFrame mbp10_df = client.timeseries.get_range( dataset=dataset, symbols=symbol, start=start, end=end, schema="mbp-10", ).to_df() # Plot top-of-book microprice plot_microprice(mbp10_df, symbol) # Plot book imbalance with 10 levels of the orderbook plot_book_imbalance(mbp10_df, symbol, 10) ``` ## Results First, we'll take a look at the microprice and compare it to the midprice. Microprice will take into account the size on the bid and offer. ![Microprice](https://databento.com/docs/microprice-0.svg.md) Since microprice is sensitive to changes in the orderbook, it may offer some predictive power for future price movements. Below, we'll take a look at book imbalance, which is a calculated indicator between -1 and 1. Book imbalance can look at multiple levels of the order book. Positive values mean there is more cumulative size on the bid than the offer, negative values mean there is more cumulative size on the offer, and a zero value means the bid and offer have the same total size. ![Book imbalance](https://databento.com/docs/microprice-1.svg.md) --- Source: https://databento.com/docs/examples/order-book/queue-position?lang=python # Queue position of an order > **See also** > > It's recommended to first start with the [Order tracking](https://databento.com/docs/examples/order-book/order-tracking.md) and [Limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) examples. ## Overview In this example, we will use a complete limit order book to track the queue position of an order. This example assumes a [First-In-First-Out](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457218479/Supported+Matching+Algorithms#FIFO) (FIFO) matching algorithm is being utilized. FIFO is the most common algorithm used in modern matching engines. When market participants have active orders in different instruments and contract months, knowing exact queue positions can be important in managing execution risk. The ability to track orders as they move throughout the limit order book can be utilized as a trading signal in automated strategies. ## Design We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) ("L3" data), which assigns a unique ID to every order in the book, to exactly track this queue position. The logic for this calculation is a straightforward extension of our [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) example. We will also use the [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md) ("L2" data) to estimate queue position, assuming we did not have access to MBO data. Suppose \(p\left(x\right)\) denotes the probability of a cancel in front when your order is positioned \(0 \leq x \leq 1\) in the queue. - When you first join the queue, \(p\left(1\right)=1\), since all cancels must happen in front of you. - When you are at the front of the queue, \(p\left(0\right)=0\), since all cancels must happen from behind. A naive assumption is that orders cancel uniformly through the queue, i.e. \(p\left(x\right)=x\). In reality, cancels are more likely to happen behind you vs. in front of you, since the orders in front of you have value from their time priority. We can model this cancellation bias with a parameter \(k\), ranging from -1 to 1. \(k=0\) results in an unbiased, uniform distribution. \[p_k\left(x\right)= \begin{cases} x^{1+k} & \text{if } k < 0 \\ x & \text{if } k = 0 \\ 1-\left(1-x\right)^{1-k} & \text{if } k > 0 \end{cases} \] We'll take a look at this uniform cancellation estimation (\(k=0\)), as well as a few weighted estimations that assumes a higher probability of cancels from behind (\(k=-0.8\) and \(k=-0.95\)). ## Example ```python from __future__ import annotations import os import random from collections import defaultdict from dataclasses import dataclass, field from itertools import takewhile import databento as db import matplotlib.pyplot as plt import pandas as pd from sortedcontainers import SortedDict @dataclass(slots=True) class PriceLevel: price: int size: int = 0 count: int = 0 def __str__(self) -> str: price = self.price / db.FIXED_PRICE_SCALE return f"{self.size:4} @ {price:6.2f} | {self.count:2} order(s)" @dataclass(slots=True) class LevelOrders: price: int orders: list[db.MBOMsg] = field(default_factory=list, compare=False) def __bool__(self) -> bool: return bool(self.orders) @property def level(self) -> PriceLevel: return PriceLevel( price=self.price, count=sum(1 for o in self.orders if not o.flags & db.RecordFlags.F_TOB), size=sum(o.size for o in self.orders), ) @dataclass(slots=True) class Book: orders_by_id: dict[int, db.MBOMsg] = field(default_factory=dict) offers: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) bids: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) def bbo(self) -> tuple[PriceLevel | None, PriceLevel | None]: return self.get_bid_level(), self.get_ask_level() def get_bid_level(self, idx: int = 0) -> PriceLevel | None: if self.bids and len(self.bids) > idx: # Reverse for bids to get highest prices first return self.bids.peekitem(-(idx + 1))[1].level return None def get_ask_level(self, idx: int = 0) -> PriceLevel | None: if self.offers and len(self.offers) > idx: return self.offers.peekitem(idx)[1].level return None def get_bid_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "B").level except KeyError: return None def get_ask_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "A").level except KeyError: return None def get_order(self, id: int) -> db.MBOMsg | None: return self.orders_by_id.get(id) def get_queue_pos(self, id: int) -> int | None: order = self.get_order(id) if not order: return None level = self._get_level(order.price, order.side) return sum(order.size for order in takewhile(lambda o: o.order_id != id, level.orders)) def get_snapshot(self, level_count: int = 1) -> list[db.BidAskPair]: snapshots = [] for level in range(level_count): ba_pair = db.BidAskPair() bid = self.get_bid_level(level) if bid: ba_pair.bid_px = bid.price ba_pair.bid_sz = bid.size ba_pair.bid_ct = bid.count ask = self.get_ask_level(level) if ask: ba_pair.ask_px = ask.price ba_pair.ask_sz = ask.size ba_pair.ask_ct = ask.count snapshots.append(ba_pair) return snapshots def apply(self, mbo: db.MBOMsg) -> None: # Trade, Fill, or None: no change if mbo.action in ("T", "F", "N"): return # Clear book: remove all resting orders if mbo.action == "R": self._clear() return # side=N is only valid with Trade, Fill, and Clear actions assert mbo.side in ("A", "B") # UNDEF_PRICE indicates the book level should be removed if mbo.price == db.UNDEF_PRICE and mbo.flags & db.RecordFlags.F_TOB: self._side_levels(mbo.side).clear() return # Add: insert a new order if mbo.action == "A": self._add(mbo) # Cancel: partially or fully cancel some size from a resting order elif mbo.action == "C": self._cancel(mbo) # Modify: change the price and/or size of a resting order elif mbo.action == "M": self._modify(mbo) else: raise ValueError(f"Unknown action={mbo.action}") def _clear(self) -> None: self.orders_by_id.clear() self.offers.clear() self.bids.clear() def _add(self, mbo: db.MBOMsg) -> None: if mbo.flags & db.RecordFlags.F_TOB: levels = self._side_levels(mbo.side) levels.clear() levels[mbo.price] = LevelOrders(price=mbo.price, orders=[mbo]) else: level = self._get_or_insert_level(mbo.price, mbo.side) assert mbo.order_id not in self.orders_by_id self.orders_by_id[mbo.order_id] = mbo level.orders.append(mbo) def _cancel(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id[mbo.order_id] level = self._get_level(mbo.price, mbo.side) assert order.size >= mbo.size order.size -= mbo.size # If the full size is cancelled, remove the order from the book if order.size == 0: self.orders_by_id.pop(mbo.order_id) level.orders.remove(order) # If the level is now empty, remove it from the book if not level: self._remove_level(mbo.price, mbo.side) def _modify(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id.get(mbo.order_id) if order is None: # If order not found, treat it as an add self._add(mbo) return assert order.side == mbo.side, f"Order {order} changed side to {mbo.side}" level = self._get_level(order.price, order.side) if order.price != mbo.price: # Changing price loses priority level.orders.remove(order) if not level: self._remove_level(order.price, mbo.side) level = self._get_or_insert_level(mbo.price, mbo.side) level.orders.append(mbo) elif order.size < mbo.size: # Increasing size loses priority level.orders.remove(order) level.orders.append(mbo) else: # Update in place level.orders[level.orders.index(order)] = mbo self.orders_by_id[mbo.order_id] = mbo def _side_levels(self, side: str) -> SortedDict: if side == "A": return self.offers if side == "B": return self.bids raise ValueError(f"Invalid {side =}") def _get_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price not in levels: raise KeyError(f"No price level found for {price =} and {side =}") return levels[price] def _get_or_insert_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price in levels: return levels[price] level = LevelOrders(price=price) levels[price] = level return level def _remove_level(self, price: int, side: str) -> None: levels = self._side_levels(side) levels.pop(price) @dataclass(slots=True) class Market: books: defaultdict[int, defaultdict[int, Book]] = field( default_factory=lambda: defaultdict(lambda: defaultdict(Book)), ) def get_books_by_pub(self, instrument_id: int) -> defaultdict[int, Book]: return self.books[instrument_id] def get_book(self, instrument_id: int, publisher_id: int) -> Book: return self.books[instrument_id][publisher_id] def bbo( self, instrument_id: int, publisher_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: return self.books[instrument_id][publisher_id].bbo() def aggregated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: agg_bbo: list[PriceLevel | None] = [None, None] # max for bids, min for asks for idx, reducer in [(0, max), (1, min)]: all_best = [b.bbo()[idx] for b in self.books[instrument_id].values()] all_best = [b for b in all_best if b] if not all_best: continue best_price = reducer(b.price for b in all_best) best = [b for b in all_best if b.price == best_price] agg_bbo[idx] = PriceLevel( price=best_price, size=sum(b.size for b in best), count=sum(b.count for b in best), ) return tuple(agg_bbo) def apply(self, mbo: db.MBOMsg) -> None: book = self.books[mbo.instrument_id][mbo.publisher_id] book.apply(mbo) def apply_bias(x: float, k: float) -> float: """ Apply bias to the random number `x` in the range [0, 1]. -1 < k < 0 results in a bias towards 1. k == 0 results in a uniform distribution 0 < k < 1 results in a bias towards 0 k == -1 always returns 1 k == 1 always returns 0 """ assert 0 <= x <= 1 # Random number between 0 and 1 assert -1 <= k <= 1 # Bias between -1 and 1 if k <= 0: return x ** (1 + k) else: return 1 - (1 - x) ** (1 - k) if __name__ == "__main__": # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" start_date = "2025-02-12" end_date = "2025-02-13" symbol = "ES.v.0" # For this example, we will retroactively pick an order to track order_id = 6414447712675 order_add_time = 1739375799442261697 order_fill_time = 1739375974175277429 order_price = 6064000000000 order_size = 1 # Now we will request MBO data. mbo_data_path = "glbx-mdp3-20250212.mbo.dbn.zst" if os.path.exists(mbo_data_path): mbo_data = db.DBNStore.from_file(mbo_data_path) else: mbo_data = client.timeseries.get_range( dataset=dataset, start=start_date, end=end_date, symbols=symbol, stype_in="continuous", schema="mbo", path=mbo_data_path, ) # Next, we will record all updates at this price while the order was active price_level_msgs: list[dict] = [] book = Book() for mbo_msg in mbo_data: # Process MBO msg book.apply(mbo_msg) # Only record updates when order is active if order_add_time <= mbo_msg.ts_event < order_fill_time: # Wait for end of event if mbo_msg.flags & db.RecordFlags.F_LAST: # Get total size at price and actual queue position price_level_msg = { "ts_event": mbo_msg.ts_event, "Depth at price level": book.get_ask_level_by_px(order_price).size, "Queue position": book.get_queue_pos(order_id), } price_level_msgs.append(price_level_msg) # Construct a DataFrame from MBO updates df = pd.DataFrame.from_records(price_level_msgs) df["Order lifetime (s)"] = (df["ts_event"] - df["ts_event"][0]) / 1e9 df = df.set_index("Order lifetime (s)") # Plot actual queue position using MBO data ax = df[["Depth at price level", "Queue position"]].plot( drawstyle="steps-post", ylabel="Contracts", ) def generate_simulations(df: pd.DataFrame, k: float) -> pd.Series: """ Generate simulations for estimated queue position. """ # Run simulations for estimated queue position sim_list = [df.index.to_list()] # Order is starting at the back of the queue total_size_list = df["Depth at price level"].to_list() start_queue_pos = total_size_list[0] # Run 100 simulations for i in range(100): c_total_size = start_queue_pos # Current size at price level c_queue_pos = start_queue_pos - order_size # Current estimated queue position estimated_queue_pos = [] for total_size in total_size_list: # Decrease in size at price level. A cancel occurred if total_size < c_total_size: cancel_size = c_total_size - total_size others_size = total_size - order_size # Size of rest of level random_number = apply_bias(random.random(), k) if random_number < (c_queue_pos / others_size): c_queue_pos -= cancel_size c_queue_pos = min(max(c_queue_pos, 0), others_size) c_total_size = total_size estimated_queue_pos.append(c_queue_pos) sim_list.append(estimated_queue_pos) col_names = [f"sim_{i}" for i in range(len(sim_list))] df_sim = pd.DataFrame(dict(zip(col_names, sim_list))) df_sim = df_sim.set_index(df.index) return df_sim.quantile(0.50, axis=1) # Next, we will also download MBP-10 data for estimating queue position mbp10_data_path = "glbx-mdp3-20250212.mbp-10.dbn.zst" if os.path.exists(mbp10_data_path): mbp10_data = db.DBNStore.from_file(mbp10_data_path) else: mbp10_data = client.timeseries.get_range( dataset=dataset, start=start_date, end=end_date, symbols=symbol, stype_in="continuous", schema="mbp-10", path=mbp10_data_path, ) # Now we will use the MBP-10 data to record similar information as above mbp10_price_level_msgs: list[dict] = [] for mbp10_msg in mbp10_data: if order_add_time <= mbp10_msg.ts_event < order_fill_time: for level_pair in mbp10_msg.levels: if level_pair.ask_px == order_price: # Get total size at price mbp_price_level_msg = { "ts_event": mbp10_msg.ts_event, "Depth at price level": level_pair.ask_sz, } mbp10_price_level_msgs.append(mbp_price_level_msg) # Construct a DataFrame from MBP-10 updates df_mbp10 = pd.DataFrame.from_records(mbp10_price_level_msgs) df_mbp10["Order lifetime (s)"] = (df_mbp10["ts_event"] - df_mbp10["ts_event"][0]) / 1e9 df_mbp10 = df_mbp10.set_index("Order lifetime (s)") # Run simulations using various weights for k in (0, -0.8, -0.95): df_mbp10[f"Estimated queue position (k={k})"] = generate_simulations(df_mbp10, k) # Add estimated queue position using MBP-10 data to plot df_mbp10.filter(regex="Estimated", axis=1).plot( ax=ax, drawstyle="steps-post", title="Actual vs. estimated queue position", style=["C3", "C5", "C2"], ylim=(0, 85), ) plt.show() ``` ## Results We can show how the exact queue position of the order within its price level decreases over time (orange), even as orders are added and removed from the book (purple). The naive model assuming uniform cancellation (green) does a poor job estimating the position, but the biased models perform better. ![Queue position](https://databento.com/docs/queue-position-0.svg.md) --- Source: https://databento.com/docs/examples/algo-trading?lang=python ## Algorithmic trading | Title | Subtitle | Image | Supported languages | | [A high-frequency liquidity-taking strategy](https://databento.com/docs/examples/algo-trading/high-frequency.md) | Implement simulated trading (paper trading) using live data. | ![example1](https://databento.com/docs/results.png.md) | Python, Live | | [Build prediction models with machine learning](https://databento.com/docs/examples/algo-trading/machine-learning.md) | Construct alpha signals using Databento and regression models from sklearn. | ![example2](https://databento.com/docs/machine-learning-thumbnail.svg.md) | Python, Historical | | [Execution slippage and markouts](https://databento.com/docs/examples/algo-trading/execution-slippage.md) | Plot markout curves for transaction cost analysis and measuring execution quality. | ![example3](https://databento.com/docs/execution-slippage-thumbnail.svg.md) | Python, Historical | | [Matching engine latencies](https://databento.com/docs/examples/algo-trading/latency.md) | Measure matching engine and feed latency with nanosecond, PTP timestamps. | ![example4](https://databento.com/docs/latency-thumbnail.svg.md) | Python, Historical | | [Using messaging rates as a proxy for implied volatility](https://databento.com/docs/examples/algo-trading/messaging-volatility.md) | Determine and compare implied volatility (IV) and messaging rates. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Mean reversion and portfolio optimization](https://databento.com/docs/examples/algo-trading/portfolio-optimization.md) | Fetch cross-sectional returns. Build a mean reversion strategy and optimal portfolios. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Pairs trading based on cointegration](https://databento.com/docs/examples/algo-trading/pairs-trading.md) | A cross-venue pairs trading strategy between WTI (CME) and Brent (ICE) crude. | ![example7](https://databento.com/docs/spread.png.md) | Python, Historical | | [Build a real-time stock screener](https://databento.com/docs/examples/algo-trading/live-stock-screener.md) | Build a live stock screener for all U.S. equity tickers. | ![example7](https://databento.com/docs/placeholder.svg.md) | Python, Historical, Live | --- Source: https://databento.com/docs/examples/algo-trading/high-frequency?lang=python # A high-frequency liquidity-taking strategy We'll introduce some terminology for this tutorial: - A **feature** is any kind of basic independent variable that we think has some predictive value. This follows machine learning nomenclature; others may refer to this as a **predictor** or **regressor** in a statistical or econometric setting. - A **trading rule** is a hardcoded trading decision. An example of a trading rule is "If there's only 1 order left at the best offer, lift the offer; if there is only 1 order left at the best bid, hit the bid". A trading rule may be a hardcoded trading decision taken when a feature value exceeds a certain threshold. - A strategy that is based on trading rules is a **rule-based** strategy. - A **liquidity-taking** strategy takes liquidity by crossing the spread with aggressive or marketable orders. - A **high-frequency** strategy is characterized by a large number of trades. ## Book skew and trading rule The simplest type of book feature is called the book skew, which is the imbalance between resting bid depth (vb) and resting ask depth (va) at the top of the book. We can formulate this as some sort of difference between vb and va. It's convenient to scale these by their order of magnitude, so we take their log differences instead. \[\text{skew} = \log(v_b)- \log(v_a) = \log \dfrac{v_b}{v_a}\] Notice that we picked this ordering simply because it's useful to formulate features such that positive values imply that we expect prices to go up; this makes it easier to debug your strategy. Intuitively, we expect higher bid depth to imply higher buy demand and hence higher prices. We can introduce a trading rule that buys when this feature exceeds some alpha threshold `k`, and sells when it goes below some threshold `-k`. ## Minimum order quantity While there's some practical advantage to trading larger clips for a liquidity-taking strategy, we'll start with a constant trade size equal to the minimum order quantity so that we minimize slippage and market impact considerations. The minimum order quantity depends on the trading platform or market that you're on. For this toy strategy, we'll use E-mini S&P 500 (ES) futures as an example and trade in clips of 1 contract. \[\text{skew} > k \rightarrow \text{Buy 1 lot}\] \[\text{skew} < -k \rightarrow \text{Sell 1 lot}\] ## Commissions This strategy will be very sensitive to commissions due to its volume, so we'll include commissions on the estimated PnL. These commissions can be found [here](https://www.cmegroup.com/company/clearing-fees.html). ## Position and risk limits This strategy is convenient because you don't have to worry about complex order and position state management. You just let it build up whatever maximum position you want. Because we expect buys and sells to be symmetrically distributed in the long run, you will eventually get out of position. However, you might not have enough margin to build up arbitrarily large positions, so for proof of concept, we'll specify a maximum position of 10 contracts. \[\text{skew} > k\text{ and pos} < 10\text{ lots} \rightarrow \text{Buy 1 lot}\] \[\text{skew} < -k\text{ and pos} > -10\text{ lots} \rightarrow \text{Sell 1 lot}\] ## Implementation These are the parameters we have so far: > **Info** > > This example uses the instrument with raw symbol ESU3. You'll need to substitute this > with your desired symbol. ```python import math import json import sys from dataclasses import dataclass, field from decimal import Decimal import pandas as pd import databento as db API_KEY = "$YOUR_API_KEY" class Config: # Alpha threshold to buy/sell, k ALPHA_THRESHOLD: int = 1.7 # Symbol DATASET = "GLBX.MDP3" SYMBOL = "ESU3" POINT_VALUE = 50 # $50 per index point # Fees VENUE_FEES_PER_SIDE: Decimal = Decimal("0.39") CLEARING_FEES_PER_SIDE: Decimal = Decimal("0.05") FEES_PER_SIDE: Decimal = VENUE_FEES_PER_SIDE + CLEARING_FEES_PER_SIDE # Position limit POSITION_MAX: int = 10 ``` Since we're only simulating liquidity-taking at minimum size, our MBP-1 schema is sufficient. You can learn more about our MBP-1 schema [here](https://databento.com/docs/schemas/mbp-1.md). To keep this example simple, we'll assume zero round-trip latency for any orders placed. This enables us to implement a simple, online calculation of PnL for our real-time trading simulation. ```python @dataclass(slots=True) class Strategy: # Dataset dataset: str # Instrument to trade symbol: str # Current position, in contract units position: int = 0 # Number of long contract sides traded buy_qty: int = 0 # Number of short contract sides traded sell_qty: int = 0 ## Total realized buy price real_total_buy_px: Decimal = 0 ## Total realized sell price real_total_sell_px: Decimal = 0 # Total buy price to liquidate current position theo_total_buy_px: Decimal = 0 # Total sell price to liquidate current position theo_total_sell_px: Decimal = 0 # Total fees paid fees: Decimal = 0 # List to track results results: list = field(default_factory=list) def run(self) -> None: client = db.Live() client.subscribe( dataset=self.dataset, schema="mbp-1", stype_in="raw_symbol", symbols=[self.symbol], # start="2023-08-08T12:00", # Burn-in start time ) for record in client: if not isinstance(record, db.MBP1Msg): continue self.update(record) def update(self, record: db.MBP1Msg) -> None: ask_size = record.levels[0].ask_sz bid_size = record.levels[0].bid_sz ask_price = record.levels[0].ask_px / Decimal("1e9") bid_price = record.levels[0].bid_px / Decimal("1e9") skew = math.log10(bid_size) - math.log10(ask_size) mid_price = (ask_price + bid_price) / 2 # Buy/sell based when skew signal is large if skew > Config.ALPHA_THRESHOLD and self.position < Config.POSITION_MAX: self.position += 1 self.buy_qty += 1 self.real_total_buy_px += ask_price self.fees += Config.FEES_PER_SIDE elif skew < -Config.ALPHA_THRESHOLD and self.position > -Config.POSITION_MAX: self.position -= 1 self.sell_qty += 1 self.real_total_sell_px += bid_price self.fees += Config.FEES_PER_SIDE # Update prices # Fill prices are based on BBO with assumed zero latency # In practice, should be worse because of alpha decay if self.position == 0: self.theo_total_buy_px = 0 self.theo_total_sell_px = 0 elif self.position > 0: self.theo_total_sell_px = bid_price * abs(self.position) elif self.position < 0: self.theo_total_buy_px = ask_price * abs(self.position) # Compute PnL theo_pnl = ( Config.POINT_VALUE * ( self.real_total_sell_px + self.theo_total_sell_px - self.real_total_buy_px - self.theo_total_buy_px ) - self.fees ) result = { "ts_strategy": str(pd.Timestamp(record.ts_recv, tz="UTC")), "bid": f"{bid_price:0.2f}", "ask": f"{ask_price:0.2f}", "skew" : f"{skew:0.3f}", "position": self.position, "trade_ct": self.buy_qty + self.sell_qty, "fees": f"{self.fees:0.2f}", "pnl": f"{theo_pnl:0.2f}", } print(json.dumps(result, indent=4)) self.results.append(result) if __name__ == "__main__": strategy = Strategy(dataset=Config.DATASET, symbol=Config.SYMBOL) while True: try: strategy.run() except KeyboardInterrupt: df = pd.DataFrame(strategy.results) df.to_csv("strategy_log.csv", index=False) sys.exit() ``` ## Results ![Results](https://databento.com/docs/results.png.md) ## Further improvements This implementation uses our simple synchronous client. For production applications, we recommend using our [asynchronous client](https://databento.com/docs/api-live/client/aiter.md?lang=python) or [callback model](https://databento.com/docs/api-live/client/add-callback.md?lang=python). A problem encountered with the book skew is that extreme values may be influenced by spoofing. One possibility is to modify the trading rule and introduce an upper limit as follows: \[\text{skew} > k \text{ and abs(skew)} < L \text{ and pos} < 10 \text{ lots} \rightarrow \text{Buy 1 lot}\] \[\text{skew} < -k \text{ and abs(skew)} < L \text{ and pos} > -10 \text{ lots} \rightarrow \text{Sell 1 lot}\] Finally, recall that we assumed zero delay in order placement and fill. It's also important to incorporate a delay when extending this example. --- Source: https://databento.com/docs/examples/algo-trading/machine-learning?lang=python # Build prediction models with machine learning ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to build high-frequency trading signals and use them to train a machine learning model. ## MBP-10 schema We'll demonstrate this example using the [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md). The MBP-10 schema contains the top ten levels of aggregated book depth. This updates on every order book event. ## Dependencies This example will use the [scikit-learn](https://pypi.org/project/scikit-learn/) package for creating a machine learning model. The [matplotlib](https://pypi.org/project/matplotlib/) package will be used for charting. These dependencies can be installed with the following: ```bash $ pip install scikit-learn matplotlib ``` ## Example ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd from sklearn.linear_model import LinearRegression # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2023-12-06T14:30:00" end = "2023-12-06T20:33:00" # Request MBP-10 for the front-month ES contract and convert to DataFrame df = client.timeseries.get_range( dataset="GLBX.MDP3", schema="mbp-10", symbols=f"{product}.v.0", stype_in="continuous", start=start, end=end, ).to_df() # Filter out trades only df = df[df["action"] == db.Action.TRADE] # Calculate midprice returns with a forward markout of 500 trades df["mid"] = (df["bid_px_00"] + df["ask_px_00"]) / 2 df["ret_500t"] = df["mid"].shift(-500) - df["mid"] df = df.dropna() # Calculate depth imbalance on top level ('skew') df["skew"] = np.log(df.bid_sz_00) - np.log(df.ask_sz_00) # Calculate order imbalance on top ten levels ('imbalance') bid_count = df[list(df.filter(regex="bid_ct_0[0-9]"))].sum(axis=1) ask_count = df[list(df.filter(regex="ask_ct_0[0-9]"))].sum(axis=1) df["imbalance"] = np.log(bid_count) - np.log(ask_count) # Split in-sample and out-of-sample split = int(0.66 * len(df)) split -= split % 100 df_in = df.iloc[:split] df_out = df.iloc[split:] # Evaluate signal correlation corr = df_in[["skew", "imbalance", "ret_500t"]].corr() print(corr.where(np.triu(np.ones(corr.shape)).astype(bool))) reg = LinearRegression(fit_intercept=False, positive=True) # Create a model using skew only reg.fit(df_in[["skew"]], df_in["ret_500t"]) pred_skew = reg.predict(df_out[["skew"]]) # Create a model using imbalance only reg.fit(df_in[["imbalance"]], df_in["ret_500t"]) pred_imbalance = reg.predict(df_out[["imbalance"]]) # Create a model using both skew and imbalance reg.fit(df_in[["skew", "imbalance"]], df_in["ret_500t"]) pred_combined = reg.predict(df_out[["skew", "imbalance"]]) # Define a function to calculate profit and loss def get_cumulative_markout_pnl(pred): df_pnl = pd.DataFrame({"pred": pred, "ret_500t": df_out["ret_500t"].values}) df_pnl.loc[df_pnl["pred"] < 0, "ret_500t"] *= -1 df_pnl = df_pnl.sort_values(by="pred") return df_pnl["ret_500t"].cumsum().values # Collect results into a DataFrame results = pd.DataFrame( { "skew": get_cumulative_markout_pnl(pred_skew), "imbalance": get_cumulative_markout_pnl(pred_imbalance), "combined": get_cumulative_markout_pnl(pred_combined), }, index=np.linspace(0, 100, num=len(df_out)), ) # Plot the results results.plot( title="Forecasting with book skew vs. imbalance", xlabel="Predictor value (percentile)", ylabel="Cumulative return", ).legend() plt.show() ``` ## Result ```plaintext skew imbalance ret_500t skew 1.0 0.47513 0.109004 imbalance NaN 1.00000 0.064666 ret_500t NaN NaN 1.000000 ``` ![Forecasting with book skew vs. imbalance](https://databento.com/docs/machine-learning-0.svg.md) --- Source: https://databento.com/docs/examples/algo-trading/execution-slippage?lang=python # Execution slippage and markouts ## What is a markout? A markout measures the price change within some arbitrary time interval before or after an event, such as a fill or a lit trading signal. Markouts are often used for transaction cost analysis (TCA) and determining execution quality, but they may also be used for alpha research. For more information on markouts, check out the [Databento Microstructure Guide](https://databento.com/microstructure/markout). ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to create markout curves. This example will use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md). This schema provides every trade and order book event that updates the top of book. In this case we'll compute the markouts by midprice change, but it's also possible to compute it on trade-to-mid or trade-to-trade basis. Markouts are usually computed at different time intervals and then averaged over multiple events to a create a single curve. ## Markout events We'll calculate markout curves for the aggressive and passive side of every trade in NVDA for the following quantities: - less than 100 shares - 100 or more shares - 200 or more shares We'll use these curves to help understand adverse selection on passive limit orders and execution slippage on aggressive market orders to help determine order execution strategy. ## Example ```python from pathlib import Path import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd NUM_TIME_SAMPLES = 1000 WINDOW_LIMITS_MICROS = 120 * 1e6 def calc_markouts( midprice: pd.Series, events: pd.DataFrame, offsets: pd.TimedeltaIndex, aggressor_markout: bool = True, ) -> pd.DataFrame: # Create an empty array to store results results = np.empty(shape=(len(events), len(offsets), 1)) # Calculate markouts for each event for i, (ts_event, event) in enumerate(events.iterrows()): markouts = midprice.reindex(offsets + ts_event, method="ffill") - event["price"] # All "event" records here are aggressive trades, so `side` is the aggressor side # Passive seller if not aggressor_markout and event["side"] == "B": markouts *= -1 # Aggressive seller elif aggressor_markout and event["side"] == "A": markouts *= -1 markouts_in_mils = markouts * 1e4 results[i] = markouts_in_mils.values.reshape(-1, 1) # Transpose the array and calculate the mean return pd.DataFrame( data=np.nanmean(results, 0), index=offsets.total_seconds() * 1e6, ) def plot_markouts( events: pd.DataFrame, midprice: pd.Series, lt_thresholds: tuple[int] = (100,), gt_thresholds: tuple[int] = (100, 200), aggressor_markout: bool = True, ) -> None: # Create log spaced intervals from -2 minutes -> +2 minutes samples = np.geomspace(1e-3, WINDOW_LIMITS_MICROS, num=NUM_TIME_SAMPLES) offsets = pd.to_timedelta( np.append(-np.flip(samples), np.append(0, samples)), unit="microseconds", ) for threshold in lt_thresholds: # Filter trade events below the size threshold events_mask = events[events["size"] < threshold] # Calculate markouts markouts = calc_markouts(midprice, events_mask, offsets, aggressor_markout) plt.plot( markouts, drawstyle="steps-post", label=f"order size < {threshold:d} (n={len(events_mask):,d})", ) for threshold in gt_thresholds: # Filter trade events above the size threshold events_mask = events[events["size"] >= threshold] # Calculate markouts markouts = calc_markouts(midprice, events_mask, offsets, aggressor_markout) plt.plot( markouts, drawstyle="steps-post", label=f"order size >= {threshold:d} (n={len(events_mask):,d})", ) # Annotations plt.title(f"Markouts - {'Aggressive' if aggressor_markout else 'Passive'} orders") plt.xlabel("Time since event (μs)") plt.ylabel("PnL/share (mils)") plt.xscale("symlog") plt.legend() plt.show() if __name__ == "__main__": # Set parameters dataset = "XNAS.ITCH" symbol = "NVDA" start = "2024-05-21" # Create historical client client = db.Historical("$YOUR_API_KEY") # Download data or read an existing file mbp_data_file = Path(f"xnas-itch-{start.replace('-', '')}-mbp-1.dbn.zst") if not mbp_data_file.exists(): mbp_data = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="mbp-1", start=start, path=mbp_data_file, ) else: mbp_data = db.DBNStore.from_file(mbp_data_file) # Convert to DataFrame df = mbp_data.to_df() df = df.dropna().set_index("ts_event") # We will use the midprice to calculate markouts midprice_s = df[["bid_px_00", "ask_px_00"]].mean(axis=1).groupby("ts_event").last() # Filter for trades (exclude non-cross and cross trades) trades = df[(df["action"] == "T") & (df["side"] != "N")] # Plot markouts for limit orders plot_markouts(trades, midprice_s, aggressor_markout=False) trades["cost"] = trades["price"] * trades["size"] agg_trades = trades.groupby("ts_event").agg( { "size": "sum", "side": "first", "cost": "sum", }, ) agg_trades["price"] = agg_trades["cost"] / agg_trades["size"] # Plot markouts for market orders plot_markouts(agg_trades, midprice_s, aggressor_markout=True) ``` ## Result ![NVDA aggressive markouts](https://databento.com/docs/execution-slippage-1.svg.md) Observe that crossing the spread with smaller orders exhibits less market impact, which agrees with our general intuition. ![NVDA passive markouts](https://databento.com/docs/execution-slippage-0.svg.md) We see the markouts are initially positive after the fill, as passive orders collect the spread and don't incur instantaneous slippage. However, the markouts are negative after roughly 1 second, showing the strong effects of adverse selection on passive orders of all sizes. --- Source: https://databento.com/docs/examples/algo-trading/latency?lang=python # Matching engine latencies ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to process instrument definition and MBO data to calculate matching engine and feed latency. The *matching engine latency* is the time between a market event and when the message was sent by the exchange. The *feed latency* is the time between that message being sent by the exchange and when Databento's servers received it. > **See also** > > For calculating latency from the [Live client](https://databento.com/docs/api-live/client.md) see [this example](https://databento.com/docs/examples/basics-live/live-latency.md). ## Definition We will use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) in this example to assign the asset to each instrument, allowing us to examine these measurements for different assets. ## MBO We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) in this example to measure the matching engine and feed latency, but any schema which contains the `ts_recv` and `ts_in_delta` timestamps can be used for this purpose. ## Timestamps Databento provides several nanosecond resolution timestamps for every record. For these calculations, we will use: `ts_event`, `ts_recv`, and `ts_in_delta`. You can review our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md) to learn more about where each of these are captured, but they are briefly described as: | Timestamp | Description | | `ts_event` | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `ts_recv` | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `ts_in_delta` | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. | ## Example We will use a few parent symbols for this analysis: `ES.FUT`, `SR3.FUT`, `6E.FUT`, `BTC.FUT`, `ZF.FUT`, `GC.FUT`, and `CL.FUT`. These assets belong to different channels in CME Globex's matching engine which will allow us to compare the matching engine latency between channels. You can read more about Databento's parent symbology [here](https://databento.com/docs/conventions/symbology.md#parent). ```python import databento as db import matplotlib.pyplot as plt import numpy as np # Set parameters dataset = "GLBX.MDP3" products = [ "ES.FUT", "SR3.FUT", "6E.FUT", "ZF.FUT", "GC.FUT", "CL.FUT", ] # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Request definition data definitions = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=products, stype_in="parent", start="2024-01-21T00:00:00-6", end="2024-01-22T17:00:00-6", ) # Request MBO data to calculate latency mbo = client.timeseries.get_range( dataset=dataset, schema="mbo", symbols=products, stype_in="parent", start="2024-01-26T05:00:00-6", end="2024-01-26T17:00:00-6", # 12 hours of mbo ) # Create a DataFrame for the mbo data mbo_df = mbo.to_df(pretty_ts=False, map_symbols=False) # Filter out any records with a bad ts_recv timestamp mbo_df[mbo_df["flags"] & db.RecordFlags.F_BAD_TS_RECV == 0] # Check for bad timestamps if (mbo_df.index == db.UNDEF_TIMESTAMP).any(): raise ValueError("Data contains one or more undefined ts_recv timestamps") if (mbo_df["ts_in_delta"] == db.UNDEF_TIMESTAMP).any(): raise ValueError("Data contains one or more undefined ts_in_delta timestamps") # Calculate latency mbo_df["latency_matching_us"] = (mbo_df.index - mbo_df["ts_in_delta"] - mbo_df["ts_event"]) / 1e3 mbo_df["latency_send_to_recv_us"] = mbo_df["ts_in_delta"] / 1e3 mbo_df = mbo_df[["instrument_id", "latency_matching_us", "latency_send_to_recv_us"]] # Get the asset and instrument class for every symbol def_df = definitions.to_df(pretty_ts=False, map_symbols=False) def_df = def_df[["instrument_id", "asset", "instrument_class"]].set_index("instrument_id") # Join the two DataFrames latency_df = mbo_df.join(def_df, on="instrument_id") # Plot send to receive latency across all symbols data = latency_df["latency_send_to_recv_us"] sorted_data = np.sort(data) cum_probability = np.linspace(0, 1, num=len(sorted_data)) plt.plot(sorted_data, cum_probability) plt.title("Send to Receive Latency") plt.grid(axis="x") plt.xscale("log") plt.ylabel("Cumulative Probability") plt.xlabel("Latency (microseconds)") plt.show() # Plot matching engine latency across all symbols data = latency_df["latency_matching_us"] sorted_data = np.sort(data) cum_probability = np.linspace(0, 1, num=len(sorted_data)) plt.plot(sorted_data, cum_probability) plt.title("Matching Engine Latency") plt.grid(axis="x") plt.xscale("log") plt.ylabel("Cumulative Probability") plt.xlabel("Latency (microseconds)") plt.show() # Plot matching engine latency for each asset for asset, group in latency_df.groupby("asset")["latency_matching_us"]: data = np.percentile(group, range(0, 101)) plt.plot( data.take([50, 90, 95, 99]), [asset] * 4, marker="|", linestyle="", markersize=20, ) plt.scatter( data, [asset] * len(data), marker=".", alpha=[1 - abs(i - 50) / 50 for i, _ in enumerate(data)], ) plt.title("Matching Engine Latency") plt.grid(axis="x") plt.xscale("log") plt.ylabel("Channel") plt.xlabel("Latency (microseconds)") plt.show() ``` ## Results ![Feed latency](https://databento.com/docs/latency-0.svg.md) ![Matching engine latency](https://databento.com/docs/latency-1.svg.md) ![Matching engine channel latency](https://databento.com/docs/latency-2.svg.md) --- Source: https://databento.com/docs/examples/algo-trading/messaging-volatility?lang=python # Using messaging rates as a proxy for implied volatility > **Info** > > The Databento team is currently working on this article. > --- Source: https://databento.com/docs/examples/algo-trading/portfolio-optimization?lang=python # Mean reversion and portfolio optimization > **Info** > > The Databento team is currently working on this article. > --- Source: https://databento.com/docs/examples/algo-trading/pairs-trading?lang=python # Pairs trading based on cointegration **Pairs trading** is a form of statistical arbitrage that takes advantage of the co-movement in prices of two instruments. It seeks to profit from the convergence of the two prices. The simplest variation of this strategy involves going long and short simultaneously on a pair of **cointegrated** instruments. But more generally, you can construct a spread from any linear combination of the two instruments with the approach in this tutorial--even going long simultaneously on two instruments. ## Setup and dependencies We'll make use of the following external libraries: - [matplotlib](https://matplotlib.org/) for plotting. - [numpy](https://numpy.org/) for data manipulation. - [pandas](https://pandas.pydata.org/) for data manipulation. - [scikit-learn](https://scikit-learn.org/) to fit the linear regression between the two price series. - [statsmodels](https://www.statsmodels.org) for the Engle-Granger two-step cointegration test. ``` import databento as db import matplotlib.pyplot as plt import numpy import pandas as pd from statsmodels.tsa.stattools import coint from sklearn.linear_model import LinearRegression ``` ## CME WTI (CL) vs. ICE Brent crude oil (BRN) We'll demonstrate the pairs trading strategy on the two most liquid crude oil products traded on CME and ICE, the [WTI crude oil futures]($PLATFORM_HOSTNAME/datasets/GLBX.MDP3/Futures/CL) (CL) [and Brent crude oil futures]($PLATFORM_HOSTNAME/datasets/IFEU.IMPACT/Futures/BRN) (BRN) contracts respectively. This also showcases the value of Databento's supplemental receive timestamps, `ts_recv`, which are used to construct our OHLCV aggregates. Most academic literature avoids cross-venue arbitrage like this because timestamps from two different exchanges are based on the different reference clocks that cannot be synced easily. `ts_recv` mitigates this problem by providing a common reference time with sub-microsecond accuracy between geographically-distant venues. ## Configuration Exchange and clearing fees can be found on [CME](https://www.cmegroup.com/company/clearing-fees.html) and [ICE](https://www.ice.com/publicdocs/IFEU_Exchange_Clearing_Fees.pdf)'s websites. Tick sizes and notional tick values can be obtained from [instrument definitions](https://databento.com/docs/schemas/instrument-definitions.md). The [Computing tick value and contract notional](https://databento.com/docs/examples/instrument-definitions/contract-notional.md) example shows the calculation. ```python # Backtest configuration start = "2024-04-01" end = "2024-05-15" # Instrument-specific configuration symbol_x = "CLM4" symbol_y = "BRN FMN0024!" tick_size_x = 0.01 tick_size_y = 0.01 tick_value_x = 10.0 tick_value_y = 10.0 fees_per_side_x = 0.70 # NYMEX corporate member fees fees_per_side_y = 0.88 # Global configuration commissions_per_side = 0.08 trading_days_count = 252 # Number of trading days in one year # Model and monetization parameters lookback = 100 entry_threshold = 1.5 exit_threshold = 0.5 p_threshold = 0.05 ``` ## Data extraction We'll define a function to download the data needed in this example. ```python def download_data( dataset: str, schema: str, symbol: str, start: str, end: str | None = None, ) -> pd.DataFrame: client = db.Historical(key="$YOUR_API_KEY") data = client.timeseries.get_range( dataset=dataset, schema=schema, symbols=symbol, start=start, end=end, ) return data.to_df() ``` We'll fetch 1-minute [OHLCV aggregates](https://databento.com/docs/schemas/ohlcv.md) for ease of demonstration, but you may specify `schema="mbp-1"` to use [MBP-1](https://databento.com/docs/schemas/mbp-1.md) instead: ```python df_ohlcv_x = download_data("GLBX.MDP3", "ohlcv-1m", symbol_x, start, end) df_ohlcv_y = download_data("IFEU.IMPACT", "ohlcv-1m", symbol_y, start, end) df_ohlcv_x = df_ohlcv_x["close"].to_frame(name="x") df_ohlcv_y = df_ohlcv_y["close"].to_frame(name="y") df = df_ohlcv_x.join(df_ohlcv_y, how="outer").ffill() ``` Since OHLCV aggregates only print when there's a trade in the interval, you'll need to use the `.ffill()` method to synchronize the two time series in cases where one leg doesn't print. > **Tip** > > Although CL and BRN are liquid futures contracts, there's the possibility > that both sides don't print in a given minute interval. A more complete > implementation should resample during every minute of the trading session. ## Engle-Granger cointegration test The basis of this strategy is to check that the residuals of a linear regression between the prices of the two instruments exhibits **stationarity**, also called the **Engle-Granger two-step cointegration test**. This implies that the two instruments' price series are cointegrated, and that the spread constructed based on this linear association of the two instruments' prices is stationary. In turn, this implies that the spread is **mean-reverting**. ## Cointegration vs. correlation Cointegration is preferred over correlation when constructing a pairs trade, since correlation doesn't apply to non-stationary time series, and financial instrument prices are typically non-stationary and trending. High correlation between such series can occur even without a meaningful relationship. For instance, two price series which are trending in the same direction but diverging in the long run may exhibit a spurious, positive correlation. On the other hand, cointegrated prices series may exhibit positive, negative, or even zero correlation over any period due to short-term fluctuations, but tend to track closely over the long run. ## Implementation The strategy is simple: First, over the lookback period (last 100 records), we run the cointegration test. If the two series are found to be cointegrated, then the coefficient of linear regression between the two series provides the ratio of the two instruments needed to construct the spread, also called the **hedge ratio**. We'll also compute the sample mean and standard deviation over this period. Then, over the forward window (also 100 records), we compute the spread (residual) and normalize it by taking its z-score. ```python # Initialization df["cointegrated"] = 0 df["residual"] = 0.0 df["zscore"] = 0.0 is_cointegrated = False lr = LinearRegression() for i in range(lookback, len(df), lookback): x = df["x"].iloc[i - lookback:i].values.reshape(-1, 1) y = df["y"].iloc[i - lookback:i].values.reshape(-1, 1) if is_cointegrated: # Compute and normalize signal on a forward window x_new = df["x"].iloc[i:(i + lookback)].values.reshape(-1, 1) y_new = df["y"].iloc[i:(i + lookback)].values.reshape(-1, 1) spread_back = y - lr.coef_ * x spread_forward = y_new - lr.coef_ * x_new zscore = (spread_forward - spread_back.mean()) / spread_back.std() df.iloc[i:(i + lookback), df.columns.get_loc("cointegrated")] = 1 df.iloc[i:(i + lookback), df.columns.get_loc("residual")] = spread_forward df.iloc[i:(i + lookback), df.columns.get_loc("zscore")] = zscore _, p, _ = coint(x, y) is_cointegrated = p < p_threshold lr.fit(x, y) ``` If the normalized residual is negative, it means the instrument associated with y is underpriced, i.e. we'll buy y and sell x to monetize the prediction that their prices will converge in the future. Conversely, if the normalized residual is positive, we'll sell y and buy x. Since the two products have a similar notional value and futures have large minimum order size, we'll trade a static hedge ratio of 1. If you're modifying this strategy to trade stocks or ETFs or products that have significant differences in volatility or notional value, you should consider using the dynamic hedge ratio `lr.coef_` to scale your position. > **Tip** > > The hedge ratio `lr.coef_` may be negative. This regularly occurs, for instance, if > one leg of the pair is an inverse ETF. ```python # Standardized residual is negative => y is underpriced => sell x, buy y # Standardized residual is positive => y is overpriced => buy x, sell y df["position_y"] = np.select([df["zscore"] < -entry_threshold, df["zscore"] > entry_threshold], [1, -1]) df["position_x"] = -df["position_y"] ``` It's common to specify a score-based or time-based exit condition. For example, we can exit the position when the spread converges to within +/-0.5 standard deviations. This has the effect of longer holding periods and lower transaction costs. ```python # Exit positions when z-score crosses +/- exit_threshold hold_y_long = np.where(df["zscore"] <= -exit_threshold, 1, 0) hold_y_short = np.where(df["zscore"] >= exit_threshold, 1, 0) # Carry forward positions until exit condition is met df["position_y"] = df["position_y"].mask((df["position_y"].shift() == -1) & (hold_y_short == 1), -1) df["position_y"] = df["position_y"].mask((df["position_y"].shift() == 1) & (hold_y_long == 1), 1) df["position_x"] = -df["position_y"] ``` Here's a visualization of the strategy. The outer dashed lines show the entry conditions while the inner dashed lines show the exit conditions: ![Strategy](https://databento.com/docs/spread.png.md) ## Estimating slippage Exact instantaneous slippage from crossing the spread can be calculated using MBP-1 data and execution timestamps from the backtest. In the interest of time, you can gauge the viability of the strategy by applying a constant spread cost per contract side. This is typically assumed to be half a tick or one tick for liquid futures, but we can also estimate it quickly from a small slice of MBP-1 data: ```python # Estimate slippage df_mbp1_x = download_data("GLBX.MDP3", "mbp-1", symbol_x, start) df_mbp1_y = download_data("IFEU.IMPACT", "mbp-1", symbol_y, start) avg_spread_x = (df_mbp1_x["ask_px_00"] - df_mbp1_x["bid_px_00"]).mean() avg_spread_y = (df_mbp1_y["ask_px_00"] - df_mbp1_y["bid_px_00"]).mean() ``` ## Backtest We can now compute the gross PnL and transaction costs: ```python # Compute backtest results df["pnl_x"] = df["position_x"].shift() * df["x"].diff() / tick_size_x * tick_value_x df["pnl_y"] = df["position_y"].shift() * df["y"].diff() / tick_size_y * tick_value_y df["volume_x"] = df["position_x"].diff().abs() df["volume_y"] = df["position_y"].diff().abs() df["tcost"] = ( df["volume_x"] * fees_per_side_x + df["volume_y"] * fees_per_side_y + # Exchange fees (df["volume_x"] + df["volume_y"]) * commissions_per_side + # Commissions (df["volume_x"] * avg_spread_x / tick_size_x / 2 * tick_value_x) + # Slippage (df["volume_y"] * avg_spread_y / tick_size_y / 2 * tick_value_y) # Slippage ) df["gross_pnl"] = df["pnl_x"] + df["pnl_y"] df["net_pnl"] = df["gross_pnl"] - df["tcost"] ``` ## Results ```python # Plot and print results daily_pnl = df["net_pnl"].groupby(df.index.date).sum() sr = daily_pnl.mean() / daily_pnl.std() * np.sqrt(trading_days_count) print(f"Sharpe ratio: {sr:.2f}") df[["gross_pnl", "net_pnl", "tcost"]].cumsum().plot(xlabel="Date", ylabel="PnL (USD)", title="Trading results") plt.legend(["Gross PnL", "Net PnL", "t-cost"]) plt.show() ``` ```python Sharpe ratio: 8.57 ``` ![Results](https://databento.com/docs/pairs-trading-0.svg.md) > **Info** > > The full code example can be found on our GitHub [here](https://gist.github.com/databento-bot/23ea8f2ef0de6f1290c1c6d2c87b60c3). ## Practical considerations and improvements This tutorial is kept intentionally short using a simple vectorized implementation. Ideally, you should use [MBP-1](https://databento.com/docs/schemas/mbp-1.md) data with an event-driven backtest loop that incorporates the delays between your system and the exchange gateways. This can be done using the [market replay](https://databento.com/docs/api-historical/helpers/dbn-store-replay.md) method. This strategy is very sensitive to slippage and the exact expiration month that's traded; non-lead month contracts will have wider spreads which can make the strategy infeasible. We assumed exchange fees and commissions that are realistic for a medium-sized trading firm or a team within a larger firm that's already a CME member firm. In practice, achieving similar rates requires a [NYMEX 106.J corporate membership](https://www.cmegroup.com/company/membership/corporate/nymex.html) and thousands of contract sides executed per day. Scaling up to larger trade sizes will significantly expand the strategy's state machine, since you'll have to handle partial executions and different sizes on each leg of the spread to achieve optimal hedge ratio. MBP-1 data can also inform trade sizing. We introduced a few free model and monetization parameters, namely `LOOKBACK`, `ENTRY_THRESHOLD`, `EXIT_THRESHOLD`. These can be optimized using any hyperparameter tuning method (e.g. grid search) and cross-validated. There are also a few common variations of this strategy to consider: - **Basket trading** by constructing combinations of more than two instruments, using another procedure such as the Johansen test. - Calibrating an Ornstein-Uhlenbeck process, which gives a parametric model for time-based exit. - Using **total least squares** instead of ordinary least squares to determine the hedge ratio. - Scanning a wide universe systematically for cointegrated pairs, to discover unintuitive relationships. However, this runs the risk of poor generalization if too many pairs are tested. Dimensionality reduction methods like PCA can improve the search and generalization error. ## Further reading Alexander, C. (2008). Market Risk Analysis: Practical Financial Econometrics (Vol. 2). Wiley. Tsay, R. S. (2010). Analysis of Financial Time Series (3rd edition). Wiley. Engle, R. F., & Granger, C. W. J. (1987). Co-integration and error correction: Representation, estimation, and testing. Econometrica, 55(2), 251-276. --- Source: https://databento.com/docs/examples/algo-trading/live-stock-screener?lang=python # Build a real-time stock screener In this example, we will build a real-time scanner for approximately 9,000 tickers in the U.S. equity markets. We'll use the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md) from the [Databento US Equities Summary](https://databento.com/docs/datasets/equs-summary.md) (`EQUS.SUMMARY`) dataset. This dataset provides end-of-day summary statistics for all RegNMS symbols via the Nasdaq NLS+ feed. These statistics include consolidated OHLCV data across all U.S. equity venues. We'll also use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) from the [Databento US Equities Mini](https://databento.com/docs/datasets/equs-mini.md) (`EQUS.MINI`) dataset. This is a derived top-of-book dataset that provides aggregated BBO and trade information from a variety of proprietary market data feeds. > **Info** > > A [Databento US Equities]($PLATFORM_HOSTNAME/equities) Standard subscription is required to run this example. ```python import databento as db import pandas as pd class PriceMovementScanner: """Scanner for detecting large price movements in all US equities.""" def __init__( self, pct_threshold: float = 0.03, # Default threshold for alert ) -> None: """Initialize scanner with a configurable threshold.""" self.pct_threshold = pct_threshold self.start = pd.Timestamp.now(tz="US/Eastern").normalize() self.premarket_start = self.start.replace(hour=4) # Start of today's pre-market session self.symbol_directory: dict[int, str] = {} self.last_day_lookup: dict[str, float] = self.get_last_day_lookup() self.is_signal_lit: dict[str, bool] = {symbol: False for symbol in self.last_day_lookup} def get_last_day_lookup(self) -> dict[str, float]: """Get yesterday's closing prices for all symbols.""" client = db.Historical() # Get OHLCV-1d data from the previous session data = client.timeseries.get_range( dataset="EQUS.SUMMARY", schema="ohlcv-1d", symbols="ALL_SYMBOLS", start=(self.start - pd.offsets.BusinessDay(1)).date(), ) # Request symbology: This is required for ALL_SYMBOLS requests # which don't automatically map instrument ID to raw ticker symbol symbology_json = data.request_symbology(client) data.insert_symbology_json(symbology_json) df = data.to_df() return df.set_index("symbol")["close"].to_dict() def scan(self, event: db.DBNRecord) -> None: """ Scan for large price movements in market data events. """ if isinstance(event, db.SymbolMappingMsg): self.symbol_directory[event.instrument_id] = event.stype_out_symbol return if not isinstance(event, db.MBP1Msg): return symbol = self.symbol_directory[event.instrument_id] # Skip if alert already triggered for the symbol if self.is_signal_lit[symbol]: return bid = event.levels[0].pretty_bid_px ask = event.levels[0].pretty_ask_px # Skip if one side of the book is empty if pd.isna(bid) or pd.isna(ask): return # Calculate change since the previous close mid = (bid + ask) / 2 previous_close = self.last_day_lookup.get(symbol) # New listing, no data from previous session if previous_close is None: return pct_change = (mid - previous_close) / previous_close # Trigger alert if the threshold is exceeded and no previous alert if abs(pct_change) > self.pct_threshold: ts = event.pretty_ts_event.tz_convert("US/Eastern") print( f"[{ts.isoformat()}] {symbol} moved by {pct_change:.2%} " f"(current: {mid:.4f}, previous: {previous_close:.4f})", ) self.is_signal_lit[symbol] = True if __name__ == "__main__": # Instantiate scanner class scanner = PriceMovementScanner() # Create a live client live = db.Live(key="$YOUR_API_KEY") # Subscribe MBP-1 schema for ALL_SYMBOLS # Start subscription at pre-market open live.subscribe( dataset="EQUS.MINI", schema="mbp-1", symbols="ALL_SYMBOLS", start=scanner.premarket_start, ) # Add callback and start live.add_callback(scanner.scan) live.start() # Run indefinitely live.block_for_close() ``` ```plaintext [2025-05-06T04:00:00.023079908-04:00] CORN moved by 4.19% (current: 19.1400, previous: 18.3700) [2025-05-06T04:00:00.046518714-04:00] CPER moved by 4.04% (current: 30.4200, previous: 29.2400) [2025-05-06T04:00:00.047119403-04:00] FETH moved by -8.56% (current: 16.6050, previous: 18.1600) [2025-05-06T04:00:00.069275113-04:00] XXRP moved by -3.80% (current: 29.3300, previous: 30.4900) [2025-05-06T04:00:00.118536539-04:00] ETHT moved by 24.40% (current: 5.7100, previous: 4.5900) ... [2025-05-06T09:33:43.338614253-04:00] XMVM moved by 19.04% (current: 62.5450, previous: 52.5400) [2025-05-06T09:33:46.814411294-04:00] BELFB moved by -5.09% (current: 65.4500, previous: 68.9600) [2025-05-06T09:33:53.530064671-04:00] CATO moved by -3.25% (current: 2.2350, previous: 2.3100) [2025-05-06T09:33:53.711869173-04:00] WRB moved by -11.42% (current: 64.2150, previous: 72.4900) [2025-05-06T09:33:55.223052174-04:00] QUS moved by -3.72% (current: 149.8250, previous: 155.6100) ``` --- Source: https://databento.com/docs/examples/corporate-actions?lang=python ## Corporate actions | Title | Subtitle | Supported languages | | [Dividends](https://databento.com/docs/examples/corporate-actions/dividends.md) | Obtain regular and special dividend information. | Python, Historical | | [New listings](https://databento.com/docs/examples/corporate-actions/new-listings.md) | Explore new listings on US and global markets. | Python, Historical | | [Splits and reverse splits](https://databento.com/docs/examples/corporate-actions/splits-and-reverse-splits.md) | Handle forward and reverse stock split capital events. | Python, Historical | | [Mergers and demergers](https://databento.com/docs/examples/corporate-actions/mergers-and-demergers.md) | Analyze company merger and demerger events. | Python, Historical | --- Source: https://databento.com/docs/examples/corporate-actions/dividends?lang=python # Dividends > **Info** > > The Databento team is currently working on this article. ## Overview Dividends are a form of capital distribution paid to shareholders. They can be issued in various forms, most commonly as cash payments or additional shares of stock. Companies may pay dividends on a regular basis, such as quarterly, or as special one-off dividends that are typically issued due to exceptional earnings or significant events. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting dividends This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all Costco dividends from the beginning of 2019 through 2020. Costco is a good example of a company that pays a regular quarterly dividend. Additionally, we can observe the inclusion of a special dividend during this period. We can query for dividends specifically by filtering for the `DIV` (Dividend) event type. Once we obtain the records, we can filter the relevant columns for analysis. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions df_raw = ref_client.corporate_actions.get_range( symbols=["COST"], start="2019", end="2021", index="ex_date", events=["DIV"], countries=["US"], ) # Filter relevant columns columns = [ "record_date", "payment_date", "payment_type", "marker", "frequency", "gross_dividend", "operating_mic", "isin", "symbol", "issuer_name", ] df = df_raw[columns] df.head(10) ``` ## Result ```plaintext record_date payment_date payment_type marker frequency gross_dividend operating_mic isin symbol issuer_name ex_date 2019-02-07 2019-02-08 2019-02-22 C INT QTR 0.57 XNAS US22160K1051 COST Costco Wholesale Corp 2019-05-09 2019-05-10 2019-05-24 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2019-08-29 2019-08-30 2019-09-13 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2019-10-31 2019-11-01 2019-11-15 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2020-02-06 2020-02-07 2020-02-21 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2020-04-30 2020-05-01 2020-05-15 C INT QTR 0.70 XNAS US22160K1051 COST Costco Wholesale Corp 2020-07-30 2020-07-31 2020-08-14 C INT QTR 0.70 XNAS US22160K1051 COST Costco Wholesale Corp 2020-10-29 2020-10-30 2020-11-13 C INT QTR 0.70 XNAS US22160K1051 COST Costco Wholesale Corp 2020-12-01 2020-12-02 2020-12-11 C SPL UN 10.00 XNAS US22160K1051 COST Costco Wholesale Corp ``` ## Special dividends Special dividends are one-time or irregular dividend payments made by a company, typically in addition to regular dividend distributions. Unlike regular dividends, which are usually paid on a fixed schedule (e.g., quarterly or annually), special dividends are declared in response to exceptional circumstances, such as significant financial gains, asset sales, or other extraordinary events. For example, consider the special dividend paid by Costco on December 11, 2020: - `marker` was `SPL` (Special) - `frequency` was `UN` (Unspecified) - The dividend was paid less than a month after the previous quarterly dividend ## Fields The table below provides detailed descriptions for each **dividend-specific** field found in the [dynamic info JSON fields](https://databento.com/docs/schemas/corporate-actions.md#dynamic-fields). | Field | Description | | `marker` | The dividend marker. See the [MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker) enum lookup table. | | `frequency` | The dividend frequency. See the [FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq) enum lookup table. | | `declaration_date` | The date the dividend was announced by the issuer. | | `period_end_date` | The end of the financial period for which the dividend is being paid. | | `foreign_ex_date` | The date on which the US dollar rate to `currency` value was taken. | | `ex_date2` | This is the ex date value for stock component of a dividend with a `payment_type` of (**B**)oth stock and cash. The main `ex_date` field will be for the cash payment portion of the same record. | | `payment_date2` | This is the payment date value for stock component of a dividend with a `payment_type` of (**B**)oth stock and cash. The main `payment_date` field will be for the cash payment portion of the same record. | | `tba_flag` | The company intends to pay a dividend but the details are yet to be announced. | | `nil_dividend` | A real dividend event but of zero payrate. | | `div_rescind` | A dividend event but of zero payrate. | | `decl_currency` | The declared currency - ISO 4217. | | `decl_gross_amount` | The declared gross amount. | | `approx_flag` | If dividend is temporarily an approximate value. | | `tax_rate` | The domestic withholding tax rate. | | `coupon` | In countries where coupons are to be submitted to obtain a benefit this field is useful. | | `coupon_id` | The coupon ID. | | `div_rate` | The rate of dividend as a percentage of par value. | | `cur_pair` | Identifies the different currencies involved for the FX rate to be applied - ISO 4217. | | `dep_fees` | Depositary Fees - applies only to `DR` dividend. | | `dep_fees_currency` | Depositary Fees currency - ISO 4217. | | `rescind_cash_div` | If cash dividend cancelled. | | `rescind_stock_div` | If stock dividend cancelled. | | `usd_rate_to_currency` | The US dollar exchange rate for dividend currency. | | `div_postponed` | Indicating postponement of dividend payment. | ## Dividend types The `marker` field specifies the type of dividend. See [MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker) enum lookup table for possible values. The table below provides detailed descriptions for each dividend type: | Value | Description | | `ANL` | **Annual**: Refers to a dividend that is paid once per year. | | `ARR` | **Arrears**: Denotes a dividend payment made to cover missed or deferred dividends from previous periods. | | `CG` | **Capital Gain**: Indicates a dividend that is categorized as a gain realized from the appreciation of an asset. | | `CGL` | **Capital Gain Long Term**: Refers to a capital gain from an asset held for more than a year before being sold, often benefiting from preferential tax treatment. | | `CGS` | **Capital Gain Short Term**: Refers to a capital gain from an asset held for less than a year before being sold, typically subject to higher tax rates. | | `FNL` | **Final**: Indicates that this dividend is the last payment related to a particular series or event. | | `INS` | **Installment**: Refers to dividends paid in multiple payments over a period rather than a single lump sum. | | `INT` | **Interim**: A dividend paid between regular annual dividends, often reflecting earnings for a part of the year. | | `ISC` | **Interest on Capital**: Represents payments made as interest on the capital invested in a company or financial product. | | `MEM` | **Memorial**: Indicates a dividend paid in memory of an individual, often reflecting a special purpose or tribute. | | `QTR` | **Quarterly**: Refers to dividends paid every three months. | | `REG` | **Regular**: Denotes dividends that are paid at regular intervals, typically on a fixed schedule. | | `SMA` | **Semi-annual**: Refers to dividends paid twice a year. | | `SPL` | **Special**: Indicates a one-time or irregular dividend payment, often separate from regular dividends. | | `SUP` | **Supplementary**: Refers to additional dividends paid in addition to regular payments, usually reflecting extra earnings or adjustments. | | `UN` | **Unspecified**: Indicates that the dividend type is not defined or categorized under the specified options. | | `VAR` | **Variable**: Refers to dividends that can vary in amount or frequency, based on company performance or other factors. | ## Dividend frequency The `frequency` field specifies the frequency of dividend payments or occurrences. See [FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq) enum lookup table for possible values. The table below provides detailed descriptions for each dividend frequency: | Value | Description | | `28D` | **28 Days**: Dividend occurs every 28 days. | | `35DAY` | **Every 35 Days**: Dividend occurs every 35 days. | | `BIM` | **Bi-Monthly**: Dividend occurs every two months. | | `DLY` | **Daily**: Dividend occurs every day. | | `FNL` | **Final**: The last dividend in a series. | | `INT` | **Interim**: Dividend occurring between regular annual payments. | | `INTONMAT` | **Arrears**: Interest paid upon the maturity of a financial product. | | `INTONTRIG` | **Interest on Trigger**: Interest paid when a specific trigger condition is met. | | `IRG` | **Irregular**: Dividend occurs at irregular intervals. | | `ITM` | **Interest on Maturity**: Interest paid upon the maturity of a financial product. | | `MNT` | **Monthly**: Dividend occurs every month. | | `QTR` | **Quarterly**: Dividend occurs every three months. | | `SMA` | **Semi-Annual**: Dividend occurs twice a year. | | `TRM` | **Trimesterly**: Dividend occurs every four months (three times a year). | | `UN` | **Unspecified**: Dividend frequency is not defined or categorized. | | `WKL` | **Weekly**: Dividend occurs every week. | --- Source: https://databento.com/docs/examples/corporate-actions/new-listings?lang=python # New listings > **Info** > > The Databento team is currently working on this article. ## Overview New listings refer to the addition of securities to a stock exchange, allowing them to be publicly traded. This can include initial public offerings (IPOs) of companies entering the public market for the first time, as well as existing companies that choose to list additional securities or switch their listings to a different exchange. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting new listings This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all new US market listings for 31st July 2024. We can query for new listings specifically by filtering for the `NLIST` (New Listing) event type. Once we obtain the records, we can filter the relevant columns for analysis. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions df_raw = ref_client.corporate_actions.get_range( symbols=["ALL_SYMBOLS"], start="2024-07-31", events=["NLIST"], countries=["US"], ) # Drop rows where there is not yet a confirmed symbol or ISIN df = df_raw.dropna(subset=["symbol", "isin"]) # Filter relevant columns columns = [ "event", "operating_mic", "isin", "symbol", "issuer_name", "security_type", ] df = df[columns] df.head(20) ``` ## Result ```plaintext event operating_mic isin symbol issuer_name security_type event_date 2024-07-31 NLIST OTCM US56564V1199 MAQCW Maquia Capital Acquisition Corp WAR 2024-07-31 NLIST ARCX US31423L5030 FLCC Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US31423L6020 FSCC Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US31423L7010 FLCV Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US31423L8000 FLCG Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US26922B4692 BDIV ETF Series Solutions Trust ETF 2024-07-31 NLIST ARCX US26922B4775 SAWG ETF Series Solutions Trust ETF 2024-07-31 NLIST ARCX US26922B4858 SAWS ETF Series Solutions Trust ETF 2024-07-31 NLIST ARCX US3899301085 BTC Grayscale Bitcoin Mini Trust ETC 2024-07-31 NLIST OTCM US56564V2007 MAQCU Maquia Capital Acquisition Corp STP 2024-07-31 NLIST OTCM KYG9879S1003 YSBIF YSB Inc. EQS 2024-07-31 NLIST OTCM US56564V1017 MAQC Maquia Capital Acquisition Corp EQS 2024-07-31 NLIST OTCM TH0010010013 SUOFF Saha-Union Public Company Ltd EQS 2024-07-31 NLIST OTCM CA09643M1059 BSKCF BluSky Carbon Inc. EQS 2024-08-01 NLIST OTCM ID1000198500 BENGF Barito Renewables Energy Tbk PT EQS 2024-08-01 NLIST OTCM FR0014005AL0 ANFPF Antin Infrastructure Partners EQS 2024-08-01 NLIST ARCX US45783Y1459 IAUG Innovator ETFs Trust ETF 2024-08-01 NLIST BATS US45783Y1376 KAUG Innovator ETFs Trust ETF 2024-08-01 NLIST BATS US45783Y1111 ZAUG Innovator ETFs Trust ETF 2024-08-01 NLIST BATS US00888H5625 AUGU AIM ETF Products Trust ETF ``` --- Source: https://databento.com/docs/examples/corporate-actions/splits-and-reverse-splits?lang=python # Splits and reverse splits > **Info** > > The Databento team is currently working on this article. ## Overview A forward stock split (often just referred to as a stock split), is when an issuer increases the number of outstanding shares by dividing each existing share into multiple shares, thereby reducing the per-share price. A reverse stock split in contrast, involves reducing the number of outstanding shares by consolidating multiple existing shares into a single share, which increases the per-share price. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting US stock splits This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all US stock splits for January 2024. We can query for all forward and reverse stock splits by filtering specifically for the following event types: - `FSPLT` (Forward Split) - `RSPLT` (Reverse Split) ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the stock split events df_raw = ref_client.corporate_actions.get_range( symbols="ALL_SYMBOLS", start="2024-01", end="2024-02", index="ex_date", events=["FSPLT", "RSPLT"], countries=["US"], ) # Filter relevant columns columns = [ "event", "ratio_old", "ratio_new", "operating_mic", "isin", "symbol", "issuer_name", "security_description", ] df = df_raw[columns] df.head(10) ``` ## Result ```plaintext event ratio_old ratio_new operating_mic isin symbol issuer_name security_description ex_date 2024-01-02 RSPLT 20.0 1.0 XNAS US70261F1030 KTTA Pasithea Therapeutics Corp Ordinary Shares 2024-01-02 RSPLT 65.0 1.0 XNAS US89157G8841 TBLT Toughbuilt Industries Inc Ordinary Shares 2024-01-03 RSPLT 500.0 1.0 OTCM US4662993025 JZZI Ideal Group of Companies Inc. Ordinary Shares 2024-01-03 RSPLT 80.0 1.0 XNAS US88032L5066 TENX Tenax Therapeutics Inc Ordinary Shares 2024-01-03 CONSD 10.0 1.0 OTCM HK8028008944 TMLSF Timeless Resources Holdings Limited Ordinary Shares 2024-01-03 RSPLT 2.0 1.0 XNAS US53934A1152 LXEH Lixiang Education Holding Co Ltd ADR 2024-01-03 RSPLT 40.0 1.0 XNAS US71360T1016 PRSO Peraso Inc Ordinary Shares 2024-01-04 CONSD 2.0 1.0 OTCM CA46123W1041 IMTCF Intrepid Metals Corp Ordinary Shares 2024-01-04 RSPLT 10.0 1.0 XNAS US4718711033 JSPR Jasper Therapeutics Inc Ordinary Shares 2024-01-04 RSPLT 16.0 1.0 XNAS US6551871022 ACON Aclarion Inc Ordinary Shares ``` ## NVDA splits This example demonstrates how to retrieve all stock split events for Nvidia Corporation since March 2018. We'll make a request from the corporate actions API for listed Nvidia (`NVDA`) securities for the entire range of dataset history (starting 2018-05-01). All records returned relate to Nvidia Corporation shares listed on Nasdaq (`XNAS`). ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the stock split events df_raw = ref_client.corporate_actions.get_range( symbols=["NVDA"], start="2018-05", index="ex_date", events=["FSPLT", "RSPLT"], countries=["US"], ) # Filter relevant columns columns = [ "event", "ratio_old", "ratio_new", "operating_mic", "isin", "symbol", "issuer_name", "security_description", ] df = df_raw[columns] df.head() ``` ## Result ```plaintext event ratio_old ratio_new operating_mic isin symbol issuer_name security_description ex_date 2021-07-20 FSPLT 1 4 XNAS US67066G1040 NVDA NVIDIA Corp Ordinary Shares 2024-06-10 FSPLT 1 10 XNAS US67066G1040 NVDA NVIDIA Corp Ordinary Shares ``` --- Source: https://databento.com/docs/examples/corporate-actions/mergers-and-demergers?lang=python # Mergers and demergers > **Info** > > The Databento team is currently working on this article. ## Overview Mergers and demergers are significant corporate actions that can fundamentally alter the structure and strategy of companies. A merger involves the combination of two or more companies into a single entity. Conversely, a demerger involves a company splitting into two or more independent entities. The following examples will highlight some different merger and demerger scenarios with a particular emphasis on the continuity of the identifiers used by the corporate actions dataset. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting mergers and demergers This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all mergers and demergers for January 2024. We can query for all merger and demerger activity by filtering specifically for the following event types: - `MRGR` (Merger) - `DMRGR` (Demerger) - `CONSD` (Consolidation) - `SOFF` (Spin-Off) ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the merger and demerger events df_raw = ref_client.corporate_actions.get_range( symbols="ALL_SYMBOLS", start="2024-01", end="2024-02", events=["MRGR", "DMRGR", "CONSD", "SOFF"], countries=["US"], ) # Filter relevant columns columns = [ "event", "operating_mic", "isin", "symbol", "issuer_name", "security_description", ] df = df_raw[columns] df.head(20) ``` ## Result ```plaintext event operating_mic isin symbol issuer_name security_description event_date 2024-01-03 SOFF XNAS SG9999000020 FLEX Flex Ltd Ordinary Shares 2024-01-03 CONSD OTCM HK8028008944 TMLSF Timeless Resources Holdings Limited Ordinary Shares 2024-01-04 CONSD OTCM CA46123W1041 IMTCF Intrepid Metals Corp Ordinary Shares 2024-01-04 MRGR XNYS US53814L1089 LTHM Livent Corp Ordinary Shares 2024-01-06 CONSD OTCM CA15722J1030 CYLYF Ceylon Graphite Corp Ordinary Shares 2024-01-06 CONSD OTCM CA74359L1058 PMCOF Prospector Metals Corp Ordinary Shares 2024-01-08 CONSD OTCM CA7481191046 QNICF Quebec Nickel Corp Ordinary Shares 2024-01-09 CONSD OTCM NO0003070609 IDXAF IDEX Biometrics ASA Ordinary Shares 2024-01-09 DMRGR OTCM CA51440M1023 LANRF Lancaster resources inc Ordinary Shares 2024-01-11 SOFF XNAS US70387R4039 PAVM PAVmed Inc Ordinary Shares 2024-01-11 CONSD OTCM SE0007692124 IMNTF Vidhance AB Ordinary Shares 2024-01-12 CONSD OTCM CA60928P1071 MEAUF Moneta Gold Inc Ordinary Shares 2024-01-16 DMRGR OTCM CA30330R1082 FUUFF F3 Uranium Corp Ordinary Shares 2024-01-16 SOFF XNYS US60784B1017 MDV Modiv Industrial Inc Ordinary Shares - Class C 2024-01-16 DMRGR OTCM CA30330R1082 FUUFF F3 Uranium Corp Ordinary Shares 2024-01-16 CONSD XASE CA98980M1095 ZOM Zomedica Corp Ordinary Shares 2024-01-16 CONSD OTCM CA00208E2087 AATGF ATI Airtest Technologies, Inc. Ordinary Shares 2024-01-17 CONSD OTCM GB00BLG2TX24 RPNRF Rapid Nutrition PLC Registered Shares 2024-01-18 CONSD OTCM CA53681M1068 GLIOF Lithium Lion Metals Inc Ordinary Shares 2024-01-18 DMRGR OTCM FR0000121220 SDXOF Sodexo Ordinary Shares ``` ## Fiat Chrysler Automobiles and PSA Group merger In January 2021, Fiat Chrysler Automobiles (`FCAU`) and PSA Group (`UG`) completed a merger to form Stellantis (`STLA`), a global automotive giant. This merger combined the strengths and resources of both companies to enhance their competitive edge in the automotive industry. **Original companies** - Fiat Chrysler Automobiles listed on the New York Stock Exchange under the ticker symbol `FCAU`. - PSA Group listed on Euronext Paris under the ticker symbol `UG`. **New company** - Following the merger, Stellantis listed on the New York Stock Exchange under the ticker symbol `STLA`. To analyze the merger to Stellantis (`STLA`), we can request all Fiat Chrysler (`FCAU`) corporate actions for January 2021 and call `.head()` on the data frame. The merger and name change is evident with the record having an `event` of `ISCHG` (Issuer Name Change), an `event_date_label` of `name_change_date` on an `effective_date` of 2021-01-17. Notice the identical `listing_id`, `security_id` and `issuer_id` records for the NYSE listed stock Stellantis (`STLA`) changing symbol to `FCAU`. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md#listing-continuity) dataset guide for further details on listing continuity. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions for Fiat Chrysler Automobiles for January 2021 df_raw = ref_client.corporate_actions.get_range( symbols=["FCAU"], start="2021-01", end="2021-02", countries=["US"], ) # Filter relevant columns columns = [ "event", "event_date_label", "issuer_id", "security_id", "listing_id", "operating_mic", "isin", "symbol", "issuer_name", ] df = df_raw[columns] df.head(10) ``` ## Result ```plaintext event event_date_label issuer_id security_id listing_id operating_mic isin symbol issuer_name event_date 2021-01-04 AGM meeting_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-14 DIV ex_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-17 ISCHG name_change_date I-166801 S-4329550 L-16138235 XNYS NL0010877643 FCAU Stellantis N.V 2021-01-18 ICC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-18 BBCC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-18 BBEC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-19 LCC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-19 BBCC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-19 BBEC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-25 ANN notification_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V ``` ## DowDuPont demerger In 2019, DowDuPont completed its planned separation into three independent, publicly traded companies. This strategic move aimed to unlock value by allowing each company to focus on its core strengths, potentially leading to better operational performance and market valuation. **Original company** - DowDuPont (`DWDP`) formed through the merger of Dow Chemical and DuPont on August 31, 2017. **New companies** - Dow Inc. listed on the New York Stock Exchange under the ticker symbol `DOW`. - DuPont de Nemours, Inc. listed on the New York Stock Exchange under the ticker symbol `DD`. - Corteva, Inc. listed on the New York Stock Exchange under the ticker symbol `CTVA`. To analyze the DowDuPont split, request all DowDuPont (`DWDP`) corporate actions for 2019 and call `.head()` on the data frame. The demerger and name changes are evident with records indicating events such as `ISCHG` (Issuer Name Change) and `SHOCH` (Shares Outstanding Change), showing transitions to Dow Inc. (`DOW`), DuPont (`DD`), and Corteva (`CTVA`). Note the `listing_id` for DuPont of `L-17235812` continued post spin-offs, retaining ticker symbol `DD`. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions for DowDuPont for 2019 df = ref_client.corporate_actions.get_range( symbols=["DWDP"], start="2019", countries=["US"], ) # Filter relevant columns columns = [ "event", "event_date_label", "issuer_id", "security_id", "listing_id", "operating_mic", "isin", "symbol", "issuer_name", ] df = df[columns] df.head(10) ``` ## Result ```plaintext event event_date_label issuer_id security_id listing_id operating_mic isin symbol issuer_name event_date 2019-01-31 SHOCH effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-02-27 DIV ex_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-02 SOFF ex_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-03 ANN notification_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-26 SHOCH effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-30 SHOCH effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-05-02 DIV ex_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-05-06 ANN notification_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-05-23 AGM meeting_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-06-01 ICC effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc ``` --- Source: https://databento.com/docs/examples/adjustment-factors?lang=python ## Adjustment factors | Title | Subtitle | Image | Supported languages | | [Applying adjustment factors](https://databento.com/docs/examples/adjustment-factors/applying-adjustment-factors.md) | Factor in the effects of capital events on securities using adjustment factors. | ![example1](https://databento.com/docs/applying-adjustment-factors-tsla-thumbnail.svg.md) | Python, Historical | | [Handling multiple stock selections](https://databento.com/docs/examples/adjustment-factors/multiple-adjustment-factors.md) | Back-adjust closing prices over a selection of stocks using adjustment factors. | ![example2](https://databento.com/docs/multiple-adjustment-factors-thumbnail.svg.md) | Python, Historical | --- Source: https://databento.com/docs/examples/adjustment-factors/applying-adjustment-factors?lang=python # Applying adjustment factors ## Overview The following examples demonstrate how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all relevant adjustment factors for corporate actions affecting a specific equity. These factors are then applied to daily close prices (using UTC date boundaries) to create a smoother time series that accounts for the impact of the capital events. > **See also** > > [Adjustment factors](https://databento.com/docs/datasets/adjustment-factors.md) dataset guide for further details and considerations for applying > adjustment factors. ## Procedure The following steps outline the procedure for both examples: - **Obtain data.** Retrieve the adjustment factors for the desired security and time range - **Pre-process data.** - Filter single listings - Filter (**A**)pply status events - Select one `option` per corporate action event - Convert the `ex_date` index to tz-aware UTC timestamps - **Apply adjustment factors.** Iterate over all factors and apply the cumulative product to all records prior to the `ex_date` ## Dividend adjustments In the first example, we will use Microsoft Corporation (MSFT) shares, known for consistently issuing quarterly dividends. Our time range will be from the beginning of June 2018 to the end of June 2024. We will observe the cumulative effect of these dividends on the price series over time and plot both the adjusted and original unadjusted daily close prices. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbol = "MSFT" start = "2018-06" end = "2025-04" # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request adjustment factors data for the relevant time range df_factors = ref_client.adjustment_factors.get_range( symbols=symbol, start=start, end=end, countries=["US"], ) # Filter listing on Nasdaq/NGS (Global Select Market) df_factors = df_factors[df_factors["operating_mic"] == "XNGS"] # Filter factors with (A)pply status df_factors = df_factors[df_factors["status"] == "A"] # Filter option 1 (default) for all factors df_factors = df_factors[(df_factors["option"] == 1)] # Convert the ex_dates index to UTC timestamps df_factors.index = pd.to_datetime(df_factors.index, utc=True) # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request daily bars data to obtain close prices data = hist_client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbol, start=start, end=end, ) df_bars = data.to_df() # Apply the adjustment factors: # Create a new dataframe for the adjusted prices df_adjusted = pd.DataFrame(index=df_bars.index) # Copy the close column from df_bars df_adjusted = df_bars[["close"]].copy() df_adjusted["close_adjusted"] = df_adjusted["close"] # Iterate through the factors and apply cumulative product for ex_date, factor in zip(df_factors.index, df_factors["factor"]): df_adjusted.loc[df_adjusted.index < ex_date, "close_adjusted"] *= factor # Plot the results plt.plot(df_adjusted["close"], label="close") plt.plot(df_adjusted["close_adjusted"], label="adjusted") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Daily Close Price (USD)") plt.title(f"{symbol}") plt.show() ``` ![MSFT adjusted close](https://databento.com/docs/applying-adjustment-factors-msft-0.svg.md) ## Forward split adjustments In the second example, we will use Tesla Inc (`TSLA`) shares, which have experienced significant price growth in recent years. Companies often split their shares to lower the stock price after rapid growth. Our analysis will cover the period from the beginning of June 2018 to the end of June 2024. Tesla underwent two major stock splits in August 2020 and August 2022, resulting in substantial step changes in the close price series. We will apply adjustment factors to account for these splits and create a smoother time series that reflects the true underlying price over time. We'll also plot both the adjusted and original unadjusted daily close prices. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbol = "TSLA" start = "2018-06" end = "2025-04" # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request adjustment factors data for the relevant time range df_factors = ref_client.adjustment_factors.get_range( symbols=symbol, start=start, end=end, countries=["US"], ) # Filter listing on Nasdaq/NGS (Global Select Market) df_factors = df_factors[df_factors["operating_mic"] == "XNGS"] # Filter factors with (A)pply status df_factors = df_factors[df_factors["status"] == "A"] # Filter option 1 (default) for all factors df_factors = df_factors[(df_factors["option"] == 1)] # Convert the ex_dates index to UTC timestamps df_factors.index = pd.to_datetime(df_factors.index, utc=True) # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request daily bars data to obtain close prices data = hist_client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbol, start=start, end=end, ) df_bars = data.to_df() # Apply the adjustment factors: # Create a new dataframe for the adjusted prices df_adjusted = pd.DataFrame(index=df_bars.index) # Copy the close column from df_bars df_adjusted = df_bars[["close"]].copy() df_adjusted["close_adjusted"] = df_adjusted["close"] # Iterate through the factors and apply cumulative product for ex_date, factor in zip(df_factors.index, df_factors["factor"]): df_adjusted.loc[df_adjusted.index < ex_date, "close_adjusted"] *= factor # Plot the results plt.plot(df_adjusted["close"], label="close") plt.plot(df_adjusted["close_adjusted"], label="adjusted") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Daily Close Price (USD)") plt.title(f"{symbol}") plt.show() ``` ![TSLA adjusted close](https://databento.com/docs/applying-adjustment-factors-tsla-0.svg.md) --- Source: https://databento.com/docs/examples/adjustment-factors/multiple-adjustment-factors?lang=python # Handling multiple stock selections ## Overview The following example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all relevant adjustment factors for corporate actions affecting a selection of stocks. These factors are then applied to daily close prices (using UTC date boundaries) to create a smoother time series that accounts for the impact of the capital events. > **See also** > > [Adjustment factors](https://databento.com/docs/datasets/adjustment-factors.md) dataset guide for further details and considerations for applying > adjustment factors. ## Procedure We'll use the same procedure as outlined in the [Applying adjustment factors](https://databento.com/docs/examples/adjustment-factors/applying-adjustment-factors.md#procedure) tutorial. The only difference is that we'll now apply the factors to multiple listings simultaneously. ## Selection The selected stocks are prominent representatives of the tech sector, chosen to illustrate various scenarios where adjustment factors are applied: - **Microsoft Corp.** (`MSFT`): Pays regular quarterly cash dividends. - **Apple Inc.** (`AAPL`): Pays regular quarterly cash dividends. - **Tesla Inc.** (`TSLA`): Underwent two forward stock splits on 2020-08-31 and 2022-08-25. - **Amazon.com Inc.** (`AMZN`): Underwent a forward stock split on 2022-06-06, and issued a cash dividend on 2022-12-27. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbols = ["MSFT", "AAPL", "TSLA", "AMZN"] start = "2018-06" end = "2025-04" # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request adjustment factors data for the relevant time range df_factors = ref_client.adjustment_factors.get_range( symbols=symbols, start=start, end=end, countries=["US"], ) # Filter listings on Nasdaq/NGS (Global Select Market) df_factors = df_factors[df_factors["operating_mic"] == "XNGS"] # Filter factors with (A)pply status df_factors = df_factors[df_factors["status"] == "A"] # Filter option 1 (default) for all factors df_factors = df_factors[(df_factors["option"] == 1)] # Convert the ex_dates index to UTC timestamps df_factors.index = pd.to_datetime(df_factors.index, utc=True) # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request daily bars data to obtain close prices data = hist_client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbols, start=start, end=end, ) df_bars = data.to_df() def apply_factors(df_adjusted, df_bars, factors_symbol, symbol): for ex_date, factor in zip(factors_symbol.index, factors_symbol["factor"]): mask = (df_adjusted.index < ex_date) & (df_bars["symbol"] == symbol) df_adjusted.loc[mask, "close_adjusted"] *= factor # Apply the adjustment factors: # Create a new dataframe for the adjusted prices df_adjusted = pd.DataFrame(index=df_bars.index) df_adjusted["symbol"] = df_bars["symbol"] df_adjusted["close_adjusted"] = df_bars["close"].copy() for symbol in symbols: # Filter the bars and factors for the current symbol bars_symbol = df_bars[df_bars["symbol"] == symbol] factors_symbol = df_factors[df_factors["symbol"] == symbol] # Apply the factors using the `apply_factors` function defined above apply_factors(df_adjusted, df_bars, factors_symbol, symbol) # Plot the results for symbol in symbols: df_symbol = df_adjusted[df_adjusted["symbol"] == symbol] plt.plot(df_symbol.index, df_symbol["close_adjusted"], label=f"{symbol} adjusted") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Daily Close Price (USD)") plt.title("Adjusted Close Prices for Selected Stocks") plt.show() ``` ![Multiple adjusted close](https://databento.com/docs/multiple-adjustment-factors-0.svg.md) --- Source: https://databento.com/docs/examples/security-master?lang=python ## Security master | Title | Subtitle | Image | Supported languages | | [Enrich instrument definitions](https://databento.com/docs/examples/security-master/enrich-instrument-definitions.md) | Use the security master to enrich instrument definitions. | ![placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Listings and delistings](https://databento.com/docs/examples/security-master/listings-and-delistings.md) | Track security listings, delistings and symbology changes across time with the security master point-in-time (PIT) records. | ![placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Market capitalization change](https://databento.com/docs/examples/security-master/market-cap-change.md) | Combine daily close prices with shares outstanding to analyze changes in market capitalization over time. | ![example3](https://databento.com/docs/market-cap-change-thumbnail.svg.md) | Python, Historical | --- Source: https://databento.com/docs/examples/security-master/enrich-instrument-definitions?lang=python # Enrich instrument definitions ## Overview This example demonstrates how to merge security master data with the [Definition schema](https://databento.com/docs/schemas/instrument-definitions.md) using the [Historical client](https://databento.com/docs/api-historical/client.md). This gives you the benefit of multiple symbology identifiers and additional security and listing information included with each record. For our purposes we'll assume the latest definition schema can be obtained on 2024-04-01. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. ## Instrument identification and symbology When identifying a listing on a specific venue, we suggest using the `nasdaq_symbol` field, as it aligns with the normalized `raw_symbol` for instruments. It's also important to correctly match the venue, being the `operating_mic` ISO 10838 Market Identifier Code (MIC) with the definitions `exchange` field. By utilizing the [ISIN](https://en.wikipedia.org/wiki/International_Securities_Identification_Number), you can more precisely identify the security within the security master and corporate actions datasets. When querying these datasets, specifying the `isin` symbology type ensures the most accurate results. ```python import databento as db import pandas as pd # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request the instrument definitions data = hist_client.timeseries.get_range( dataset="XNAS.ITCH", symbols=["AAPL"], start="2024-04-01", limit=1, ) df_definition = data.to_df() # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the latest security master df_security_master = ref_client.security_master.get_last( symbols=["AAPL"], countries=["US"], ) # Merge the dataframes by joining the definition `symbol` with the security master `symbol` df_combined = pd.merge( df_definition, df_security_master, left_on="symbol", right_on="symbol", how="inner", ) # Display the combined records print(df_combined.head()) ``` ## Result ```plaintext ts_event rtype publisher_id instrument_id action side depth price size flags ts_in_delta sequence symbol ts_record listing_id listing_group_id security_id issuer_id listing_status listing_source listing_created_date listing_date delisting_date issuer_name security_type security_description primary_exchange exchange operating_mic nasdaq_symbol local_code isin us_code bbg_comp_id bbg_comp_ticker figi figi_ticker fisn lei sic cik gics naics cic cfi incorporation_country listing_country register_country trading_currency multi_currency segment_mic_name segment_mic structure lot_size par_value par_value_currency voting vote_per_sec shares_outstanding shares_outstanding_date ts_created 0 2024-04-01 08:00:05.048097430+00:00 0 2 38 T A 0 171.85 1 130 169598 280962 AAPL 2024-08-05 02:41:52+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0.00001 USD V 1 15204137000 2024-07-19 2024-11-01 03:02:18.020386+00:00 ``` --- Source: https://databento.com/docs/examples/security-master/listings-and-delistings?lang=python # Listings and delistings ## Overview This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all point-in-time (PIT) security master records for Google Inc. We can use these to track Google's security listings across time. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. ## Google securities Google launched in 1998, quickly becoming the leading global search engine. It went public in 2004 through an IPO (prior to the security master dataset coverage). In 2015, Google restructured to form Alphabet Inc., a holding company designed to manage its diverse ventures more effectively, ensuring focused growth across various sectors. Alphabet Inc., the parent company of Google, issues three types of stock to accommodate different investor needs and maintain founder control: Class A, B, and C shares. Summary of Alphabet's share structure: - Class A Shares (`GOOGL`): Publicly listed, 1 vote per share - Class B Shares: Not publicly listed, 10 votes per share, held by founders and early investors - Class C Shares (`GOOG`): Publicly listed, no voting rights This structure ensures that while the company benefits from public capital through Class A and C shares, the founders retain overriding control through their Class B holdings. ## Security master PIT records We'll make a request from the security master API for Google securities publicly listed in the US starting 2015-01-01. Because there are so many records, we'll call `.head(40)` on the dataframe to view a selection. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request security master records for Google df = ref_client.security_master.get_range( symbols=["GOOGL", "GOOG"], countries=["US"], start="2015", ) df.head(40) ``` ## Result ```plaintext ts_record listing_id listing_group_id security_id issuer_id listing_status listing_source listing_created_date listing_date delisting_date issuer_name security_type security_description primary_exchange exchange operating_mic symbol nasdaq_symbol local_code isin us_code bbg_comp_id bbg_comp_ticker figi figi_ticker fisn lei sic cik gics naics cic cfi incorporation_country listing_country register_country trading_currency multi_currency segment_mic_name segment_mic structure lot_size par_value par_value_currency voting vote_per_sec shares_outstanding shares_outstanding_date ts_created ts_effective 2015-10-05 00:00:00+00:00 2015-08-10 23:00:00+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN NaN 2015-10-05 2024-11-13 14:04:47.783780+00:00 2016-02-01 00:00:00+00:00 2016-02-12 13:28:12+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 292580627 2016-02-01 2024-11-01 06:45:30.486798+00:00 2016-02-01 00:00:00+00:00 2016-02-12 13:28:12+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 345539303 2016-02-01 2024-11-13 14:04:47.783780+00:00 2016-02-05 00:00:00+00:00 2016-02-15 13:33:55+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 292585668 2016-02-05 2024-11-13 14:04:47.783655+00:00 2016-04-11 00:00:00+00:00 2016-05-03 13:14:54+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 293658526 2016-04-11 2024-11-13 14:04:47.783655+00:00 2016-04-21 00:00:00+00:00 2016-05-04 07:22:00+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 293675151 2016-04-21 2024-11-01 06:45:30.486798+00:00 2016-10-27 00:00:00+00:00 2016-11-04 13:15:10+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 296087212 2016-10-27 2024-11-01 06:45:30.486798+00:00 2017-04-19 00:00:00+00:00 2017-05-01 07:39:14+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 297614546 2017-04-19 2024-11-13 14:04:47.783655+00:00 2018-01-31 00:00:00+00:00 2018-02-07 13:19:08+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 298492525 2018-01-31 2024-11-01 06:45:30.486798+00:00 2018-04-18 00:00:00+00:00 2018-04-25 13:13:11+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 298656198 2018-04-18 2024-11-01 06:45:30.486798+00:00 2018-07-18 00:00:00+00:00 2018-07-25 14:03:11+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349883498 2018-07-18 2024-11-01 06:45:30.486923+00:00 2018-10-19 00:00:00+00:00 2018-10-29 07:26:56+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349615045 2018-10-19 2024-11-13 14:04:47.783780+00:00 2018-10-19 00:00:00+00:00 2018-10-29 07:26:56+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349615045 2018-10-19 2024-11-13 14:04:47.783780+00:00 2019-01-31 00:00:00+00:00 2019-02-06 13:07:42+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 299360029 2019-01-31 2024-11-01 06:45:30.486798+00:00 2019-01-31 00:00:00+00:00 2019-02-06 13:07:42+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349291348 2019-01-31 2024-11-01 06:45:30.486923+00:00 2019-01-31 00:00:00+00:00 2019-02-06 13:07:42+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349291348 2019-01-31 2024-11-01 06:45:30.486923+00:00 2019-07-19 00:00:00+00:00 2019-07-29 04:36:44+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 347344583 2019-07-19 2024-11-13 14:04:47.783780+00:00 2020-01-27 00:00:00+00:00 2020-02-06 13:20:30+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 299895185 2020-01-27 2024-11-13 14:04:47.783655+00:00 2020-01-27 00:00:00+00:00 2020-02-06 13:20:30+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 299895185 2020-01-27 2024-11-13 14:04:47.783655+00:00 2020-01-27 00:00:00+00:00 2020-02-06 13:20:30+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 340979832 2020-01-27 2024-11-13 14:04:47.783780+00:00 2020-04-07 00:00:00+00:00 2020-04-27 03:38:30+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300047170 2020-04-07 2024-11-13 14:04:47.783655+00:00 2020-07-23 00:00:00+00:00 2020-08-03 05:05:14+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300471156 2020-07-23 2024-11-01 06:45:30.486798+00:00 2020-07-23 00:00:00+00:00 2020-08-03 05:05:14+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 333631113 2020-07-23 2024-11-01 06:45:30.486923+00:00 2020-10-22 00:00:00+00:00 2020-11-02 03:37:23+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300643829 2020-10-22 2024-11-13 14:04:47.783655+00:00 2020-10-22 00:00:00+00:00 2020-11-02 03:37:23+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 329867212 2020-10-22 2024-11-13 14:04:47.783780+00:00 2021-04-06 00:00:00+00:00 2021-04-26 04:11:13+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 324511693 2021-04-06 2024-11-01 06:45:30.486923+00:00 2021-04-20 00:00:00+00:00 2021-04-30 13:06:34+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300746844 2021-04-20 2024-11-01 06:45:30.486798+00:00 2021-07-20 00:00:00+00:00 2021-07-29 13:17:56+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 320168491 2021-07-20 2024-11-13 14:04:47.783780+00:00 2021-10-19 00:00:00+00:00 2021-10-28 13:14:13+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 317737778 2021-10-19 2024-11-01 06:45:30.486923+00:00 2022-01-25 00:00:00+00:00 2022-02-03 13:09:07+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300754904 2022-01-25 2024-11-13 14:04:47.783655+00:00 2022-01-25 00:00:00+00:00 2022-02-03 13:09:07+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 315639479 2022-01-25 2024-11-13 14:04:47.783780+00:00 2022-04-05 00:00:00+00:00 2022-04-11 03:37:16+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300758936 2022-04-05 2024-11-01 06:45:30.486798+00:00 2022-04-05 00:00:00+00:00 2022-04-25 03:51:25+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 315573679 2022-04-05 2024-11-01 06:45:30.486923+00:00 2022-04-05 00:00:00+00:00 2022-04-25 03:51:25+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 315573679 2022-04-05 2024-11-01 06:45:30.486923+00:00 2022-04-19 00:00:00+00:00 2022-04-29 03:51:45+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 313376417 2022-04-19 2024-11-13 14:04:47.783780+00:00 2022-07-27 00:00:00+00:00 2022-07-28 03:37:17+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 5996000000 2022-07-27 2024-11-13 14:04:47.783655+00:00 2022-07-27 00:00:00+00:00 2022-07-28 03:37:17+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 6163000000 2022-07-27 2024-11-13 14:04:47.783780+00:00 2022-10-18 00:00:00+00:00 2022-10-27 13:16:16+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 5973000000 2022-10-18 2024-11-13 14:04:47.783655+00:00 2023-01-26 00:00:00+00:00 2023-02-06 04:10:55+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 5956000000 2023-01-26 2024-11-01 06:45:30.486798+00:00 2023-04-18 00:00:00+00:00 2023-04-27 07:20:35+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 5874000000 2023-04-18 2024-11-13 14:04:47.783780+00:00 ``` --- Source: https://databento.com/docs/examples/security-master/market-cap-change?lang=python # Market capitalization change ## Overview This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve point-in-time (PIT) security master records for Apple Inc. alongside daily bar aggregates. By combining daily close prices with shares outstanding, we can analyze changes in Apple's total market capitalization over time. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. ## Market capitalization Market capitalization, or "market cap," is the total value of a company's outstanding shares of stock, representing its overall market value. It is typically calculated by multiplying the current stock price by the total number of outstanding shares: \[ \text{Market Capitalization} = \text{Share Price} \times \text{Shares Outstanding} \] This straightforward calculation provides a snapshot based on the latest share price but does not account for complexities like share class variations, potential stock buybacks, or other corporate actions that might affect a company's actual valuation over time. ## Apple daily bars Using the [Historical client](https://databento.com/docs/api-historical/client.md) client, we can retrieve OHLCV-1d bars from 2018-05-01. These daily bars provide close prices that we can use to calculate and plot Apple's market capitalization. Note that these bars are aligned with UTC date boundaries, so the close price represents the price at UTC midnight, which may differ from the local market session close price. > **See also** > > [Aggregate bars (OHLCV)](https://databento.com/docs/schemas/ohlcv.md) schema guide for further details. ## Security master PIT records Next, we'll use the security master API to request point-in-time (PIT) records for Apple Inc. stocks (`AAPL`) publicly listed in the US since 2018-05-01. We apply the `countries` filter with `"US"` to ensure we capture only US-listed securities, in this case Apple Inc. listed on Nasdaq (`XNAS`). For securities listed on multiple exchanges, additional filtering on the returned DataFrame may be necessary to isolate the desired listing. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbol = "AAPL" start = "2018-05" end = "2025-04" # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request OHLCV-1d data to obtain close prices data = hist_client.timeseries.get_range( dataset="XNAS.ITCH", schema="ohlcv-1d", symbols=["AAPL"], start="2018-05", end="2024-11", ) df_bars = data.to_df() # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request security master records for Apple df_security_master = ref_client.security_master.get_range( symbols=["AAPL"], countries=["US"], start=start, ) df_security_master["shares_outstanding_date"] = pd.to_datetime( df_security_master["shares_outstanding_date"], utc=True, ) df_security_master.set_index("shares_outstanding_date") # Filter and resample the shares outstanding to daily frequency, forward filling missing values df_shares_outstanding = ( df_security_master[["shares_outstanding_date", "shares_outstanding"]] .set_index("shares_outstanding_date") .resample("D") .ffill() ) # Align both dataframes on the date index df_market_cap = df_bars[["close"]].join(df_shares_outstanding, how="left") # Calculate market capitalization by multiplying close price with shares outstanding df_market_cap["market_cap"] = df_market_cap["close"] * df_market_cap["shares_outstanding"] df_market_cap["market_cap"] = df_market_cap["market_cap"] / 1e9 # Convert to billions # Plot the results plt.plot(df_market_cap["market_cap"], label="market_cap") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Market Capitalization (Billion USD)") plt.title(f"{symbol}") plt.show() ``` ![AAPL market cap](https://databento.com/docs/market-cap-change-0.svg.md) --- Source: https://databento.com/docs/faqs # FAQs | Title | Subtitle | Image | Supported languages | | [Client libraries vs. APIs](https://databento.com/docs/faqs/differences-our-client-libraries-apis.md) | Compare methods to programmatically access Databento's services. | ![Api cloud image](https://databento.com/docs/libraries-apis.svg.md) | Integration | | [Streaming vs. batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md) | Compare our two methods for receiving historical data. | ![Streaming vs. batch download image](https://databento.com/docs/streaming-vs-batch.svg.md) | Data requests, Historical only | | [Usage-based pricing and credits](https://databento.com/docs/faqs/usage-pricing-and-data-credits.md) | Details on how our metered pricing and historical data credits work. | ![Usage pricing image](https://databento.com/docs/usage-pricing.svg.md) | Pricing | | [Instruments and products](https://databento.com/docs/faqs/instruments-and-products.md) | Differences between products, instruments, and listings. | ![Instrument and product card image](https://databento.com/docs/instruments-and-product.svg.md) | Pricing | | [Venues and publishers](https://databento.com/docs/faqs/venues-and-publishers.md) | Differences between venues, publishers, and other types of data sources. | ![Venues and publishers image](https://databento.com/docs/venues-and-publishers.svg.md) | Pricing | | [MBP-1 vs. TBBO vs. BBO schemas](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md) | Explore the differences between three similar schemas. | ![MBP-1 TBBO image](https://databento.com/docs/mbp1-vs-tbbo-vs-bbo.svg.md) | Data formats | --- Source: https://databento.com/docs/faqs/differences-our-client-libraries-apis # Client libraries vs. APIs There are multiple ways to access data programmatically through Databento. The easiest way is to use one of our official client libraries for the following languages: - [Python](https://github.com/databento/databento-python) - [C++](https://github.com/databento/databento-cpp) - [Rust](https://github.com/databento/databento-rs) These client libraries reduce the amount of code you need to write and adopt our recommended best practices. If you are using a language that does not have an official client library, you can still access all of our historical features through our [HTTP API](https://databento.com/docs/api-historical.md?lang=http) and all of our live features through our [Raw API](https://databento.com/docs/api-live.md?live=raw), which is built on a regular TCP/IP socket. If you are still unsure, here are some considerations that can help you decide the best choice: - **Historical vs. live.** Our HTTP API exposes historical and delayed intraday data, whereas our Raw API only exposes live data. Our client libraries provide both historical and live data by implementing a common layer over both our HTTP API and Raw API. - **Web browser support.** Most browsers do not provide support for raw TCP sockets. If you are writing a browser application, you should use our HTTP API to ensure compatibility with your end users. - **Security.** If your organization has security policies that make it inconvenient for you to install or update an external client library, we recommend implementing your own client using our APIs directly. - **Real-time performance.** Our Raw API uses binary-encoded data messages and will have the lowest overhead for live data. The Python, Rust, and C++ client libraries wrap the Raw API for live data, so they can't be more efficient than the API itself. See our [latency page](https://databento.com/latency) to compare their latencies in real conditions. - **Similarities to REST.** Our HTTP API is a collection of RPC-style methods. Although it is not a REST API, you will find it familiar as it shares many of the standard practices adopted by HTTP-based REST APIs. - **Pricing.** You pay the same price for the same data, regardless of the access method. [We only charge you based on the size of the data in its original binary encoding]($PLATFORM_HOSTNAME/pricing). For example, the same data message will be larger in our CSV encoding than our binary encoding, but there are no extra charges for this overhead. You can toggle between the different client libraries and APIs anywhere in our documentation. By default, our Python client library is selected. ![Flowchart](https://databento.com/docs/choice.svg.md) --- Source: https://databento.com/docs/faqs/streaming-vs-batch-download # Streaming vs. batch download We support both **streaming** and **batch download** as means of receiving historical data. Both methods yield exactly the same data. They differ only in the way that the data is received. The table below compares the two methods. | | Streaming | Batch download | | Usage | Load data directly in your application via our [API or client libraries](https://databento.com/docs/faqs/differences-our-client-libraries-apis.md). | Download flat files from the [Download center](https://databento.com/docs/portal/download-center.md). HTTP and FTP are supported. | | Cost | You will be charged every time you stream the same data. | Download the same data multiple times over a 30 day period, for no additional charge. | | Size | Recommended for data requests under 5 GB. | Recommended for data requests over 5 GB. | | Customization | Limited | Full set of advanced customizations available. | | Wait time | None | Usually takes several minutes to prepare before the data is available in the Download center. | | Request method | [API](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md) only | Either manually using a [Batch download request](https://databento.com/docs/portal/batch-download.md) or programmatically via our [API](https://databento.com/docs/api-historical/batch/batch-submit-job.md). | Here are some details to consider when deciding which method is best for you: - **Usage.** With streaming, you can access data immediately when it is needed in your application. Streaming is suitable for smaller, on-demand workflows such as retrieving reference data for production trading, initializing a user's price chart for a given symbol on a display application, or exploring data in an interactive environment. - **Cost.** Since streaming is designed for one-off tasks, you will be charged each time for duplicate stream requests. If you intend to access the same data repeatedly -- for instance in parallel simulations, multiple backtests or daily ETL pipelines -- you can do so more efficiently with a batch download of the data onto your system. - **Size.** Batch download lets you fetch the same data multiple times without additional charge. Hence, it is more suitable for larger data requests where there is risk of disconnection while your data is being transmitted. - **Customization** and **wait time.** Our streaming infrastructure is optimized for instant retrieval of small data requests in [predetermined schemas](https://databento.com/docs/schemas.md). When you request a batch download, our system needs time to prepare the data files. This preparation time lets us service additional customizations that are not possible on our streaming infrastructure. - **Request method.** A batch download request can be submitted using the [Batch download]($PLATFORM_HOSTNAME) on our portal, using our HTTP API, or using any of our official client libraries. > **Info** > > Both methods are suitable for **market replay**. All of our historical data > is sequenced in the order of arrival on our live service and includes > receive timestamps. This enables you to replay events in the exact order as > they would've been received under live market conditions. --- Source: https://databento.com/docs/faqs/usage-pricing-and-data-credits # Usage-based pricing and credits ### Usage-based pricing Historical data is priced in $/GB. Every month, you pay only for the historical data that you've used. There is no monthly subscription fee for usage-based historical data. Live data requires a Standard, Plus, or Unlimited subscription plan. See our [pricing page]($PLATFORM_HOSTNAME/pricing) for plans by dataset, and the [plans and live data guide](https://databento.com/docs/portal/live-data.md) for licensing details. ### Historical data Historical data is billed per byte consumed. Databento meters data by its uncompressed size in binary encoding. For no additional charge, you can also request data in CSV and JSON encodings. We convert the data from binary, so you only pay for the data sized in the smaller encoding. For historical data, all delivery methods (like HTTP and FTP) incur no additional charge. ### Streaming and batch download You can receive data through either streaming or batch download. Here's how billing works for each method: - **Streaming.** You're billed for every outbound byte of data that's sent on our network. If connection cuts out in the middle of a streaming request, you won't be charged for the remaining unsent data that you had requested. - **Batch download.** You're billed once for the request. Then, you can download the data as many times as you want over 30 days -- with no charge. > **See also** > > See the [streaming vs. batch download FAQ](https://databento.com/docs/faqs/streaming-vs-batch-download.md) for more information on the differences between these download methods. ### Pricing quotes You can find estimated quotes for datasets by searching for them on the **Browse** page of your Databento portal, and then clicking on the dataset detail page. For a definitive quote, select and then customize your data request on the **Data request** page. ### Data credits All new users receive $125 in free credits upon signup. Credits can be used on historical data, or can be used towards the cost of the first month of a subscription plan. These credits are shared across your team and expire after 6 months. Each team is eligible for one set of credits. Credits are automatically applied before any charges are made to the user. The amount credits remaining can be found on the [billing page]($PLATFORM_HOSTNAME/portal/billing). --- Source: https://databento.com/docs/faqs/instruments-and-products # Instruments and products Our documentation and web platform uses the terms **product**, **instrument**, and **listing** for different purposes. This article clarifies the differences between these terms. ### Products On Databento, a **product** (also called a parent product or parent in the context of futures or options) refers to any real or synthetic asset whose data is provided by a market. A product describes a group of instruments belonging to a given economic sector or market segment. Also, products are often _fungible_, meaning they may be traded on more than one venue. In the case of equities, this means that products and instruments are the same asset, in contrast to derivatives such as futures and options products. The search on our [home page]($PLATFORM_HOSTNAME) looks up products across our dataset coverage. To find individual instruments, click on any of the products in the search results. A futures product for a certain underlying has a range of expirations. To avoid ambiguity, we will refer to the collection of all expirations as a **parent**, and specific expiration as an **instrument**. So, for example, `ES.FUT` is the parent of `ESM0` and `ESZ0`. Note: we classify exchange-traded spreads between futures outrights as part of the futures products. An options product for a certain underlying has a range of both expirations and strikes. To avoid ambiguity, we will refer to the collection of all expirations and strikes as a **parent**, and specific expiration with strike as an **instrument**. So, for example, `MSFT.OPT` is the parent of `MSFT20210205C210`. Note: we classify option combinations as part of the options products. ### Instruments On Databento, an **instrument** (also called a child instrument or child) is a tradable asset, real or synthetic, on a specific market. Instruments define all attributes of what is traded, e.g. product complex, product group, expiration, and strike price. Some publishers use the term **security** in lieu of what we consider an instrument. This becomes somewhat of a misnomer when applied to derivatives, which are not strictly securities. As such, we prefer to use the term **instrument** as it more broadly includes both securities and derivatives. Moreover, some markets provide multiple datasets which provide different levels of visibility. Two different datasets from the same market may exhibit different data for the same instrument. For instance, FX ECNs will often provide a premium feed with full visibility of real-time trades across all liquidity pools on the ECN as well as an entry feed with only delayed trade prints for a subset of liquidity. Since we support multiple datasets from the same market, we also need to distinguish if you're requesting data for EUR/USD on the premium feed or the entry feed. To resolve these identification issues, we use the term **listing** to refer to a tradeable entity in a specific dataset from a specific publisher. As such, we consider AAPL on NASDAQ TotalView-ITCH 5.0, AAPL on NYSE OpenBook Ultra, and AAPL on NYSE Trades as three distinct _listings_. The following table provides more of such examples. ### Listings On Databento, a listing is an instrument specific to a certain venue, which is not transferable or representable across other venues (_non-fungible_). While this makes a listing synonymous with an instrument for equities, it's important to be mindful of the distinction if you intend on extending into other asset classes. | | Equities | FX | Futures | Options | | Product | `AAPL`, `MSFT` | `EUR/USD` | `ES.FUT`, `GE.FUT` | `MSFT.OPT`, `ES.OPT`, `EW1.OPT` | | Instrument | `AAPL`, `MSFT` | `EUR/USD` *spot* | `ESM0`, `ESZ0` | `MSFT20210205C210`,`ESZ0 C3620`,`ESZ0 P3620`,`EW1Z0 C3500` | | Listing | `AAPL` *on Nasdaq*, `AAPL` *on NYSE Arca* | `EUR/USD` *on Cboe SEF* | `ESM0` *on CME Globex* | `MSFT20210205C210` *on BOX*, `ESZ0 C3620` *on CME Globex* | --- Source: https://databento.com/docs/faqs/venues-and-publishers # Venues and publishers You may have seen other data sources refer to all trading venues as an *exchange*. This naming convention is often harmless, especially if you are only interested in a small set of venues that actually implement a centralized exchange model. However, as you expand to more asset classes or markets with Databento, it becomes increasingly likely that you will encounter different venue models such as ECNs and ATSs. Since terms like *exchange*, *ATS*, and *ECN* have specific regulatory meanings, we refrain from conflating *exchange* with other trading venues. We use the terms "**trading venue**" and **venue** interchangeably. We use the term ***publisher*** to refer to any data provider on Databento. A publisher may also be a venue. > **Info** > > A vast majority of US equities data consumers are receiving data > disseminated by the Securities Information Processors (SIPs) --namely > the Consolidated Tape Association (CTA) and the UTP Plan. The SIPs provide > a consolidated view of all protected bid/ask quotes and trades from US > equities trading venues. On Databento, we refer to the SIPs as ***publishers*** > but not market operators. ### Exchange An **exchange** is a venue registered with the SEC under [Section 6](https://www.nyse.com/publicdocs/nyse/regulation/nyse/sea34.pdf) of the Securities Exchange Act of 1934. The SEC lists all current exchanges registered, [here](https://www.sec.gov/fast-answers/divisionsmarketregmrexchangesshtml.html). ### OTC markets **OTC markets** are dealer networks that trade securities that don't meet exchange listing requirements (such as "unlisted" securities) and instruments that don't trade on a formal exchange (such as bonds, ADRs, and derivatives). The Financial Industry Regulatory Authority (FINRA) regulates broker-dealers that operate in the over-the-counter (OTC) market. ### ATS An **Alternative Trading System (ATS)** is a non-exchange trading venue that matches buyers and sellers to counterparties for transactions. ATSs are usually regulated as broker-dealers instead of as exchanges. ATSes meet the definition of exchange under federal securities law. They aren't required to register as a national securities exchange, as long as the ATS operates under the exemption provided under [Exchange Act Rule 3a1-1(a)](https://www.law.cornell.edu/cfr/text/17/240.3a1-1). Since ATSes are non-exchanges, ATS transactions don't appear on national exchange order books. Some traders use this advantage to conceal trading from public view, and reduce the effect of large trades. An ATS must be approved by the United States Securities and Exchange Commission (SEC). For a current list of SEC-recognized ATSes, see [here](https://www.sec.gov/foia/docs/atslist.htm). The equivalent term under European legislation is a ***multilateral trading facility (MTF)***. ATSes are inclusive of **electronic communications networks (ECNs)**, **cross networks**, **call networks**, and **dark pools**. ### ECN **ECNs** are a type of alternative trading system (ATS) that trade listed stocks and other exchange-traded products outside of traditional stock exchanges. As an ATS, ECNs are required to register with the SEC as broker-dealers as well as register as members of FINRA. ECNs are computerized systems that internally and automatically match limit orders, charging a fee for each transaction. These venues attempt to eliminate the third party's role in executing orders entered by an exchange market maker or an over-the-counter market maker. They permit such orders to be executed against in whole or in part. ### Dark pools **Dark pools** are a type of ATS that operate within private groups. They are private exchanges or forums for securities trading. Though their legal validity is subject to local regulations, they are popular for participants who want to disassociate their identity from market activity, and for participants who want to save money on transaction fees. | Type | Overview | Examples | | Exchange | Regulated by the SEC under [ Section 6 ]( https://www.nyse.com/publicdocs/nyse/regulation/nyse/sea34.pdf ) of the Securities Exchange Act of 1934 | Nasdaq, NYSE, Cboe, TYO | | OTC markets | Regulated by FINRA. Dealer networks that trade unlisted securities and instruments that don't trade on formal exchanges (such as bonds, ADRs, and derivatives). | Best Market (OTCQX), the Venture Market (OTCQB), and the Pink Open Market | | ATS | Regulated as broker-dealers. Exempt from SEC "exchange" regulation under [ Exchange Act Rule 3a1-1(a) ]( https://www.law.cornell.edu/cfr/text/17/240.3a1-1 ). | BrokerTec, Nasdaq Fixed Income | | ECN | A type of ATS. A computerized system that automatically matches limit orders. | Pepperstone, FXCM, IC Markets | | MTF | The European equivalent of an ATS. Regulated under MiFID II. | Chi-X Europe | | Dark pools | A type of ATS. Operated within private groups as exchanges or forums. Popular for participants who want to disassociate their identity with market activity or save money on transaction fees. | Robinhood, CrossFinder, Sigma X, Citi-Match, MS Pool, Instinet, Liquidnet, ITG Posit, GETCO, Knight | --- Source: https://databento.com/docs/faqs/difference-between-mbp-and-tbbo # MBP-1 vs. TBBO vs. BBO schemas > **Info** > > This article also applies to the CMBP-1, TCBBO, and CBBO schemas. The MBP-1, TBBO, and BBO schemas are very similar, as they share the same fields and all cover top-of-book information in different ways. ### Key differences: Event update space The easiest way to think of the differences between the MBP-1, TBBO and BBO schemas is to consider their update space: - [MBP-1](https://databento.com/docs/schemas/mbp-1.md) is in book update space. - [BBO on trade (TBBO)](https://databento.com/docs/schemas/tbbo.md) is in trade space. - [BBO on interval (BBO)](https://databento.com/docs/schemas/bbo.md) is in time space, subsampled at 1-second or 1-minute intervals. ### Recommended use cases Another way to understand the differences between the schemas is to think of their use cases. - **Every top of book update**: Of the three schemas, MBP-1 is the only one that provides every top of book update. If you need the most granular of these three schemas, you should use MBP-1. - **Illiquid instruments**: For illiquid instruments that trade infrequently, the BBO schema is generally more useful than TBBO. If an instrument doesn't trade for extended periods of time, the best bid and offer retrieved from the last TBBO message can be stale. - **Options**: The BBO schema is most useful for option instruments, since they have a very low ratio of trades to book updates and will exhibit an extremely large volume of MBP-1 updates relative to BBO or TBBO updates. Options MBP-1 data can be costly or difficult to process while options TBBO data can be stale. A majority of our options users use BBO instead of MBP-1. - **Trading signals based on trades**: Since most of the price discovery happens around the time of a trade, TBBO can be used to construct many useful signals without the bloat of MBP-1. - **Display apps**: If you're building a display app that doesn't show every update and just needs to display a reasonably recent trade price, the BBO schema is more convenient than TBBO or MBP-1 because the BBO schema includes a forward-filled last sale price. In other words, the BBO schema is easier to manage because it's stateless with regards to the trade price. Moreover, your app usually has to handle less traffic when using the BBO schema. ### Deriving TBBO and BBO from MBP-1 You can always use MBP-1 and derive the other two schemas. - You can derive the TBBO schema by removing any event from the MBP-1 schema that does not have an `action` of `T`, for trades. - You can derive the BBO schema by subsampling the MBP-1 schema at either a 1-second or 1-minute interval. However, it is not possible to derive BBO from TBBO or vice versa. ### Visual differences: MBP-1 vs. TBBO The following graph illustrates the difference between MBP-1s and TBBO. One can observe that the bid-ask spread can be updated in MBP-1 without a trade event. That is not the case for TBBO, where every data point has an associated trade. ![MBP-1 Bid/Ask for NFLX](https://databento.com/docs/difference-between-mbp-and-tbbo-0.svg.md) ### Visual differences: MBP-1 vs. BBO-1s The following graph illustrates the difference between MBP-1s and BBO-1s on an options contract. Observe that BBO-1s has much fewer BBO updates, making it easier to handle. ![MBP-1 Bid/Ask for ESM4 P4800](https://databento.com/docs/difference-between-mbp-and-tbbo-1.svg.md) --- Source: https://databento.com/docs/portal # Portal How to use the Databento portal and its features. | Title | Subtitle | Image | Supported languages | | [Data catalog](https://databento.com/docs/portal/data-catalog.md) | Browse datasets, instrument details, and API examples. Download sample data. | ![Data catalog items image](https://databento.com/docs/data-catalog.svg.md) | Datasets | | [Batch download](https://databento.com/docs/portal/batch-download.md) | Instant pricing for live and historical selections. Request historical data. | ![Request summary image](https://databento.com/docs/batch-download.svg.md) | Data requests, Pricing | | [Data usage](https://databento.com/docs/portal/data-usage.md) | Monitor live and historical usage across your team and different datasets. | ![Dataset and live image](https://databento.com/docs/data-usage.svg.md) | Data requests, Datasets, Team | | [API keys](https://databento.com/docs/portal/api-keys.md) | Create, edit, regenerate, and delete API keys. Used to authenticate requests. | ![Sample API key image](https://databento.com/docs/api-keys.svg.md) | Data requests, Billing | | [Download center](https://databento.com/docs/portal/download-center.md) | Access batch download requests from your portal and API. Files expire after 30 days. | ![Download center image](https://databento.com/docs/download-center.svg.md) | Data requests, Historical only | | [Team](https://databento.com/docs/portal/team.md) | Add unlimited teammates, set monthly limits, and streamline accounting--for free. | ![Team management table image](https://databento.com/docs/team.svg.md) | Team, Billing | | [Billing](https://databento.com/docs/portal/billing.md) | View invoices, set team-wide monthly limits, and manage payment methods. | ![Example billing items image](https://databento.com/docs/billing.svg.md) | Billing, Invoices | | [Live data](https://databento.com/docs/portal/live-data.md) | Determine your subscriber status, activate live data, and manage licenses. | ![Example live data progress image](https://databento.com/docs/live-data.svg.md) | Datasets, Licensing, Live only | --- Source: https://databento.com/docs/portal/data-catalog # Data catalog ![Data catalog](https://databento.com/docs/data-catalog-0.png.md) ### Overview The data catalog is on the **[Browse]($PLATFORM_HOSTNAME)** page of your Databento portal. In addition, you can browse datasets from different exchanges, like CME and Nasdaq. Each dataset consists of products (read more about the definition of a product [here](https://databento.com/docs/faqs/instruments-and-products.md)). ### Searching for products You can also search for products on the **[Browse]($PLATFORM_HOSTNAME)** page by typing directly into the search bar. You can filter your results based on the following: - **Asset class.** Search within equities, FX, futures, and options. - **Category**. Search within raw market data, reference data, and textual data. - **Schema.** Search within MBO, MBP, TBBO, Trades, and more. Learn more about our supported schemas [here](https://databento.com/docs/schemas.md). - **Publisher.** Search within venues (such as CME Globex, Nasdaq, OPRA) and vendors (for instance EDI). - **Region.** Search within North America, Europe, and Asia. - **Available history.** Search within ranges from one month to at least 3 years. Clicking on a product in the results will take you to its [Instrument details page]($PLATFORM_HOSTNAME/portal/data-catalog#instrument-details-page) ### Dataset details page ![Dataset details page](https://databento.com/docs/data-catalog-1.png.md) > **Info** > > Not all datasets will contain every section shown in the above picture. Click on the dataset's icon to find its details page. This page includes the following sections: - **Products.** Includes details about the parent products available in the dataset. See [Instruments and products](https://databento.com/docs/faqs/instruments-and-products.md) for more information. - **Specifications.** Includes details about the dataset -- like its assets, region, market, schemas, encodings and capture origin. - **License.** Includes information about any applicable licensing required to access this dataset. - **Sample data.** Includes real data extracts for quality assurance. You can view these extracts in different schemas. - **API examples.** Includes minimal examples of using the Historical API with this dataset. - **Pricing.** Includes a monthly pricing estimate, by schema, for both live and historical data. You can select the entire dataset, or select individual products. > **Info** > > The monthly prices shown in the pricing calculator are only estimates, calculated from data > sizes over the previous 30 days. For more accurate pricing, see the [metadata.get\_cost](https://databento.com/docs/api-historical/metadata/metadata-get-cost.md) > method. ### Searching within datasets On each dataset's details page, you can also search for products within that dataset. Search for a symbol or product name directly in the search bar. You can also filter your results with the following: - **Asset class.** Search within equities, FX, futures, and options. - **Product group.** Filters include interest rates, equities, energy, agriculture, and more. - **EOD volume.** Rank results by End-of-Day (EOD) volume. - **Open interest.** Rank results by open interest. ### Instrument details page ![Instrument details page](https://databento.com/docs/data-catalog-2.png.md) > **Info** > > A chart will appear in lieu of an instrument list when viewing equity products. Click on a product to open up its instrument details page. This page includes the following sections: - **Instrument list.** Instruments with specific expirations (or strikes) are bundled under this parent product (for instance `ES`). - You can choose to show all active instruments under this product on a specific date. - You can sort the instrument list by symbol, expiration, last price, volume, and open interest. - **Sample data.** You can view sample data for all available schemas in this dataset. --- Source: https://databento.com/docs/portal/batch-download # Batch download ![Batch download overview](https://databento.com/docs/batch-download-0.png.md) ### Overview After you've selected a product from a dataset, you can customize and submit your download request for flat files. We support DBN, CSV, and JSON formats, allowing our data to be integrated with various platforms and database solutions. > **Info** > > Within one request, you can download multiple products from the same dataset -- with no limit to > the number of products. However, you can't download various products from different datasets > within one request. ### Data selection ![Data selection](https://databento.com/docs/batch-download-1.png.md) - **Schema.** Specify the schema you want to receive your data in -- MBO, MBP, TBBO, etc. You can read more about our supported schemas [here](https://databento.com/docs/schemas.md). After you select a schema, we specify the usage rate ($/GB). - **Time range.** You can specify your data time range. After you select a time range, we specify the data usage amount. ### Basic customization ![Basic customization](https://databento.com/docs/batch-download-2.png.md) Choose your preferred encoding, compression, and delivery method for no additional charge. - **Encoding.** DBN has the fastest transfer speed and smallest size. CSV works best for spreadsheets and JSON for web applications. Learn more about how to read DBN files [here](https://databento.com/docs/conventions/databento-binary-encoding.md). - **Compression.** Choose between zstd or no compression option. Learn more about how to decompress zstd files [here](http://facebook.github.io/zstd/). - **Delivery method.** Direct download (HTTP or FTP) is currently the only option available, with S3 and disk coming soon. ### Advanced customization ![Advanced customization](https://databento.com/docs/batch-download-3.png.md) Split your data into different files based on the following: - **Duration.** Split data into one file per day, week, or month. - **Size limit.** Split files above a size limit of 2, 4, or 10GB. - **Instrument.** Split files by instrument. The following are additional customizations available for CSV and JSON encodings: - **Price format.** Scale prices to decimal strings. - **Timestamp format.** Format timestamps as ISO 8601 strings. - **Symbol field.** Add a symbol field to the data. ![Request Summary](https://databento.com/docs/batch-download-4.png.md) After you've customized your download request details, submit the request by clicking the **Submit request** button on the right side of the page. We will queue and process your request, and the files will be ready for use on your Databento portal's [Download center](https://databento.com/portal/download-center) page. You can read more about accessing and managing downloads in our [Download center](https://databento.com/docs/portal/download-center.md) documentation. --- Source: https://databento.com/docs/portal/data-usage # Data usage ![Data Usage](https://databento.com/docs/data-usage-0.png.md) ### Overview The data usage page allows users to monitor their most recent and historical data usage. Additionally, Owners and Admins can monitor data usage for members of their team. ### Recent data usage This section shows data usage for the most recent 12 hours. The following metrics are shown: - **Cost.** The total cost in dollars. - **Gigabytes.** The total amount of data used in gigabytes. - **API keys.** The number of API keys currently in use with open connections. - **Connections.** The total number of open connections. You can filter these metrics based on the following: - **Dataset.** Filter by dataset. - **API Key.** Filter by the API key. - **Mode.** Filter by mode. Options are **Historical Streaming, Historical Batch**, and **Live**. - **Historical Streaming.** Streaming requests made through the [Historical.timeseries](https://databento.com/docs/api-historical/timeseries.md) endpoint. - **Historical Batch.** Batch download requests made through either the [Historical.batch](https://databento.com/docs/api-historical/batch.md) endpoint, or the [online portal](https://databento.com/docs/portal/batch-download.md). - **Live.** Real-time data streamed from the [Live API](https://databento.com/docs/api-live.md). ### Past data usage ![Past data usage](https://databento.com/docs/data-usage-1.png.md) This section shows live and historical data usage for a selected time range. You can view data usage based on the following: - **Date range.** The specific date range to filter on. - **Dataset.** View data usage by dataset. - **API Key.** View data usage by API Key. You can sort these results by the following: - **Cost (%).** The cost as a percentage of the total. - **Cost ($).** The total cost in dollars. - **Size (GB).** The total size of the data in gigabytes. > **See also** > If viewing this data by dataset, you can use the caret next to the dataset name to view > data usage by mode and schema. --- Source: https://databento.com/docs/portal/api-keys # API keys ![API Keys](https://databento.com/docs/api-keys-0.png.md) ### Overview An **API key** is an authentication key generated by our API to identify the source making a request to the Databento platform. API keys add a layer of security to your account. With them, you can authenticate data requests without using your account username or password. You can also assign specific permissions to team members and limit data access. Protect your API key like your password--never give it to a third party. There is a limit of 100 API Keys on your account. If the maximum number of keys is reached, a `403` error status will be returned when attempting to create a new key. ### Creating an API key 1. Click the **Create a new key** button. 2. Provide a descriptive name for your API key (API key names must be unique, and up to **64 characters long**). 3. Click on the **Create** button. ### Editing an API key 1. Click on the pencil icon for the key you want to edit. 2. Type a new name for the API key. 3. Click on the **Save** button. ### Rotating an API key We recommend that you regularly rotate all API keys for security reasons. This limits the time any one key is active, reducing potential billing impact if a key becomes compromised. See the end of this guide for other best practices to [secure your API keys](https://databento.com/docs/portal/api-keys.md#securing-your-api-keys). 1. Click on the rotate icon for the key you want to rotate. 2. Click on the **Rotate** button to confirm. > **Note**: Any live applications currently using the API key will be disconnected. ### Deleting an API key 1. Click on the trash can icon for the key you want to delete. 2. Click on the **Delete** button to confirm. > **Note**: Before you delete a key, we recommend you first test that it's no longer in use. ### Copying an API key 1. Click on the copy icon for the key you want to copy. The API key has now been copied to your clipboard. ### Securing your API keys Since API keys are used to authenticate requests, it's important to secure them. If your API key is exposed and used to access data, your account will incur billing charges. Aside from regularly changing your API keys, here are some best practices to keep them secure: - **Don't store your API keys within source code under version control.** Instead, we recommend setting your API keys as environment variables. If you need to store an API key in a file, it's advised to store it in an `.env` file that's read at runtime. - **Regenerate any API key that may have been compromised.** If you have reason to believe that an API key has been compromised, delete it immediately. You can create a new one in your Databento portal. - **Use a different API key for each application.** This allows you to isolate incidents and regenerate the key for one application without affecting others. - **Turn on historical usage limits.** Users with a compromised API key are liable for any unauthorized charges. For additional protection against misuse, you can [set up usage limits](https://databento.com/docs/portal/billing.md#manage-historical-monthly-limit) in your Databento portal. These limits can't be adjusted via API. ### Revoked API key Your account security is important to us. If we find that an API key has been made public, whether on GitHub, in our Slack community, or elsewhere, we'll revoke the key to protect your account. A revoked API key can't be used for authentication, so to remedy this, rotate the revoked key or create a new key. --- Source: https://databento.com/docs/portal/download-center # Download center ![Download center](https://databento.com/docs/download-center-0.png.md) ### Overview You can access and manage all data downloads on the Download center page of your Databento portal. After you submit a download request, we prepare the flat files on the backend and then save them to the Download center. ### Finding previous downloads You can search for downloads in the Download center search bar. You can search by the following: - **Request timestamp.** Search for a specific date range. - **Dataset.** Search by dataset. - **Schema.** Search by schema. - **Encoding.** Search by encoding (DBN, CSV, JSON). You can filter your search results with the following: - **Request timestamp.** Filter by a specific date range. - **Dataset.** Filter by dataset. - **Schema.** Filter by schema. - **Encoding.** Filter by encoding (DBN, CSV, JSON). - **Status.** Downloads have statuses of either **Queued, Processing, Ready,** or **Expired**. You can also sort by request time and billed size. ### Interpreting download statuses Downloads have the following statuses: - **Queued.** We process download requests one by one. When a request has been **Queued,** it's been placed at the end of the "line" and will be processed. - **Processing.** The request is at the front of the "line." We are actively preparing the files on the backend. - **Ready.** The request is now a file that's ready for your use. - **Expired.** The file has expired and can no longer be downloaded. You need to submit a new download request to access the data. Files expire 30 days after a download request has been submitted. ### Download details ![Download details](https://databento.com/docs/download-center-1.png.md) > **See also** > > Data can be downloaded from the API with [batch.download](https://databento.com/docs/api-historical/batch/batch-download.md). Clicking on a download will open up the details of the request. This page will show various details about the download request, which includes the symbols, time range, and customizations selected. Each file can be downloaded one at a time by clicking on the download icon next to each file. Additionally, all files can be downloaded at once in a zip file by clicking on "Download all". ### Support files Each completed batch job package includes the following three supporting JSON files: - `condition.json` - `manifest.json` - `metadata.json` - **condition.json**: This file outlines the dataset condition per date and aids in identifying the appropriate data for processing, including any missing or pending days. The same information can be obtained from [metadata.get\_dataset\_condition](https://databento.com/docs/api-historical/metadata/metadata-get-dataset-condition.md) endpoint. - **manifest.json**: This file lists all the batched data and support files contained in the package, including file names, sizes, and SHA-256 hashes. You can use this information to confirm that your data is complete and free of tampering or corruption. The same information can be obtained from [batch.list\_files](https://databento.com/docs/api-historical/batch/batch-list-files.md) endpoint. - **metadata.json**: This file describes the original request for the batch download, including any advanced customization options. The same information can be obtained from the [batch.get\_job\_details](https://databento.com/docs/api-historical/batch/batch-get-job-details.md) endpoint. If the requested data does not contain symbology information, one or more symbology support files will be included. Each data file will come with an associated symbology file. This file will be named similarly to the data file, except it will end with `*.symbology.json`. This file is only provided when symbol mapping was not requested when using the CSV or JSON encodings. Additionally, for CSV encodings an alternative CSV representation of this file is provided for convenience. These symbology support files provide symbology mappings for the batched data. This allows you to interpret the integer instrument IDs and map these to their corresponding raw symbols across date intervals. The same information can be obtained from the [symbology.resolve](https://databento.com/docs/api-historical/symbology/symbology-resolve.md) endpoint. ### FTP ![Download details](https://databento.com/docs/download-center-2.png.md) Databento also supports downloading data with FTP. FTP is ideal for downloading large datasets reliably, with the ability to browse directories and download multiple files at once. It's well suited for downloading recurring batch requests. Various FTP clients such as [FileZilla](https://filezilla-project.org/) and [lftp](https://lftp.yar.ru/) can be used to download data. --- Source: https://databento.com/docs/portal/team # Team ![Team](https://databento.com/docs/team-0.png.md) ### Overview The following roles are available for team members: - **Owner.** Ability to manage the team, billing, and data usage. - **Admin.** Have the same administrative access as the Owner, but can't transfer team ownership or delete the team. - **Accounting.** View-only access to billing and licensing information. - **Standard.** Data users that don't have any administrative access. A team member can have the following statuses: - **Active.** An active member of the team. - **Invited.** This user has a pending invitation to the team. - **Deactivated.** This member has had their role revoked on the team. - **Note.** A deactivated member can have their access restored by the **Owner** or an **Admin**. When you create a Databento account without a team invitation, you are the team **Owner**. > **See also** > > There is only one **Owner** per team, but you can [transfer ownership](https://databento.com/docs/portal/team.md#transfer-ownership) to a different > team member. ### Member permissions summary | Permissions | **Owner** | **Admin** | **Accounting** | **Standard** | | Transfer team ownership | ✓ | | | | | Delete team | ✓ | | | | | View team member lists | ✓ | ✓ | ✓ | ✓ | | Send and cancel invites | ✓ | ✓ | | | | Update and edit members | ✓ | ✓ | | | | Delete and deactivate members | ✓ | ✓ | | | | View invoices | ✓ | ✓ | ✓ | | | Download invoices | ✓ | ✓ | ✓ | | | View licensing documents | ✓ | ✓ | ✓ | | | View personal monthly spend | ✓ | ✓ | ✓ | ✓ | | Set team historical monthly limit | ✓ | ✓ | | | | Make payments | ✓ | ✓ | | | | Update payment method | ✓ | ✓ | | | | Monitor members' data usage | ✓ | ✓ | | | | Manage data licenses | ✓ | ✓ | | | | Use data | ✓ | ✓ | ✓ | ✓ | ### Invite a team member > **Info** > > Only the **Owner** and **Admins** can invite a member to the team. 1. Click on the **Invite** button. 2. In the **Email** field, enter the email address of the team member you want to invite (this email cannot be associated with an already existing Databento account). 3. Choose the team member's role. 4. Additionally, you can set a monthly limit for this user and a message with the invitation. 5. Click the **Send** button. The invited member will receive an email notification. Before an invited member accepts the invitation, their account status is shown as **Invited**. ### Deactivate a team member > **Info** > > Only the **Owner** and **Admins** can deactivate a team member. A deactivated member cannot log into their account, and their existing integrations are removed. They stay on the team list, but they're shown as deactivated. 1. Click on the overflow menu for the selected team member, and select the option to deactivate. 2. Confirm the deactivation in the confirmation dialogue. To reactivate a member, go to the team list and select the member you want to reactivate. Upon reactivation, their account is fully restored, and they can sign in with their original credentials. ### Delete a team member > **Info** > > Only the **Owner** and **Admins** can delete a team member. Deletion permanently removes a member from the team and deletes their account. The member is removed from the team list, and their existing integrations are removed. 1. Click on the overflow menu for the selected team member, and select the option to delete. 2. Confirm the deletion in the confirmation dialogue. A deleted member can't be reactivated. They can only be added back to the team as a new member. ### Transfer ownership > **Info** > > Only the current **Owner** can transfer the **Owner** role to a different member. 1. Click on the overflow menu for the team member you want to make the new **Owner**, and select the edit option. 2. From the Role dropdown, select the **Owner** role. 3. Confirm the transfer in the confirmation dialogue. Upon transfer of the **Owner** role, the previous Owner's role will change to **Admin**. ### Delete a team > **Info** > > Only the current **Owner** can delete a team. To enable team deletion, you must first cancel all data licenses and pay any outstanding balances. After these steps are complete, navigate to your [Settings page](https://databento.com/settings) by clicking on your avatar in the upper right-hand corner. 1. Under Team deletion, click the Delete button. 2. You will be prompted to confirm the deletion by entering your password. 3. After deletion, all team members will immediately be signed out and lose both portal and API access. > **Warning** > > Team deletion is permanent, and you won't be able to recover your team. ![Delete team](https://databento.com/docs/team-1.png.md) --- Source: https://databento.com/docs/portal/billing # Billing ![Billing page](https://databento.com/docs/billing-0.png.md) ### Overview Owners and Admins have edit access to all billing features, while Accounting members have view-only access. For more information on roles, see our [Team management guide](https://databento.com/docs/portal/team.md). The current balance on your account, as well as any remaining credits, will be shown in the top left corner of the billing page. > **Info** > > Applicable credits are automatically subtracted from the current balance. Clicking the **View Details** button in this section will pull up a detailed description of your current balance. ![Current balance](https://databento.com/docs/billing-1.png.md) ### Update payment method Payment method(s) can be updated after account creation. 1. Click the **Update** button in the payment method section. 2. You have the option to add, update, or delete credit cards. 3. Additionally, you can set different cards as your default method. > **Info** > > Databento only supports ACH and wire transfer for members with a Plus or Unlimited plan. ![Update credit card](https://databento.com/docs/billing-2.png.md) ### Manage historical monthly limit 1. Click the **Manage** button in the usage-based access section. 2. Turn on the **Enable pay as you go** toggle. 3. Turn on the **Set a monthly limit** toggle and enter the desired limit amount. 4. To receive an email when you've approached a part of your limit, turn on the **Receive email notifications** toggle and set the desired limit. ![Monthly limit](https://databento.com/docs/billing-3.png.md) > **Info** > > Databento will block further historical data requests once your monthly usage exceeds your limit. > If you don't have a historical monthly limit set, your account will be set to **Unlimited**. Your > data requests will never be blocked based on usage amount. Live data requests are not subject to > this limit; this usage is always **Unlimited**. ### Billing timeline Each billing cycle starts on the first day of the month and ends on the last day of the month. Each invoice is available on the first day of the following month. Your invoice includes all download requests initiated until 23:59:59 UTC on the last day of the month. > **Info** > > Your payment date is always 5 calendar days after the end of the month. - The **Date** column of the **Invoices** table marks the last day of the billing cycle it represents. The **Date** column doesn't represent the day the invoice is first available. - **Example.** You open a Databento account on June 3. Your billing cycle ends on June 30, and your first invoice is available on July 1. All your download requests from June appear in your first invoice. Your first payment is due on July 5. ### Invoice status There are four possible invoice statuses: - **Upcoming payment.** The invoice has been sent but not yet paid. An invoice status is **Upcoming payment** between the last day of the month and the payment date. - Depending on your activity, your invoice may be changed during this time. - **Example.** You submitted a download request shortly before 23:59:59 UTC on the last day of the month. Data continues to be downloaded to your account, so your invoice is changed to reflect this. - **Paid.** The invoice payment was successful. - **Past due.** The invoice payment was unsuccessful. - **Canceled.** Databento manually cancels the invoice charge. - You can't do this directly in your portal, but you can reach out to our support team and request an invoice cancellation. ### Failed or late payments If your payment fails, we immediately email you a notification with details. Your account is on hold from requesting new data and downloading existing datasets. Your invoice is also marked as **Past due**. To resolve your payment issue and unblock data access, update your payment method. Here's what to expect if we still can't process a payment: - **≤ 5 business days after the payment date.** We'll keep retrying your payment method automatically. If the payment is made successfully, your invoice will be marked as **Paid.** - **> 5 business days after the payment date.** Your invoice will be marked as **Past due.** You'll need to work with our support team to manually resolve the late payment. Here are common reasons why failed payments may happen: - **Insufficient funds.** Make sure there are sufficient funds on your card, especially toward your monthly bill date. - **You reached your credit card limit.** Some credit cards have limits for the maximum amount that can be charged in a single transaction. If you have a limit on your credit card, you can decrease your payment threshold to match this amount. - **Processing/verification issue.** Most payments fail when your bank or payment provider experiences an issue processing your payment. Check the expiration date on your credit card to make sure it isn't expiring for your future charges. - **Incorrect bank account information**: Make sure the bank account information (ex: account number, bank code, or bank name) in your Payment Settings is correct. ### Reference dataset pricing Databento offers three subscription plans for reference datasets: - The Standard plan includes access to any 1,000 symbols. - The Unlimited (US only) plan includes all 10,000+ US symbols. - The Unlimited (US & Global) plan includes all 310,000+ US and global symbols. > **Info** > > Symbol limits are based on the [**I**nternational **S**ecurities **I**dentification **N**umber (ISIN)](https://www.isin.org/) - > ISO 6166 system.
ISIN codes don't always correspond with the symbols used on exchanges. If you don't need full coverage of US or global securities, the **Standard plan** provides access to a limited number of symbols (ISINs) on an annual basis. We keep track and count any new symbols when they're requested on our API. You won't have to decide on these symbols upfront. We reset this each time your subscription renews. - **Initial allocation.** Up to 1,000 symbols. - **Allocation management.** - Each successful request that includes a new symbol will count towards the allocated symbol limit. - API requests exceeding the allocated symbol limit will return an authorization error. - Our support team can provide you with a list of symbols that you've consumed. - **Additional allocation.** Additional symbols can be requested for $1 per month, but this will reset the contract term. - **Plan upgrades.** You can upgrade from Standard to Unlimited at any time. The remainder of your term on the Standard plan gets terminated, and we'll start you on a new 12-month term. For more plan comparisons and pricing information, see the calculator on our [corporate actions page](https://databento.com/corporate-actions). --- Source: https://databento.com/docs/portal/live-data # Plans and live data ![Live data portal](https://databento.com/docs/live-data-0.png.md) ### Overview As a licensed distributor for 60+ trading venues, Databento redistributes real-time and intraday market data from the last 24 hours. The [Plans and live data page]($PLATFORM_HOSTNAME/portal/live-data) of your portal allows you to manage your subscription plans and live data licensing. ### Activate live data > **Info** > > Real-time data is included with a Standard, Plus, or Unlimited plan. View our [pricing page]($PLATFORM_HOSTNAME/pricing) to explore live access by dataset. 1. Navigate to the [Plans and live data page]($PLATFORM_HOSTNAME/portal/live-data) of your Databento portal. 2. Sign up for a qualifying subscription. 3. Complete a short questionnaire for your desired dataset to determine your subscriber status. You can also upload an existing license agreement. Instant approval is available for certain venues and use cases. Your contracts will be auto-generated to meet venue licensing requirements, and you can view clear pricing breakdowns before signing your documents. Databento will then operate as a vendor of record (VoR) and report your devices under our license agreement. ### Venue licensing requirements Each venue defines subscriber statuses, use cases, device counts, and their corresponding license fees differently. License fees refer to any monthly charges imposed by venues for the internal and external use (display, non-display, and distribution) of live market data for personal or commercial applications. - **Internal.** Any use of market data within the licensed company or individual user's private environment. - **External.** Any use of market data that involves sharing, disseminating, or making data visible to external parties outside the licensed organization or individual. - **Personal.** Market data is used solely for a non-professional individual's own investment decisions, research, or educational purposes and cannot be shared or leveraged for profit. - **Commercial.** Market data is used within a company, by a professional individual, or in applications that generate profit, serve clients, or support business operations. Databento standardizes venue terminology to streamline licensing workflows. See the table below for general definitions and examples. | Display use | Market data is shown ("displayed") in a human-readable, visible, or audible format on a terminal, screen, or other device internally. **Internal** use for **personal** or **commercial** applications. | | Non-display use | Market data is used without being displayed in a human-readable format, such as being processed directly by a device. This includes algorithmic trading (e.g., automated order or quote generation), internal analytics, or other non-public backend applications. Note that some venues don't automatically consider API use as a non-display use case. **Internal** use for a **commercial** application. Some venues support **personal** non-display use. | | Distribution | Market data is shared or distributed to external users, e.g., raw data feeds on a white-labeled API or displaying real-time quotes on a customer-facing web app. **Real-time distribution** is typically within 10 minutes of receipt, while **delayed distribution** is at least 10 minutes after receipt and before the end of the trading session. Cutoff times may vary by venue. **External** use for a **commercial** application. | > **See also** > > [Databento's market microstructure guide](https://databento.com/microstructure) or [three-part licensing FAQ series](https://databento.com/blog/introduction-market-data-licensing) for > more information and examples. ### Applying for a license agreement Most commercial users* without an existing license will need to apply for a direct license agreement with the venue before they can access live data on Databento or via any other data vendor. Databento facilitates this process for you. To get started, click **Activate live data** on [Plans and live data page]($PLATFORM_HOSTNAME/portal/live-data) and complete the license questionnaire. 1. Start the application by completing the form. 2. Databento will use this information to submit a data feed request (DFR) to the venue on your behalf. 3. The venue will contact you via email to complete any additional paperwork and confirm license fees. 4. Databento will receive approval from the venue and activate live data on the effective date of your license agreement. > **Info** > > *IEX TOPS, ICE Europe Commodities, ICE Europe Financials, ICE Endex, and ICE Futures US offer instant access for commercial use cases. ### Venue license fees License fees are assessed by the venue and are typically charged per user or device each month. If you're licensed through Databento, fees are passed through without an upcharge. However, a 2.9% Stripe processing fee is applied to license fees. These are paid monthly in advance, and you can terminate at any time. This table summarizes 2026 license fees per dataset, alongside an overview of each venue's supported applications and a link to its official fee schedule. The monthly license fees listed below reflect the most common use cases among Databento users. | Dataset | Supported applications | Monthly license fees | | [Blue Ocean ATS MEMOIR Depth](https://databento-media.s3.us-east-1.amazonaws.com/media/blue-ocean-market-data-fee-schedule.pdf) | Commercial | $1,500/firm (display) $5,000/firm (non-display) $1,500 distribution + $8/user | | [CFE](https://databento-media.s3.us-east-1.amazonaws.com/media/cboe-market-data-fee-schedule.pdf) | Personal | Included with Standard plan | | | Commercial | $750/firm (display or non-display) $750 distribution + $3/personal user or $17/commercial user | | [CME](https://api.databento.com/static/licensing/cme/cme-market-data-fee-list.pdf) | Personal | Included with Standard plan (up to 2 devices) | | | Commercial | $973/exchange $2,170 distribution + $35.40/personal user or $119.80/commercial user | | **EEX (Plus)** | Personal and commercial | €0/team (display or non-display) | | [EEX (Unlimited)](https://www.mds.deutsche-boerse.com/resource/blob/4694152/0bd9dad3047b437ed880ab0f656e0480/data/MDDA_Price_List_14_4.pdf#page=6) | Personal and commercial | €3,777/team (display or non-display) | | [Eurex](https://www.mds.deutsche-boerse.com/resource/blob/3685372/bed9ddba2d52df15d41928c240c6a272/data/MDDA_Price_List_13_7.pdf#page=35) | Personal | Included with Standard plan (up to 2 devices) | | | Commercial | €1,817/firm (non-display) €3,074.00 distribution | | [ICE Europe Commodities](https://databento-media.s3.us-east-1.amazonaws.com/media/ice-futures-market-data-fee-schedule.pdf) | Personal and commercial | $865/location + $144/terminal (display or non-display) | | [ICE Europe Financials](https://databento-media.s3.us-east-1.amazonaws.com/media/ice-futures-market-data-fee-schedule.pdf) | Personal and commercial | $740/location + $123/terminal (display or non-display) | | [ICE Endex](https://databento-media.s3.us-east-1.amazonaws.com/media/ice-futures-market-data-fee-schedule.pdf) | Personal and commercial | $865/location + $144/terminal (display or non-display) | | [ICE Futures US](https://databento-media.s3.us-east-1.amazonaws.com/media/ice-futures-market-data-fee-schedule.pdf) | Personal and commercial | $795/location + $132/terminal (display or non-display) | | **Databento US Equities Mini** | Personal and commercial | No license fees | | [Nasdaq Basic with NLS Plus](https://data.nasdaq.com/price-list?category=U.S.+Equities) | Commercial only | $1,680/firm + $14.10/user (display or non-display) $2,140 distribution + $0.50/personal user or $14.10/commercial user | | [Nasdaq TotalView-ITCH](https://data.nasdaq.com/price-list?category=U.S.+Equities) | Commercial only | $1,690/firm + $84.00/user (display or non-display) $4,230 distribution + $15.00/personal user or $84.00/commercial user | | [NYSE Texas Integrated](https://www.nyse.com/publicdocs/nyse/data/NYSE_Market_Data_Pricing.pdf) | Commercial only | No license fees | | [NYSE National Trades and BBO](https://www.nyse.com/publicdocs/nyse/data/NYSE_Market_Data_Pricing.pdf) | Commercial only | No license fees | | [NYSE Order Imbalances](https://www.nyse.com/publicdocs/nyse/data/NYSE_Market_Data_Pricing.pdf) | Commercial only | $2,500/firm (non-display) $0 distribution | | [NYSE Arca Order Imbalances](https://www.nyse.com/publicdocs/nyse/data/NYSE_Market_Data_Pricing.pdf) | Commercial only | $1,000/firm (non-display) $0 distribution | | [NYSE American Order Imbalances](https://www.nyse.com/publicdocs/nyse/data/NYSE_Market_Data_Pricing.pdf) | Commercial only | $1,000/firm (non-display) $0 distribution | | [Cboe EDGX Depth](https://databento-media.s3.us-east-1.amazonaws.com/media/cboe-market-data-fee-schedule.pdf) | Commercial only | $1,500/firm + $2,500 distribution | | [OPRA](https://api.databento.com/v0/licensing/documents/opra/opra-fee-schedule.pdf) | Personal | $1.25/display or non-display device | | | Commercial | $2,000/firm (display or non-display) $1,500 distribution + $1.25/personal user or $31/commercial user | --- Source: https://databento.com/docs/quickstart?lang=python # Quickstart --- Source: https://databento.com/docs/quickstart/set-up?lang=python ## Set up Databento You need an API key to request data from Databento. [Sign up]($PLATFORM_HOSTNAME/signup) and you will automatically receive an API key. Each API key is a 32-character string starting with `db-`, that can be found on the [API keys]($PLATFORM_HOSTNAME/portal/keys) page on your portal. > **Info** > > Every new account receives [$125 in free data credits](https://databento.com/docs/faqs/usage-pricing-and-data-credits.md). > These credits allow you to start receiving actual historical data immediately > and verify your integration at no cost. --- Source: https://databento.com/docs/quickstart/choose-service?lang=python ## Choose a service Databento provides two data services: historical and live. These two services are nearly identical, but we separate them due to licensing fees and the differences between request-response and real-time streaming APIs. You can choose to integrate just one or both services. | | Historical | Live | Reference | | Coverage | Historical and delayed intraday data within the exchange's live data restrictions. | Intraday data from within the last 24 hours and real-time data. | Historical with intraday updates. | | Pricing | Usage-based or flat-rate. No monthly license fees. | Flat-rate. Monthly license fees apply. | Dataset license fees apply. | | Access | Client libraries (C++, Python, and Rust) and API (HTTP). | Client libraries (C++, Python, and Rust) and API (Raw). | Client libraries (Python and Rust) and API (HTTP). | --- Source: https://databento.com/docs/quickstart/build-first-app?lang=python ## Build your first application Historical Live Reference --- Source: https://databento.com/docs/quickstart/historical?lang=python ## Getting historical data Select how you will be integrating Databento below to see installation instructions. By default, Python has been selected for you. If you don't see an official client library for your preferred language, you can still integrate our historical service through its HTTP API. Install our Python client library with pip. Python 3.10+ is required. ```sh $ pip install databento ``` A simple Databento historical application looks like this. This replays 10 minutes of trades of the entire CME Globex market event-by-event. Copy this to a file `main.py`. Then, run the file with `python main.py`. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols="ALL_SYMBOLS", start="2022-06-02T14:20:00", end="2022-06-02T14:30:00", ) data.replay(print) ``` You can modify this application to specify particular [instruments](https://databento.com/docs/faqs/instruments-and-products.md), and [schemas](https://databento.com/docs/schemas/whats-a-schema.md). Let's get `ESM2` and `NQZ2` data in 1-second OHLCV bars. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=["ESM2", "NQZ2"], schema="ohlcv-1s", start="2022-06-06T14:30:00", end="2022-06-06T14:40:00", ) data.replay(print) ``` You have successfully written your first historical data application with Databento! Here are shortcuts to some of the next steps you can take: - To download a large amount of data to disk, see how to do a [batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md) of data files. - To get another dataset, just swap the dataset. You can get a list of datasets and their names from our [metadata](https://databento.com/docs/api-historical/metadata.md). - You can use our [symbology](https://databento.com/docs/api-historical/symbology.md) API to find other instrument IDs. - You can also find dataset names and instrument IDs interactively from our [search]($PLATFORM_HOSTNAME). To learn more, read the full documentation for our historical service under [API reference - Historical](https://databento.com/docs/api-historical.md). --- Source: https://databento.com/docs/quickstart/live?lang=python ## Getting live data Select how you will be integrating Databento below to see installation instructions. If you don't see an official client library for your preferred language, you can still integrate our service via the Raw API. Install our Python client library with pip. Python 3.10+ is required. ```sh $ pip install -U databento ``` A simple Databento live application looks like this. This prints 10 seconds of trades for all ES futures contracts event-by-event. Copy this to a file `main.py`. Then, run the file with `python main.py`. ```python import databento as db client = db.Live(key="$YOUR_API_KEY") client.subscribe( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols="ES.FUT", ) client.add_callback(print) client.start() client.block_for_close(timeout=10) ``` You can modify this application to specify particular [instruments](https://databento.com/docs/faqs/instruments-and-products.md), and [schemas](https://databento.com/docs/schemas/whats-a-schema.md). Let's get `ES.FUT` and `NQ.FUT` data in 1-second OHLCV bars. ```python import databento as db client = db.Live(key="$YOUR_API_KEY") client.subscribe( dataset="GLBX.MDP3", schema="ohlcv-1s", stype_in="parent", symbols=["ES.FUT", "NQ.FUT"], ) client.add_callback(print) client.start() client.block_for_close(timeout=10) ``` You have successfully written your first live data application with Databento! To learn more, read the full documentation for our live service under [API reference - Live](https://databento.com/docs/api-live.md). --- Source: https://databento.com/docs/quickstart/reference?lang=python ## Getting reference data Select how you will be integrating Databento below to see installation instructions. By default, Python has been selected for you. If you don't see an official client library for your preferred language, you can still integrate our reference service through its HTTP API. Install our Python client library with pip. Python 3.10+ is required. ```sh $ pip install databento ``` A simple Databento reference application looks like this. This request fetches all dividend event records for Microsoft from January 2019 until June 2024. You can modify this request to specify particular [symbols](https://databento.com/docs/faqs/instruments-and-products.md), and [events](https://databento.com/docs/datasets/corporate-actions.md#events). Copy this to a file `main.py`. Then, run the file with `python main.py`. ```python import databento as db client = db.Reference("$YOUR_API_KEY") df = client.corporate_actions.get_range( start="2019", end="2024-07", symbols=["MSFT"], events=["DIV"], countries=["US"], ) print(df) ``` You have successfully written your first reference data application with Databento! Here are shortcuts to some of the next steps you can take: - Read the corporate actions [dataset guide](https://databento.com/docs/datasets/corporate-actions.md). - Become familiar with the reference data [enum lookup tables](https://databento.com/docs/conventions/reference-data-enums.md). - Go through the corporate actions [examples](https://databento.com/docs/examples/corporate-actions.md). To learn more, read the full documentation for our reference service under [Reference API](https://databento.com/docs/api-reference-data.md). --- Source: https://databento.com/docs/quickstart/new-user-guides?lang=python ## New user guides **[What's a schema?](https://databento.com/docs/schemas/whats-a-schema.md)** See fields and columns included with each schema. ![Supported Schemas](https://databento.com/docs/supported-chemas.svg.md) **[Symbology](https://databento.com/docs/conventions/symbology.md)** How to interpret instrument IDs, symbols, and definitions. ![Symbology](https://databento.com/docs/quickstart-symbology.svg.md) **[Normalization format](https://databento.com/docs/conventions/normalization.md)** Learn more about our data normalization. ![Normalization Format](https://databento.com/docs/normalization-format.svg.md) **[DBN files and encoding](https://databento.com/docs/conventions/databento-binary-encoding.md)** Save and read our data in DBN, our lightning-fast binary format. ![DBN Encoding](https://databento.com/docs/files-and-encoding.svg.md) **[Equities](https://databento.com/docs/examples/equities/equities-introduction.md)** Get started on equities data like Nasdaq TotalView-ITCH. ![Equities](https://databento.com/docs/equities.svg.md) **[Equity options](https://databento.com/docs/examples/options/equity-options-introduction.md)** Get started on equity options data like OPRA. ![Equity Options](https://databento.com/docs/equity-options.svg.md) **[Futures](https://databento.com/docs/examples/futures/futures-introduction.md)** Get started on futures data like CME Globex MDP 3.0. ![Futures](https://databento.com/docs/futures.svg.md) **[Options on futures](https://databento.com/docs/examples/options/options-on-futures-introduction.md)** Get started on options on futures data. ![Futures Options](https://databento.com/docs/futures-options.svg.md) --- Source: https://databento.com/docs/release-notes # Release notes --- Source: https://databento.com/docs/release-notes/release-notes-cpp # C++ ## 0.69.0 - Upcoming #### Enhancements - Added `Tbbo` variant to `RType` for TBBO records in DBN version 4 - Added `Unsubscribe()` to `LiveBlocking` and `LiveThreaded` for removing symbols from a live session's subscriptions - Added `Unsubscriptions()` to `LiveBlocking` and `LiveThreaded` - Modified `Resubscribe()` to resubscribe only to active subscriptions #### Breaking changes - Added `sent_at` field to `LiveSubscription` ## 0.68.1 - 2026-09-30 #### Enhancements - Optimized DBN decoding to decode records in batches through libdbn_c - Upgraded libdbn_c version to 0.71.0 - Upgraded default cpp-httplib version to 0.57.1 #### Bug fixes - Removed clang-format target creation when not the main project ## 0.68.0 - 2026-09-18 #### Enhancements - Added new venue, dataset, and publishers for Databento Core Indices - Upgraded libdbn_c version to 0.70.0 - Upgraded default cpp-httplib version to 0.54.1 ## 0.67.0 - 2026-09-01 #### Enhancements - Added `UnsubscribeAck` to the `SystemCode` enum for acknowledging unsubscribe requests - Added support for building against cpp-httplib 0.52.0 and newer, which changed the underlying container of `httplib::Params` and `httplib::Headers` - Removed the `httplib.h` include from the public headers, cutting about a third of the preprocessed size of `record.hpp`, `historical.hpp`, and `live.hpp` - Upgraded libdbn_c version to 0.69.0 - Upgraded default cpp-httplib version to 0.53.1 #### Bug fixes - Fixed incorrect count in DBN metadata causing decoding to read into the start of records - Fixed CMake error when the consuming project already defines a `zstd::libzstd` target (credit: Mamadou Babaei) ## 0.66.0 - 2026-08-25 #### Enhancements - Added a dependency on libdbn_c, the C interface to the [DBN library](https://github.com/databento/dbn), which CMake downloads as a prebuilt library or can be built from source with Cargo - Decoding DBN is now done via the C interface of the DBN library rather than a separate C++ implementation - Added CMake configurations `DATABENTO_BUILD_DBN_FROM_SOURCE` for always building libdbn_c from source and `DATABENTO_DBN_SOURCE_DIR` for building it from a local DBN checkout - Added an overload of `Historical::MetadataListFields` that accepts a `dataset` parameter. Without it, the returned fields are for the latest DBN encoding version, which may not match a specific dataset's schema - Changed `LiveBlocking::Reconnect` to reuse its DBN decoder rather than rebuilding it, so the read buffer keeps the capacity it grew #### Breaking changes - Changed `Historical::BatchListJobs` to request the new short response format of the `batch.list_jobs` endpoint and return `std::vector`, which contains only the `id`, `state`, and `ts_received` fields for each job. Use `Historical::BatchGetJobDetails` to fetch the complete details of an individual job - Removed `DbnDecoder::DecodeMetadataVersionAndSize`, `DbnDecoder::DecodeMetadataFields`, `DbnDecoder::DecodeRecordCompat`, and `DbnDecoder::NeedsUpgrade`, which the DBN library now handles #### Bug fixes - Changed decoding DBN version 3 data with `VersionUpgradePolicy::UpgradeToV2` to throw a `DbnResponseError`, which the policy was already documented to do, instead of decoding the data as-is #### Deprecations - Deprecated `Historical::BatchListJobsFull`, which returns the full job details like `BatchListJobs` did previously. The `batch.list_jobs` endpoint will stop returning full job details at a future date, at which point this method will be removed ## 0.65.0 - 2026-08-18 #### Enhancements - Added `SType` variants `ListingId`, `IssuerId`, and `SecurityId` for the reference data API ## 0.64.0 - 2026-08-11 #### Breaking changes - Changed `LiveBlocking` to throw a `LiveApiError` instead of an `InvalidArgumentError` when the gateway rejects authentication, so the reason is no longer mislabeled as an invalid `key` argument #### Bug fixes - Fixed Live connect timeouts being reported as `Operation now in progress`, and added the gateway, port, and timeout to the error message - Fixed a Live connect interrupted by a signal failing instead of continuing to wait for the connection - Fixed only the first address being tried when a Live gateway hostname resolves to more than one ## 0.63.0 - 2026-08-04 #### Enhancements - Added `ReplayDataAgedOut` variant to `ErrorCode` - Added new venues, datasets, and publishers for JPX - Upgraded default date version to 3.0.5 ## 0.62.1 - 2026-07-28 #### Enhancements - Upgraded default cpp-httplib version to 0.51.0 #### Bug fixes - Fixed `DbnEncoder::EncodeRecord` overload for `WithTsOut` failing to compile ## 0.62.0 - 2026-07-14 #### Enhancements - Changed `VersionUpgradePolicy` to be backed by `std::uint8_t` with explicit values starting at 1 - Added `HistoricalBuilder::SetHttpClientConfig` for customizing the underlying HTTP client ## 0.61.0 - 2026-07-07 #### Enhancements - Added new publisher values for Cboe Titanium Cboe Global Indices Feed - Added `StatType` variants `MwcbLevel1`, `MwcbLevel2`, and `MwcbLevel3` - Added `StatType` variants `AuctionCollarReferencePrice`, `AuctionCollarUpperPrice`, and `AuctionCollarLowerPrice` - Upgraded default cpp-httplib version to 0.49.0 #### Breaking changes - Renamed the following Venue, Dataset, and Publisher: - `CGIF` to `MAIN` - `CGIF.TITANIUM` to `MAIN.CGIF` - `CGIF.TITANIUM.CGIF` to `MAIN.CGIF.MAIN` ## 0.60.0 - 2026-06-16 #### Enhancements - Added `MatchAlgorithm` variant `Allocation` ## 0.59.0 - 2026-06-02 #### Enhancements - Added `Historical::BatchGetJobDetails` for querying the details for a single batch job - Added new venues, datasets, and publishers for US Equities Securities Information Processors #### Bug fixes - Fixed `HistoricalBuilder::SetAddress` not working with HTTPS ## 0.58.0 - 2026-05-26 - Added logging for gateway `ErrorMsg` and `SystemMsg` records in `LiveBlocking`. Suppress by raising the minimum level or by passing a `NullLogReceiver` - Upgraded default cpp-httplib version to 0.46.0 ## 0.57.0 - 2026-05-12 #### Enhancements - Added `Case()`, `DetailMessage()`, and `DocsUrl()` getters on `HttpResponseError`. These parse the JSON error envelope returned by the historical API and expose its fields directly - Included the error `case` (when present) in the message returned by `HttpResponseError::what()` #### Breaking changes - Removed `HttpResponseError::ResponseBody()`. The raw body is still embedded in the message returned by `what()`; use the new `Case()`, `DetailMessage()`, and `DocsUrl()` getters for structured access ## 0.56.0 - 2026-05-05 #### Enhancements - Added `Index` `InstrumentClass` variant - Added `StatType` variants `IndicativeClosePrice` and `VenueSpecificPrice1` ## 0.55.0 - 2026-04-28 #### Enhancements - Improved `DbnDecoder` throughput on current-version data and `AsIs` workloads by caching whether the upgrade policy-version combination requires upgrading, skipping the per-record `DecodeRecordCompat` dispatch on the fast path - Made `detail::Buffer` shifts explicit to avoid redundant moves during record decoding - Added new publisher values for Cboe Titanium Cboe Global Indices Feed - Added `Year` to `SplitDuration` enum for yearly historical batch job submissions - Upgraded default cpp-httplib version to 0.43.1 - Upgraded default nlohmann/json version to 3.12.0 ## 0.54.0 - 2026-04-21 #### Enhancements - Added new publisher values for OPRA MEMX MX2 Options and IEX Options ## 0.53.0 - 2026-04-08 #### Enhancements - Added `TryNextRecord` and `FillBuffer` to `LiveBlocking` for more fine-grained control around I/O - Added `TimeoutConf` struct and `SetTimeoutConf()` builder method for configuring connect and auth timeouts on the Live client (defaults to 10s and 30s) - Added `SessionId()` and `Timeouts()` getters to `LiveBlocking` and `LiveThreaded` ## 0.52.0 - 2026-03-31 #### Enhancements - Added client-side heartbeat timeout detection: `NextRecord` throws `HeartbeatTimeoutError` if no data is received for `heartbeat_interval` + 5 seconds (defaults to 35 seconds) - Changed `SlowReaderBehavior::Skip` to send "skip" instead of "drop" to the gateway #### Breaking changes - `NextRecord` now throws `LiveApiError` instead of `DbnResponseError` when the gateway closes the session. Code catching `DbnResponseError` for this case should be updated ## 0.51.0 - 2026-03-17 #### Enhancements - Added support for `progress` field in `BatchJob` response - Added blocking (pull-based) `TimeseriesGetRange` overloads that return a `DbnStore` with `GetMetadata()` and `NextRecord()` methods, unifying the consumption pattern across historical, live, and file sources - Generalized `DbnFileStore` into `DbnStore` which accepts any `IReadable` source. `DbnFileStore` is retained as a type alias for backwards compatibility - Upgraded default `httplib` version to 0.37.2 ## 0.50.0 - 2026-03-03 #### Enhancements - Added logic to set `code` when upgrading version 1 `ErrorMsg` to newer versions ## 0.49.0 - 2026-02-24 #### Enhancements - Added `SkippedRecordsAfterSlowReading` to the `ErrorCode` enum for gateway errors originating from slow client catch-up ## 0.48.0 - 2026-02-18 #### Enhancements - Added `SlowReaderBehavior` enum and `LiveBuilder::SetSlowReaderBehavior()` to configure gateway behavior when client falls behind - Added `SlowReaderBehavior()` getter to `LiveBlocking` and `LiveThreaded` #### Bug fixes - Added conversion for missing schemas for function `RTypeFromSchema` - Added explicit optional construction in `json_helpers.hpp` (credit: Enrico Detoma) ## 0.47.0 - 2026-02-04 #### Enhancements - Added Zstd compression support to live clients which can be enabled with `LiveBuilder::SetCompression()`. It's disabled by default - Added `Compression()` getter to `LiveBlocking` and `LiveThreaded` - Upgraded default `httplib` version to 0.30.1 #### Breaking changes - Added an overload to the `ReadSome` method on `IReadable` for timeout support ## 0.46.1 - 2026-01-27 #### Enhancements - Added new publisher for Blue Ocean ATS (`OCEA_MEMOIR_OCEA`) ## 0.46.0 - 2026-01-20 #### Enhancements - Added new off-market publisher for Cboe Futures Exchange (`XCBF_PITCH_XOFF`) - Added new `StatType` variants to be used by `XCBF.PITCH` dataset: - `UpperPriceLimit` - `LowerPriceLimit` - `BlockVolume` - `VenueSpecificVolume1` ## 0.45.0 - 2025-12-10 #### Enhancements - Added download retry, resumption, and checksum verification to `Historical::BatchDownload` - Added new venue, dataset, and publisher for Cboe Futures Exchange (`XCBF.PITCH`) - Upgraded default `httplib` version to 0.28.0 ## 0.44.0 - 2025-11-18 #### Enhancements - Added logic to set `code` when upgrading version 1 `SystemMsg` to newer versions #### Bug fixes - Added missing `` include to `constants.hpp` (credit: Ognian Milanov) ## 0.43.0 - 2025-10-22 #### Enhancements - Made `Unset` a fully-supported variant of `SystemCode` and `ErrorCode` #### Breaking changes - Removed `mode` parameter in `Historical::MetadataGetCost()` ## 0.42.0 - 2025-08-19 #### Enhancements - Added `EndOfInterval` variant to `SystemCode` #### Breaking changes - Removed `bill_id` field from `BatchJob` struct ## 0.41.0 - 2025-08-12 #### Enhancements - Added static `Builder()` methods to the clients - Improved debug logging in live clients - Added `PUBLISHER_SPECIFIC` flag #### Breaking changes - Removed unused `Received` variant from `JobState` enum ## 0.40.0 - 2025-07-29 #### Enhancements - Added `OhlcvEod` variant to `RType` - Added documentation to more enums and record structs - Aligned string representation of `RType` with Rust #### Breaking changes - Switched to `std::byte` for most padding fields in record structs. These fields are now initialized by default - Hidden padding fields now begin with `_` #### Bug fixes - Added missing `IndexTs()` method to `v1::InstrumentDefMsg`, `v2::InstrumentDefMsg`, and `v1::SymbolMappingMsg` - Fixed error in templated overload of `DbnDecoder::EncodeRecord()` - Removed unused `Disk` and `S3` variants from `Delivery` enum ## 0.39.1 - 2025-07-22 #### Bug fixes - Fixed potential corrupted data in `DbnFileStore::Replay` ## 0.39.0 - 2025-07-15 #### Breaking changes - Renamed the following Venue, Dataset, and Publishers: - `XEER` to `XEEE` - `XEER.EOBI` to `XEEE.EOBI` - `XEER.EOBI.XEER` to `XEEE.EOBI.XEEE` - `XEER.EOBI.XOFF` to `XEEE.EOBI.XOFF` #### Bug fixes - Fixed potential corrupted records when upgrading DBN versions in live clients - Removed unused `Intraday` variant from `DatasetCondition` enum ## 0.38.2 - 2025-07-01 #### Enhancements - Added `LogPlatformInfo` function which logs the client version, compiler, and operating system info to aid troubleshooting - Added compiler and operating system info to the user agent to aid troubleshooting - Standardized `client` info sent by live clients to match historical - Added methods to the client builders to extend the user agents with a custom string #### Bug fixes - Fixed missing implementation for `LiveBuilder::SetBufferSize()` - Fixed checking of warnings from server in Historical API in `TimeseriesGetRange()` and `TimeseriesGetRangeToFile()` ## 0.38.1 - 2025-06-25 #### Enhancements - Added `range_by_schema` field to `DatasetRange` struct - Changed historical `TimeseriesGetRange()` and `TimeseriesGetRangeToFile()` methods to use a `POST` request to allow for requesting supported maximum of 2000 symbols - Added logging around `Historical::BatchDownload` - Changed the following Venue, Publisher, and Dataset descriptions: - "ICE Futures Europe (Financials)" renamed to "ICE Europe Financials" - "ICE Futures Europe (Commodities)" renamed to "ICE Europe Commodities" #### Bug fixes - Fixed handling of `null` `last_modified_date` in `MetadataGetDatasetCondition` response - Fixed default `ShouldLog` implementation ## 0.38.0 - 2025-06-10 #### Enhancements - Made the buffer size used by the live clients when reading from the TCP socket configurable through the `LiveBuilder::SetBufferSize()` method - Added log level prefix to `ConsoleLogReceiver` output - Added `iomanip` compatibility: fill, precision, and width to `pretty::Px` (formerly `FixPx`) - Added new `pretty::Ts` helper type for human-readable formatting of `UnixNanos` #### Breaking changes - Live client instances can only be created through the `LiveBuilder` class - Changed `HeartbeatInterval()` getters on `LiveBlocking` and `LiveThreaded` to return an `std::optional` - Added new optional `ShouldLog` virtual method to `ILogReceiver` to filter the levels of log messages that will be sent to the receiver #### Deprecations - Deprecated `FixPx` in favor of `pretty::Px` which has consistent naming with the API and the Python and Rust client libraries #### Bug fixes - Ensure `CPPHTTPLIB_OPENSSL_SUPPORT` is defined at all locations where `cpp-httplib` is included ## 0.37.1 - 2025-06-03 #### Bug fixes - Fixed issue where not all data was processed in `Historical::TimeseriesGetRange()` - Fixed issue with pointer arithmetic in `Buffer::Write()` - Fixed issue where more data than necessary was copied in `Buffer::Shift()` ## 0.37.0 - 2025-06-03 #### Breaking changes - Changed the type of the `output_dir` parameter in `HistoricalClient::BatchDownload()` to a `std::filesystem::path` and the return type to `std::filesystem::path` - Changed the type of the `file_path` parameter in `HistoricalClient::TimeseriesGetRangeToFile()` to a `std::filesystem::path` - Changed the type of the `file_path` parameter in the `DbnFileStore`, `InFileStream`, and `OutFileStream` constructors to a `std::filesystem::path` - Made `stype_in` and `schema` optionals in `Metadata` to align with how `Metadata` is represented in other languages and removed the boolean `has_mixed_stype_in` and `has_mixed_schema` fields #### Bug fixes - Fixed behavior where encoding metadata could lower the `version` ## 0.36.0 - 2025-05-27 This version marks the release of DBN version 3 (DBNv3), which is the new default. API methods and `DbnDecoder` support decoding all versions of DBN, but now default to upgrading data to version 3. #### Enhancements - Added `UpgradeToV3` variant to `VersionUpgradePolicy` enum - Version 1 and 2 structs can be converted to version 3 structs with templated `Upgrade` method - Metadata will now always be encoded with a length divisible by 8 bytes for better alignment - Added `is_last` field to live subscription requests which will be used to improve the handling of split subscription requests #### Breaking changes - Release of DBN version 3: - Definition schema: - Updated `InstrumentDefMsg` with new `leg_` fields to support multi-leg strategy definitions. - Expanded `raw_instrument_id` to 64 bits to support more venues. Like other 64-bit integer fields, its value will now be quoted in JSON - Removed `trading_reference_date`, `trading_reference_price`, and `settl_price_type` fields which will be normalized in the statistics schema - Removed `md_security_trading_status` better served by the status schema - Updated `asset` to 11 bytes and `kAssetCstrLen` to match - Statistics schema: - Updated `StatMsg` has an expanded 64-bit `quantity` field. `kUndefStatQuantity` has been updated to match - The previous `StatMsg` has been moved to `v2::StatMsg` or `StatMsgV2` #### Bug fixes - Fixed "Zstd error decompressing: Operation made no progress over multiple calls, due to output buffer being full" error with `TimeseriesGetRange` - Fixed missing implementation of `HistoricalBuilder::SetLogReceiver` ## 0.35.1 - 2025-05-20 #### Bug fixes - Added missing `operator<<` and `ToString` implementations for `Cmbp1Msg` ## 0.35.0 - 2025-05-13 #### Enhancements - Added a `v3::StatMsg` record with an expanded 64-bit `quantity` field - Added `kDbnVersion` constants to each version namespace: `v1`, `v2`, and `v3` - Added `kUndefStatQuantity` constants to each version namespace - Added new off-market publishers for Eurex, and European Energy Exchange (EEX) - Increased live subscription symbol chunking size ## 0.34.2 - 2025-05-06 #### Bug fixes - Fixed potential for unaligned records in live and historical streaming requests ## 0.34.1 - 2025-04-29 #### Enhancements - Added `InstitutionalPrioritization` variant to `MatchingAlgorithm` #### Bug fixes - Improved memory usage of historical streaming requests (`TimeseriesGetRange`) ## 0.34.0 - 2025-04-22 #### Enhancements - Added `SystemCode` and `ErrorCode` enums to indicate types of system and error messages - Converting a `v1::SystemMsg` to a `v2::SystemMsg` now sets to `code` to the heartbeat value - Introduced `kAssetCstrLen` constant containing the size of `asset` field in `InstrumentDefMsg` #### Breaking changes - Increased the size of `asset` field in `v3::InstrumentDefMsg` from 7 to 11. The `v3::InstrumentDefMsg` message size remains 520 bytes. - Set minimum CMake version to 3.24 - Changed type of `code` field in `SystemMsg` to `SystemCode` - Changed type of `code` field in `ErrorMsg` to `ErrorCode` #### Bug fixes - Changed `TriState` to a regular enum to handle unexpected values - Fixed `ccache` support - Changed to explicitly only support being built as a static library ## 0.33.0 - 2025-04-15 #### Enhancements - Added `id` field to `LiveSubscription` requests, which will be used for improved error messages - Removed Windows-only `dirent` dependency #### Breaking changes - Changed `DbnDecoder`, `FileStream`, `IReadable`, `IWritable` to work on `byte`s ## 0.32.1 - 2025-04-07 #### Bug fixes - Fixed CMake error when `zstdTargets.cmake` exists - Reverted `vcpkg` baseline change ## 0.32.0 - 2025-04-02 #### Enhancements - Upgraded default date version to 3.0.3 - Updated `vcpkg` baseline #### Breaking changes - Upgraded default cpp-httplib version to 0.20.0 which requires OpenSSL >= 3.0 ## 0.31.0 - 2025-03-18 #### Enhancements - Added new venues, datasets, and publishers for ICE Futures US, ICE Europe Financial products, Eurex, and European Energy Exchange (EEX). ## 0.30.0 - 2025-02-11 #### Enhancements - Added `Resubscribe()` methods to `LiveBlocking` and `LiveThreaded` to make it easier to resume a live session after losing the connection to the live gateway - Added `Subscriptions()` getter methods to `LiveBlocking` and `LiveThreaded` for getting all active subscriptions - Added `CommoditySpot` `InstrumentClass` variant ## 0.29.0 - 2025-02-04 #### Enhancements - Fixed documentation for using external versions of libraries (credit: ElBellaCiao) #### Breaking changes - Updated the minimum supported C++ standard to C++17 ## 0.28.0 - 2025-01-21 #### Breaking changes - Updated enumerations for unreleased datasets and publishers. #### Enhancements - Added new dataset `EQUS.MINI` and new publishers `EQUS.MINI.EQUS`, `XNYS.TRADES.EQUS` #### Bug fixes - Changed historical metadata methods with `symbols` parameter to use a `POST` request to allow for requesting supported maximum of 2000 symbols ## 0.27.0 - 2025-01-07 #### Breaking changes - Converted the `UserDefinedInstrument` enum class to an enum to safely allow handling invalid data and adding future variants - Updated the value of the `kMaxRecordLen` constant for the changes to `InstrumentDefMsg` in version 3 #### Enhancements - Added `v3` namespace in preparation for future DBN version 3 release. DBN version 2 remains the current and default version - Added `v3::InstrumentDefMsg` record with new fields to support normalizing multi-leg strategy definitions - Removal of statistics-schema related fields `trading_reference_price`, `trading_reference_date`, and `settl_price_type` - Removal of the status-schema related field `md_security_trading_status` ## 0.26.0 - 2024-12-17 #### Breaking changes - Removed deprecated `Packaging` enum and `packaging` field that's no longer supported by the API - Renamed `VersionUpgradePolicy::Upgrade` to `UpgradeToV2` in preparation for a future DBN version 3 #### Enhancements - Created separate namespaces for each DBN version to create a consistent way to refer to record types from a particular DBN version regardless of whether the record changed between versions ## 0.25.0 - 2024-11-12 #### Enhancements - Added new IntelligentCross venues `ASPN`, `ASMT`, and `ASPI` #### Deprecations - Deprecated `Packaging` enum and `packaging` field on `BatchJob`. These will be removed in a future version. All files from a batch job can be downloaded with the `BatchDownload` method on the historical client ## 0.24.0 - 2024-10-22 #### Enhancements - Added new `None` `Action` variant that will be gradually rolled out to historical and live `GLBX.MDP3` data ## 0.23.0 - 2024-09-25 #### Enhancements - Added new `Cmbp1Msg` - Added new consolidated publisher values for `XNAS.BASIC` and `DBEQ.MAX` #### Breaking changes - Changed the layout of `CbboMsg` to better match `BboMsg` - Renamed `Schema::Cbbo` to `Schema::Cmbp1` #### Deprecations - Deprecated `Packing::Tar` and renamed it to `TarDeprecated`. This variant will be removed in a future version when it is no longer supported by the API ## 0.22.0 - 2024-08-27 #### Enhancements - Added `Intraday` variant to `DatasetCondition` in preparation for intraday data being available from the historical API - Added new publisher values for `XCIS.BBOTRADES` and `XNYS.BBOTRADES` #### Breaking changes - Removed previously deprecated `DatasetCondition::Bad` variant ## 0.21.0 - 2024-07-30 #### Enhancements - Added new publisher value for `DBEQ.SUMMARY` #### Breaking changes - Renamed `SType::Nasdaq` variant to `SType::NasdaqSymbol` - Renamed `SType::Cms` variant to `SType::CmsSymbol` #### Bug fixes - Added missing `ToString` and `FromString` branches for `SType::NasdaqSymbol` and `SType::CmsSymbol` - Removed `has_header_v` variable template that broke C++11 compatibility ## 0.20.1 - 2024-07-16 #### Enhancements - Improved installation with `CMake`: license is now installed, transitive dependencies are configured when importing package ## 0.20.0 - 2024-07-09 This release adds support for encoding DBN within the C++ client. It also improves historical symbology support with the new `TsSymbolMap` class that handles mapping historical records to a text symbol. To support this class, several types for date fields were changed from strings or ints to `date::year_month_day`. #### Enhancements - Added `TsSymbolMap` to support historical symbology where mappings change between days - Added `DbnEncoder` class for encoding DBN data - Added blocking API similar to `LiveBlocking` to `DbnFileStore` with new `GetMetadata` and `NextRecord` methods - Added `BboMsg` record struct for future `bbo-1m` and `bbo-1s` schemas - Added `PitSymbol` map constructor from `Metadata` and a `date::year_month_day` - Added `Metadata::CreateSymbolMap` and `Metadata::CreateSymbolMapForDate` methods for creating symbology maps from historical metadata - Added blocking API similar to `LiveBlocking` to `DbnFileStore` - Added `SymbologyResolution::CreateSymbolMap` method for creating a symbology map from a symbology resolution response - Added `InFileStream` and `OutFileStream` helper classes for reading and writing binary output respectively #### Breaking changes - Added new dependency on [Howard Hinnant's date library](https://howardhinnant.github.io/date/date.html) - Added `ILogReceiver*` parameter to all `DbnDecoder` constructors and one `DbnFileStore` constructor - Removed type `StrMappingInterval`. `MappingInterval` is now also used in `SymbologyResolution`. - Changed `Bbo1sMsg` and `Bbo1mMsg` to be aliases for `BboMsg` - Changed type of `start_date` and `end_date` in `MappingInterval` to `date::year_month_day` - Added `stype_in` and `stype_out` fields to `SymbologyResolution` to support creating a `TsSymbolMap` ## 0.19.1 - 2024-06-25 #### Enhancements - Added `Upgrade()` method to `Metadata` to update the metadata fields according to a `VersionUpgradePolicy` - Added new publisher values for `XNAS.BASIC` and `XNAS.NLS` #### Bug fixes - Fixed issue where `Metadata` wasn't upgraded when passing `VersionUpgradePolicy::Upgrade` ## 0.19.0 - 2024-06-04 #### Enhancements - Added configurable `heartbeat_interval` parameter for live clients that determines the timeout before heartbeat `SystemMsg` records will be sent. It can be configured via the `SetHeartbeatInterval` method of the `LiveBuilder` - Added `SetAddress` method to `LiveBuilder` for configuring a custom gateway address without using the constructor directly - Added new `UncrossingPrice` `StatType` variant - Added new publisher values for `XNAS.BASIC` - Added `SetDataset(Dataset)` overload to `LiveBuilder` - Added new off-market publisher values for `IFEU.IMPACT` and `NDEX.IMPACT` #### Breaking changes - Added `heartbeat_interval` parameter to the `Live` constructors - Removed `start_date` and `end_date` fields from `DatasetRange` struct in favor of `start` and `end` - Removed live `Subscribe` method overloads with `use_snapshot` parameter in favor of separate `SubscribeWithSnapshot` method #### Bug fixes - Fixed overloading of live `Subscribe` methods - Fixed live subscribing with default-constructed `UnixNanos` - Fixed descriptions for `FINN` and `FINY` publishers ## 0.18.1 - 2024-05-22 #### Enhancements - Added live `Subscribe` function overload with `use_snapshot` parameter - Added `GetIf` method to `Record` that allows `if` chaining for handling multiple record types - Added record type checking to `Record::Get` method to catch programming errors and prevent reading invalid data - Added `ToString` for `Record` #### Bug fixes - Added missing symbol chunking for live `Subscribe` overloads with `const std::string&` `start` parameter ## 0.18.0 - 2024-05-14 #### Breaking changes - Changed `FlagSet` to be more class-like: - Added predicate methods and setters for each bit flag - Improved string formatting - Removed bitwise operators. Bitwise operations can be performed by first casting to a `std::uint8_t` or calling the `Raw()` method - Changed format of `display_factor` and `price_ratio` to a fixed-precision decimal for `InstrumentDefMsg` and `InstrumentDefMsgV1` to match existing values and DBN crate - Changed format of `unit_of_measure_qty` to a fixed-precision decimal for `InstrumentDefMsgV1` to match `InstrumentDefMsg` ## 0.17.1 - 2024-04-08 #### Enhancements - Added support for Conan-installed zstd (credit: Jakob Lövhall) #### Bug fixes - Added missing copying of `ts_event` when upgrading structs from DBNv1 to DBNv2 - Fixed setting of compiler warnings and warnings that had accumulated ## 0.17.0 - 2024-04-01 #### Enhancements - Added `StatusMsg` record, and `StatusAction`, `StatusReason`, `TradingEvent`, and `TriState` enums - Added `CbboMsg` record and corresponding `ConsolidatedBidAskPair` structure - Added new enum values for `Schema` and `RType` corresponding to new schemas `cbbo`, `cbbo-1s`, `cbbo-1m`, `tcbbo`, `bbo-1s`, `bbo-1m` - Changed format `unit_of_measure_qty` to a fixed-precision decimal - Added logic to skip `find_package` call if `nlohmann_json` and `httplib` targets already exist (credit: Asen Kovachev) - Add specific instructions for installing dependencies on Ubuntu and macOS (credit: Camron Godbout) #### Bug fixes - Fixed out-of-order initialization in `DbnDecoder` (credit: Jakob Lövhall) #### Breaking changes - Renamed publishers from deprecated datasets to their respective sources (`XNAS.NLS` and `XNYS.TRADES` respectively) #### Deprecations - Deprecated dataset values `FINN.NLS` and `FINY.TRADES` ## 0.16.0 - 2024-03-01 #### Enhancements - Added new publisher values for consolidated DBEQ.MAX - Added constructor to `WithTsOut` that updates `length` to the correct value to account for the extra 8 bytes - Upgraded default cpp-httplib version to 0.14.3 (last to still support OpenSSL 1.1) - Upgraded default nlohmann_json version to 3.11.3 #### Breaking changes - Changed default `upgrade_policy` to `Upgrade` so by default the primary record types can always be used - Renamed `dummy` field in `ImbalanceMsg` and `StatMsg` to `reserved` #### Bug fixes - Fixed handling of `ts_out` when decoding DBNv1 and upgrading to version 2 - Fixed missing logic to upgrade `ErrorMsgV1` and `SystemMsgV1` when decoding DBN with `VersionUpgradePolicy::Upgrade` - Added missing `StatType::Vwap` variant used in the ICE datasets - Added missing `ToString` and `operator<<` handling for `StatType::ClosePrice` and `StatType::NetChange` - Fixed potential for invalid reads when decoding C strings in `DbnDecoder` ## 0.15.0 - 2024-01-16 #### Breaking changes - Increased size of `SystemMsg` and `ErrorMsg` to provide better messages from Live gateway - Increased length of `err` and `msg` fields for more detailed messages - Added `is_last` field to `ErrorMsg` to indicate the last error in a chain - Added `code` field to `SystemMsg` and `ErrorMsg`, although currently unused - Added new `is_last` parameter to `ErrorMsg::new` - Decoding these is backwards-compatible and records with longer messages won't be sent during the DBN version 2 migration period - Renamed previous records to `ErrorMsgV1` and `SystemMsgV1` ## 0.14.1 - 2023-12-18 #### Enhancements - Added `PitSymbolMap` helper for keeping track of symbology mappings in Live - Added new publisher value for OPRA MIAX Sapphire #### Bug fixes - Fixed misaligned read undefined behavior when decoding records ## 0.14.0 - 2023-11-23 This release adds support for DBN v2. DBN v2 delivers improvements to the `Metadata` header symbology, new `stype_in` and `stype_out` fields for `SymbolMappingMsg`, and extends the symbol field length for `SymbolMappingMsg` and `InstrumentDefMsg`. The entire change notes are available [here](https://github.com/databento/dbn/releases/tag/v0.14.0). Users who wish to convert DBN v1 files to v2 can use the `dbn-cli` tool available in the [databento-dbn](https://github.com/databento/dbn/) crate. On a future date, the Databento live and historical APIs will stop serving DBN v1. This release is fully compatible with both DBN v1 and v2, and so the change should be seamless for most users. #### Enhancements - Added support for DBN encoding version 2 (DBNv2), affecting `SymbolMappingMsg`, `InstrumentDefMsg`, and `Metadata` - Version 1 structs can be converted to version 2 structs with the `ToV2()` method - Added `symbol_cstr_len` field to `Metadata` to indicate the length of fixed symbol strings - Added `stype_in` and `stype_out` fields to `SymbolMappingMsg` to provide more context with live symbology updates - Added `IndexTs` methods to every record type which returns the primary timestamp - Added `VersionUpgradePolicy` enum to allow specifying how to handle decoding records from prior DBN versions - Added `InstrumentDefMsgV2` and `SymbolMappingMsgV2` type aliases - Added `kDbnVersion` constant for current DBN version - Added `kSymbolCstrLen`, `kSymbolCstrLenV1`, and `kSymbolCstrLenV2` constants for the length of fixed-length symbol strings in different DBN versions - Added new publisher values in preparation for IFEU.IMPACT and NDEX.IMPACT datasets - Added new publisher values for consolidated DBEQ.BASIC and DBEQ.PLUS - Added `kMaxRecordLen` constant for the length of the largest record type - Added ability to convert `FlagSet` to underlying representation #### Breaking changes - The old `InstrumentDefMsg` is now `InstrumentDefMsgV1` in `compat.hpp` - The old `SymbolMappingMsg` is now `SymbolMappingMsgV1` in `compat.hpp` - Converted the following enum classes to enums to allow safely adding new variants: `SecurityUpdateAction` and `SType` - Renamed `dummy` to `reserved` in `InstrumentDefMsg` - Removed `reserved2`, `reserved3`, `reserved4`, and `reserved5` from `InstrumentDefMsg` - Moved position of `strike_price` within `InstrumentDefMsg` - Removed deprecated `SecurityUpdateAction::Invalid` variant ## 0.13.1 - 2023-10-23 #### Enhancements - Added new publisher values in preparation for DBEQ.PLUS - Added `ToIso8601` for `UnixNanos` for converting to human-readable ISO8601 datetime string - Added `kUndefTimestamp` and `kUndefStatQuantity` constants - Added flag `kTob` for top-of-book messages ## 0.13.0 - 2023-09-21 #### Enhancements - Added `pretty_px` option for `BatchSubmitJob`, which formats prices to the correct scale using the fixed-precision scalar 1e-9 (available for CSV and JSON text encodings) - Added `pretty_ts` option for `BatchSubmitJob`, which formats timestamps as ISO 8601 strings (available for CSV and JSON text encodings) - Added `map_symbols` option to `BatchSubmitJob`, which appends the raw symbol to every record (available for CSV and JSON text encodings) reducing the need to look at the `symbology.json` file - Added `split_symbols` option for `BatchSubmitJob`, which will split files by raw symbol - Added `encoding` option to `BatchSubmitJob` to allow requesting non-DBN encoded data through the client - Added `map_symbols`, `pretty_px`, and `pretty_ts` to `BatchJob` response - Added `ARCX.PILLAR.ARCX` publisher - Added `ClosePrice` and `NetChange` `StatType`s used in the `OPRA.PILLAR` dataset #### Breaking changes - Remove `default_value` parameter from `Historical::SymbologyResolve` ## 0.12.0 - 2023-08-24 #### Enhancements - Added the `Publisher`, `Venue`, and `Dataset` enums - Added `Publisher` getters to `Record` and `RecordHeader` to convert the `publisher_id` to its enum ## 0.11.0 - 2023-08-10 #### Enhancements - Added `raw_instrument_id` to definition schema - Added `operator==` and `operator!=` implementations for `DatasetConditionDetail` and `DatasetRange` #### Breaking changes - Changed `MetadataListPublishers` to return a `vector` - `MetadataListFields`: - Changed return type to `vector` - Made `encoding` and `schema` parameters required - Removed `dataset` parameter - `MetadataListUnitPrices`: - Changed return type to `vector` - Made `dataset` parameter required - Removed `mode` and `schema` parameters #### Bug fixes - Fixed installation of `nlohmann_json` when using bundled version - Added missing `operator!=` implementations for `Metadata`, `MappingInterval`, and `SymbolMapping` ## 0.10.0 - 2023-07-20 #### Enhancements - Added preliminary support for Windows - Added `LiveThreaded::BlockForStop` to make it easier to wait for one or more records before closing the session - Changed `TimeseriesGetRange` to request a Zstd-compressed result for more efficient data transfer - Switched `BatchSubmitJob` to use form data to avoid query param length limit - Switched `SymbologyResolve` to use POST request with form data to avoid query param length limit #### Breaking changes - Changed size-related fields and `limit` parameters to use `std::uint64_t` for consistency across architectures #### Bug fixes - Remove usage of non-portable `__PRETTY_FUNCTION__` ## 0.9.1 - 2023-07-11 #### Enhancements - Added constants for dataset IDs for Databento Equity Basic and OPRA Pillar - Added `const char*` getters to records for fixed-length `char` arrays - Added `RType` getter to `Record` #### Bug fixes - Added batching for live subscriptions to avoid hitting max message length - Fixed bug in Zstd decompression - Fixed `Historical::BatchDownload` truncating file before writing each chunk ## 0.9.0 - 2023-06-13 #### Enhancements - Added `Reconnect` methods to `LiveBlocking` and `LiveThreaded` - Added optional `exception_callback` argument to `LiveThreaded::Start` to improve error handling options - Added batch download support data files (`condition.json` and `symbology.json`) - Added support for logging warnings from Historical API - Relaxed 10 minute minimum request time range restriction #### Breaking changes - Changed `use_ts_out` default to `false` #### Bug fixes - Fixed missing definition for `operator==` for `ImbalanceMsg` ## 0.8.0 - 2023-05-16 #### Enhancements - Changed `end` and `end_date` to optional to support new forward-fill behaviour #### Breaking changes - Renamed `booklevel` MBP field to `levels` for brevity and consistent naming - Removed `open_interest_qty` and `cleared_volume` fields from definitions schema that were always unset ## 0.7.0 - 2023-04-28 #### Enhancements - Added initial support for live data with `LiveBlocking` and `LiveThreaded` clients - Added support for statistics schema - Added `SystemMsg` and `ErrorMsg` records for use in live data - Added `strike_price`, `strike_price_currency`, and `instrument_class` to `InstrumentDefMsg` - Added `FixedPx` helper class for formatting fixed prices - Added configurable log receiver `ILogReceiver` - Added `instrument_class`, `strike_price`, and `strike_price_currency` to definition schema - Added additional `condition` variants for `DatasetConditionDetail` (degraded, pending, missing) - Added additional member `last_modified_date` to `DatasetConditionDetail` - Added `has_mixed_schema`, `has_mixed_stype_in`, and `ts_out` to `Metadata` to support live data - Added optional `compression` parameter to `BatchSubmitJob` #### Breaking changes - Removed `related` and `related_security_id` from `InstrumentDefMsg` - Renamed `BatchJob.cost` to `cost_usd` and value now expressed as US dollars - Renamed `SType::ProductId` to `SType::InstrumentId` and `SType::Native` to `SType::RawSymbol` - Renamed `RecordHeader::product_id` to `instrument_id` - Renamed `InstrumentDefMsg::symbol` to `raw_symbol` - Renamed `SymbolMapping::native_symbol` to `raw_symbol` - Changed `expiration` and `action` type to `UnixNanos` - Changed some fields to enums in `InstrumentDefMsg` #### Deprecations - Deprecated `SType::Smart` to split into `SType::Parent` and `SType::Continuous` #### Bug fixes - Fixed parsing of `BatchSubmitJob` response - Fixed invalid read in `DbnDecoder` - Fixed memory leak in `TryCreateDir` ## 0.6.1 - 2023-03-28 #### Breaking changes - Removed usage of unreliable `std::ifstream::readsome` #### Bug fixes - Fixed Zstd decoding of files with multiple frames ## 0.6.0 - 2023-03-24 #### Enhancements - Added support for imbalance schema - Added support for decoding `ts_out` field - Added flags `kSnapshot` and `kMaybeBadBook` #### Breaking changes - Removed `record_count` from `Metadata` - Changed `Historical::BatchDownload` to return the paths of the downloaded files ## 0.5.0 - 2023-03-13 #### Enhancements - Added `Historical::MetadataGetDatasetRange` #### Breaking changes - Changed `MetadataGetDatasetCondition` to return `vector` - Removed `MetadataListCompressions` (redundant with docs) - Removed `MetadataListEncodings` (redundant with docs) - Removed optional `start` and `end` params from `MetadataListSchemas` (redundant) - Renamed `FileBento` to `DbnFileStore` ## 0.4.0 - 2023-03-02 #### Enhancements - Added live gateway resolution - Added `SymbolMappingMsg` and `ErrorMsg` records - Added `Action` and `Side` enums - Added `available_start_date` and `available_end_date` to `DatasetConditionInfo` - Made `start_date` and `end_date` optional for `Historical::MetadataGetDatasetCondition` - Improved API for `flags` record fields - Added `PKGBUILD` to demonstrate installation - Disabled unit testing by default #### Breaking changes - Removed `is_full_universe` and `is_example` fields from `BatchJob` - Refactored rtypes - Introduced separate rtypes for each OHLCV schema - Renamed DBZ to DBN - Renamed `DbzParser` to `DbnDecoder` - Renamed `TimeseriesStream` to `TimeseriesGetRange` - Changed `kAllSymbols` representation #### Bug fixes - Fixed usage of as a system library ## 0.3.0 - 2023-01-06 #### Enhancements - Added support for definition schema - Added option for CMake to download gtest - Updated `Flag` enum #### Breaking changes - Standardized getter method names to pascal case - Renamed `is_full_book` to `is_full_universe` - Renamed `TickMsg` to `MboMsg` - Changed `flags` fields to unsigned #### Bug fixes - Fixed cancellation in `Historical::TimeseriesStream` - Fixed race condition in `Historical::TimeseriesStream` exception handling - Fixed gtest linker error on macOS ## 0.2.0 - 2022-12-01 #### Enhancements - Added `Historical::MetadataGetDatasetCondition` - Improved Zstd CMake integration #### Bug fixes - Fixed requesting all symbols for a dataset ## 0.1.0 - 2022-11-07 - Initial release with support for historical data --- Source: https://databento.com/docs/release-notes/release-notes-python # Python ## 0.88.0 - 2026-10-21 #### Enhancements - `zstandard` is now only a dependency on Python versions below 3.14; on Python 3.14 and later the standard library `compression.zstd` module is used instead - Added a direct dependency on `certifi` 2026.10.19 or later so that HTTPS connections no longer trust the compromised eMudhra intermediate CAs (emSign ECC Root CA - G3) #### Bug fixes - Fixed the return type annotation of `metadata.get_dataset_range()` to include the per-schema ranges under the `schema` key ## 0.87.0 - 2026-09-22 #### Enhancements - Added new venue, dataset, and publishers for Databento Core Indices - Asynchronous iteration of `Live` no longer waits for records on a thread pool executor, reducing per-record overhead - Upgraded `databento-dbn` to 0.70.0 #### Bug fixes - Fixed an issue where cancelling asynchronous iteration of `Live` could drop a record ## 0.86.0 - 2026-09-01 #### Enhancements - Added `UNSUBSCRIBE_ACK` to the `SystemCode` enum for acknowledging unsubscribe requests - Upgraded `databento-dbn` to 0.69.0 - Fixed incorrect count in DBN metadata causing decoding to read into the start of records #### Bug fixes - Fixed an issue where `InstrumentMap.clear()` didn't clear the LRU cache for `InstrumentMap.resolve()` (credit: @Tomperez98) ## 0.85.0 - 2026-08-25 #### Enhancements - Added support for the `short` parameter of `batch.list_jobs`. When set to `True`, the response will contain only the `id`, `state`, and `ts_received` fields for each job - Upgraded `databento-dbn` to 0.68.0 ## 0.84.0 - 2026-08-18 #### Enhancements - Added support for `listing_id`, `issuer_id`, and `security_id` as `stype_in` values for the security master and corporate actions reference data endpoints, and `security_id` for the adjustment factors endpoint - Added a `dataset` parameter to `MetadataHttpAPI.list_fields()`. If not provided, the returned fields are for the latest DBN encoding version, which may not match a specific dataset's schema - Upgraded `databento-dbn` to 0.67.0: - Improved performance of record field accessors that convert a raw value to an enum, such as `side()`, `action()` and `rtype()`, by no longer copying the description when constructing the conversion error ## 0.83.0 - 2026-08-04 #### Enhancements - Added `CorporateActionsHttpAPI.list_events()` and `list_enums()` for fetching documentation on supported corporate action event types and enum values - Added `REPLAY_DATA_AGED_OUT` variant to `ErrorCode` enum - Added new publisher values for JPX - Added an `allocate_isins` parameter to `AdjustmentFactorsHttpAPI.get_range()`, `CorporateActionsHttpAPI.get_range()`, `SecurityMasterHttpAPI.get_range()`, and `SecurityMasterHttpAPI.get_last()` to control whether new ISINs are allocated for plans that are ISIN-limited - Upgraded `databento-dbn` to 0.65.0: - Added `finish()` method to the Python Transcoder and a context manager to call this method on exit. This will ensure compressed output contains the end-of-frame block #### Bug fixes - Fixed an issue where the `Live` client's monitor task would not get recreated for the new session after a reconnection ## 0.82.0 - 2026-07-21 #### Enhancements - Added a `loop` parameter to the `Live` client to provide user level control of which event loop the client's connection uses - The `Live` class will no longer create a background thread and event loop upon import - Upgraded `databento-dbn` to 0.63.0: - Added an integer representation for `VersionUpgradePolicy`: the enum is now has a value property, a from_int, and comparison against integers ## 0.81.0 - 2026-07-07 #### Enhancements - Upgraded `databento-dbn` to 0.62.0: - Added `StatType` variants `MwcbLevel1`, `MwcbLevel2`, and `MwcbLevel3` - Added `StatType` variants `AuctionCollarReferencePrice`, `AuctionCollarUpperPrice`, and `AuctionCollarLowerPrice` - Changed the default `size` field to 0 in `MboMsg`, `TradeMsg`, `Mbp1Msg`, `Mbp10Msg`, `BboMsg`, `Cmbp1Msg`, and `CbboMsg` ## 0.80.0 - 2026-06-16 #### Enhancements - Upgraded `databento-dbn` to 0.61.0: - Added `MatchAlgorithm` variant `Allocation` - Added new publisher values for Cboe Titanium Cboe Global Indices Feed #### Bug fixes - Fixed an issue where the `Live` client would throw a `TypeError` when connecting using `uvloop` #### Breaking changes - Renamed the following Venue, Dataset, and Publisher: - `CGIF` to `MAIN` - `CGIF.TITANIUM` to `MAIN.CGIF` - `CGIF.TITANIUM.CGIF` to `MAIN.CGIF.MAIN` ## 0.79.0 - 2026-06-02 #### Enhancements - Added new venues, datasets, and publishers for US Equities Securities Information Processors - Changed the `batch.download` method to use the API's `batch.download` endpoint for downloading a batch job as a ZIP archive - Added method for `batch.get_job_details` to access the full details of a batch job - Upgraded `databento-dbn` to 0.59.0: - Improvements to `ts_out` decoding and serialization #### Bug fixes - Fixed some thread-unsafe behavior in `Live.start()` and `Live.terminate()` which would call methods from the client's event loop in the main thread ## 0.78.0 - 2026-05-12 #### Enhancements - Added time-based backpressure to the live client: pauses reading records from the live gateway when the internal queue spans more than 1 second of data by `ts_index` (`ts_recv` when present, otherwise `ts_event`) - Upgraded `databento-dbn` to 0.58.0: - Added flat per-level properties (`bid_px_00`, `pretty_ask_px_03`, etc.) to the Python `MBP1Msg`, `MBP10Msg`, `BBOMsg`, `CMBP1Msg`, and `CBBOMsg` record classes for parity with the DataFrame layout - Added `INDEX` `InstrumentClass` variant - Added `StatType` variants `INDICATIVE_CLOSE_PRICE` and `VENUE_SPECIFIC_PRICE_1` ## 0.77.0 - 2026-04-28 #### Enhancements - Added new publisher values for Cboe Titanium Cboe Global Indices Feed - Added `YEAR` to `SplitDuration` enum to support yearly historical batch job submission - Upgraded `databento-dbn` to 0.56.0: - Improved Python `DBNDecoder.decode()` performance with pre-allocated output buffers - Added `DBNDecoder.write_and_decode()` to combine write and decode in a single call ## 0.76.0 - 2026-04-21 #### Enhancements - Added new publisher values for OPRA MEMX MX2 Options and IEX Options - Upgraded `databento-dbn` to 0.55.0: - Performance improvements to `DBNDecoder` ## 0.75.0 - 2026-04-07 #### Enhancements - Upgraded `databento-dbn` to 0.53.0: - Made `ts_out` a permanent field on all Python record types, replacing the dynamic `__dict__` attribute. `ts_out` returns an `int` (`UNDEF_TIMESTAMP` when not set) - Removed `__dict__` from all Python record classes, eliminating a separate per-instance allocation ## 0.74.1 - 2026-03-31 #### Enhancements - Upgraded `databento-dbn` to 0.52.1: - Added `__index__` to all int-representable and char-backed Python enums, enabling use with `int()`, `hex()`, and as sequence indices - Fixed memory leak in Python bindings where every record object leaked ~64 bytes due to a `pyo3` 0.28 regression in `#[pyclass(dict)]` deallocation. Downgraded `pyo3` to 0.27.2 - Removed unnecessary `dict` from `BidAskPair` and `ConsolidatedBidAskPair` Python classes - Fixed Python type stubs to reflect that `record_size()` is a method, not a property - Fixed Python type stubs for record fields to indicate which fields are writable (e.g. `publisher_id`, `instrument_id`, `price`, `size`) and added `@setter` stubs for enum fields (e.g. `action`, `side`) ## 0.74.0 - 2026-03-24 #### Enhancements - Changed `SlowReaderBehavior.SKIP` to send "skip" instead of "drop" to the gateway - Upgraded `databento-dbn` to 0.52.0: - Added `SYMBOL_CSTR_LEN` constant and versioned variants (`SYMBOL_CSTR_LEN_V1`, `SYMBOL_CSTR_LEN_V2`, `SYMBOL_CSTR_LEN_V3`) to Python, including in each versioned module (`v1`, `v2`, `v3`) as `SYMBOL_CSTR_LEN` - Added `v1`, `v2`, and `v3` submodule imports to `databento_dbn.__init__` so they are accessible as attributes (e.g. `databento_dbn.v1`) ## 0.73.0 - 2026-03-10 #### Enhancements - Upgraded `databento-dbn` to 0.51.0: - Added logic to set `code` when upgrading version 1 `ErrorMsg` to newer versions ## 0.72.0 - 2026-02-26 #### Enhancements - Upgraded `databento-dbn` to 0.50.0: - Added `SkippedRecordsAfterSlowReading` to the `ErrorCode` enum for gateway errors originating from slow client catch-up. ## 0.71.0 - 2026-02-17 #### Enhancements - Added `slow_reader_behavior` field to `AuthenticationRequest` message - Added `SlowReaderBehavior` enum - Added support for using compression in the live API: - Added `compression` parameter to the `Live` client constructor - Added `compression` property to the `Live` client - Added `compression` field to `AuthenticationRequest` - Upgraded `databento-dbn` to 0.49.0: - Added support for decompressing Zstd in the Python `DBNDecoder` and new optional `compression` parameter ## 0.70.0 - 2026-01-27 #### Enhancements - Added new off-market publisher for Cboe Futures Exchange (`XCBF_PITCH_XOFF`) - Upgraded `databento-dbn` to 0.48.0: - Changed the `__repr__` implementation of all records in Python to be more Pythonic - Added new `StatType` variants to be used by `XCBF.PITCH` dataset: - `UPPER_PRICE_LIMIT` - `LOWER_PRICE_LIMIT` - `BLOCK_VOLUME` - `VENUE_SPECIFIC_VOLUME_1` - Added new publisher for Blue Ocean ATS (`OCEA_MEMOIR_OCEA`) - Reduced the log level of end-of-interval `SystemMsg` records to the debug level ## 0.69.0 - 2026-01-13 #### Enhancements - Upgraded `databento-dbn` to 0.46.0: - Added `DBNRecord` union type to Python which includes all record types - Removed `Record` class from Python type stubs to match code: the record classes don't share a base class. Use `DBNRecord` instead. #### Breaking changes - Removed `DBNRecord` union type from `databento.common.types`, use `databento_dbn.DBNRecord` instead ## 0.68.2 - 2026-01-06 #### Bug fixes - Fix use of `Self` in `GatewayControl.parse()` ## 0.68.1 - 2025-12-16 #### Enhancements - Improved the performance of `DBNStore.to_df()` symbol mapping ## 0.68.0 - 2025-12-09 This release adds support for Python 3.14. #### Enhancements - Added support for Python 3.14 - Functions which accept a path as an argument now expand user directories - Upgraded `databento-dbn` to 0.45.0: - Added support for Python 3.14 ## 0.67.0 - 2025-12-02 #### Enhancements - Added a property `Live.subscription_requests` which returns a list of tuples containing every `SubscriptionRequest` for the live session - Changed the return value of `Live.subscribe()` to `int`, the value of the subscription ID, which can be used to index into the `Live.subscription_requests` property - Added feature to automatically monitor for hung connections in the `Live` client - Hung connections will be disconnected client side with a `BentoError` - Added new venue, dataset, and publisher for Cboe Futures Exchange (`XCBF.PITCH`) #### Breaking changes - Several log messages have been reformatted to improve clarity and reduce redundancy, especially at debug levels - The `map_symbols` parameter for `Historical.batch.submit_job()` now defaults to `True` for JSON and CSV encodings ## 0.66.0 - 2025-11-18 #### Enhancements - Added a property `Live.session_id` which returns the streaming session ID when the client is connected - Streams added with `Live.add_stream()` which do not define an exception handler will now emit a warning if an exception is raised while executing the callback - Callback functions added with `Live.add_callback()` which do not define an exception handler will now emit a warning if an exception is raised while executing the callback - Upgraded `databento-dbn` to 0.44.0: - Added logic to set `code` when upgrading version 1 `SystemMsg` to newer versions #### Bug fixes - Streams opened by `Live.add_stream()` will now close properly when the streaming session is closed ## 0.65.0 - 2025-11-11 #### Deprecations - Deprecated `mode` parameter in `metadata.get_cost`, which will be removed in a future release #### Enhancements - Added export of `CBBOMsg` and `BBOMsg` from `databento_dbn` to the root `databento` package - Upgraded `databento-dbn` to 0.43.0: - Added export of `F_PUBLISHER_SPECIFIC` constant to Python - Added explicit `Unset` variant for `SystemCode` and `ErrorCode` - Changed Python getters for enum fields to return the underlying type when no known variant can be found. As a result, these getters no longer raise an exception #### Breaking changes - Removed support for Python 3.9 due to end of life ## 0.64.0 - 2025-09-30 #### Enhancements - Upgraded `databento-dbn` to 0.42.0: - Added `ts_index` and `pretty_ts_index` properties for records in Python which provides the timestamp that is most appropriate for indexing - Fixed type stub for `channel_id` to allow None #### Enhancements - Reduced the log level of `SystemMsg` records in the `Live` client to debug - Increased the log level of `SystemMsg` records with the code `SystemCode.SLOW_READER_WARNING` to warning #### Bug fixes - Fixed type hint for `start` parameter in `Live.subscribe()` ## 0.63.0 - 2025-09-02 #### Enhancements - Upgraded `databento-dbn` to 0.41.0 #### Bug fixes - Fixed an issue where calling `Live.stop()` would not clean up the client state once the socket is closed ## 0.62.0 - 2025-08-19 This release delivers a number of breaking changes to the Python interface for DBN records to provide a cleaner and more consistent API. #### Breaking changes - Removed `bill_id` from the response of `batch.list_jobs()` and `batch.submit_job()` - Upgraded `databento-dbn` to 0.40.0: - Removed `hd` property from records in Python. Header fields are accessible directly from the record - Removed ability to directly instantiate most enums from an `int` in Python and coercion from `int` in `__eq__`. They can still be instantitated with the `from_int` class method. Write `Side.from_int(66)` instead of `Side(66)` and `Side.BID == Side.from_int(66)` instead of `Side.BID == 66`. Affected enums: - `Side` - `Action` - `InstrumentClass` - `MatchAlgorithm` - `UserDefinedInstrument` - `SecurityUpdateAction` - `SType` - `Schema` - `Encoding` - `Compression` - `TriState` - Removed string coercion in `__init__` and `__eq__` for `RType`, `SystemCode`, and `ErrorCode` enums in Python. It can still be instantiated from a `str` with the `from_str` class method. Write `RType.from_str("mbo")` instead of `RType("mbo")` and `RType.TRADES == RType.from_str("trades")` instead of `RType.TRADES == "trades"` #### Enhancements - Added `END_OF_INTERVAL` variant to `SystemCode` enum ## 0.61.0 - 2025-08-12 #### Breaking changes - Modified the `states` parameter in `batch.list_jobs()` #### Enhancements - Added `JobState` enum - Added export of `SystemCode` and `ErrorCode` from `databento_dbn` to the root `databento` package - Added `F_PUBLISHER_SPECIFIC` flag to `RecordFlags` enum #### Bug fixes - Bumped the minimum version requirement for `requests` to 0.27.0 ## 0.60.0 - 2025-08-05 #### Enhancements - Added `parquet_schema` option to `DBNStore.to_parquet()` for overriding the pyarrow schema - Upgraded `databento-dbn` to 0.39.0: - Added `side()` and `unpaired_side()` methods to `ImbalanceMsg` that convert the fields of the same name to the `Side` enum - Added `pretty_auction_time` property in Python for `ImbalanceMsg` - Added `action` and `ts_in_delta` getters to `BboMsg` - Added `ts_recv` getter to `StatusMsg` - Added missing floating-point price getters to `InstrumentDefMsg` record types from all DBN versions - Added more floating-point price getters to `ImbalanceMsg` - Added floating-point price getter to `StatMsg` - Standardize Python `__init__` type signatures - Changed `auction_time` field in `ImbalanceMsg` to be formatted as a timestamp - Fixed a regression where some enum constructors no longer raised a `DBNError` in Python #### Bug fixes - Removed unused `S3` and `Disk` variants from `Delivery` enum ## 0.59.0 - 2025-07-15 #### Enhancements - Upgraded `databento-dbn` to 0.37.1: - Fix buffer growth in `DbnFsm::write_all()`, which is used by `DBNDecoder.write()` #### Breaking changes - Renamed the following Venue, Dataset, and Publishers: - `XEER` to `XEEE` - `XEER.EOBI` to `XEEE.EOBI` - `XEER.EOBI.XEER` to `XEEE.EOBI.XEEE` - `XEER.EOBI.XOFF` to `XEEE.EOBI.XOFF` ## 0.58.0 - 2025-07-08 #### Enhancements - Changed the `tz` parameter in `DBNStore.to_df()` to accept `datetime.tzinfo` instead of `pytz.BaseTzInfo` explicitly - Modified the dependency specification for `databento_dbn` to allow for compatible patch versions - Upgraded `databento-dbn` to 0.36.2: - Fixed change in behavior where Python `DBNDecoder.decode()` wouldn't always decode all available data on the first call ## 0.57.1 - 2025-06-17 #### Enhancements - Changed the following Venue, Publisher, and Dataset descriptions: - "ICE Futures Europe (Financials)" renamed to "ICE Europe Financials" - "ICE Futures Europe (Commodities)" renamed to "ICE Europe Commodities" - Upgraded `databento-dbn` to 0.36.1: - Fixed setting of ts_out property of DbnFsm based on decoded metadata. This was preventing ts_out from being correctly decoded in the Python DBNDecoder - Fixed decoding of `ts_out` with first records in DBNDecoder #### Bug fixes - Fixed an issue where DBN records from the Live client where not having their `ts_out` populated ## 0.57.0 - 2025-06-10 #### Enhancements - Upgraded `databento-dbn` to 0.36.0: - Added missing Python type stubs for several leg properties of `InstrumentDefMsg` #### Bug fixes - Fixed an issue where the zstandard frame size could limit the size of `DataFrame` objects returned by `DBNStore.to_df()` when a `count` was specified #### Deprecations - Deprecated `int` and `pd.Timestamp` types for `start_date` and `end_date` parameters which will be removed in a future release ## 0.56.0 - 2025-06-03 #### Breaking changes - Updated the names of several subfields in the `Reference.corporate_actions.get_range(...)` response, under the `date_info`, `event_info`, and `rate_info` fields. The following subfields were renamed: - `decl_currency` renamed to `declared_currency` - `decl_gross_amount` renamed to `declared_gross_amount` - `f_x_rate` renamed to `fx_rate` - `iss_new_name` renamed to `issuer_new_name` - `iss_old_name` renamed to `issuer_old_name` - `new_bbg_company_id` renamed to `new_bbg_comp_id` - `new_bbg_company_tk` renamed to `new_bbg_comp_ticker` - `new_bbg_exh_id` renamed to `new_figi` - `new_bbg_exh_tk` renamed to `new_figi_ticker` - `new_min_tra_qty` renamed to `new_min_trading_qty` - `new_mktsg_id` renamed to `new_market_segment_id` - `new_reg_s144_a` renamed to `new_reg_s144a` - `new_unit_sec_id` renamed to `new_unit_security_id` - `offeree_iss_id` renamed to `offeree_issuer_id` - `offeror_iss_id` renamed to `offeror_issuer_id` - `old_bbg_company_id` renamed to `old_bbg_comp_id` - `old_bbg_company_tk` renamed to `old_bbg_comp_ticker` - `old_bbg_exh_id` renamed to `old_figi` - `old_bbg_exh_tk` renamed to `old_figi_ticker` - `old_min_tra_qty` renamed to `old_min_trading_qty` - `old_mktsg_id` renamed to `old_market_segment_id` - `old_reg_s144_a` renamed to `old_reg_s144a` - `old_unit_sec_id` renamed to `old_unit_security_id` - `pp_sec_id` renamed to `pp_security_id` - `poolfactor` renamed to `pool_factor` - `pre_offer_q_ty` renamed to `pre_offer_qty` - `qual_st_cap_gains` renamed to `qual_short_term_cap_gains` - `redem_percentage` renamed to `redemption_percentage` - `st_cap_gains` renamed to `short_term_cap_gains` - `sec_new_name` renamed to `security_new_name` - `sec_old_name` renamed to `security_old_name` - `section199_a_foreign_tax_paid` renamed to `section199a_foreign_tax_paid` - `section199_a_inc_div` renamed to `section199a_inc_div` - `section199_a_st_cap_gain` renamed to `section199a_short_term_cap_gain` - `tra_isin` renamed to `trading_isin` - `tra_sec_id` renamed to `trading_security_id` - `us_deposit_receiptateto_currency` renamed to `usd_rate_to_currency` - `cashbak` renamed to `cash_back` - `companyulsory_acq_date` renamed to `compulsory_acq_date` - `frankdiv` renamed to `franked_div` - `lead_plntiff_deadline_date` renamed to `lead_plaintiff_deadline_date` - `maxprice` renamed to `max_price` - `minprice` renamed to `min_price` - `redem_premium` renamed to `redemption_premium` - `redem_price` renamed to `redemption_price` - `unit_frankdiv` renamed to `unfranked_div` #### Enhancements - Upgraded `databento-dbn` to 0.35.1 ## 0.55.1 - 2025-06-02 #### Bug fixes - Fixed decoding of DBN versions 1 and 2 statistics in `DBNStore.to_df()` ## 0.55.0 - 2025-05-29 #### Enhancements - Added `exchanges` parameter to `Reference.corporate_actions.get_range(...)` - Added `is_last` field to live subscription requests which will be used to improve the handling of split subscription requests - Upgraded `databento-dbn` to 0.35.0: - This version delivers DBN version 3 (DBNv3), which is the new default - Improved the performance of the Python `DBNDecoder` #### Bug fixes - Fixed an issue where `JSONDecodeError` would not be caught when using `simplejson` with `requests` (credit: @xuanqing94) ## 0.54.0 - 2025-05-13 #### Enhancements - Added new off-market publishers for Eurex, and European Energy Exchange (EEX) - Increased live subscription symbol chunking size - Upgraded `databento-dbn` to 0.34.0 ## 0.53.0 - 2025-04-29 #### Enhancements - Upgraded `databento-dbn` to 0.33.1: - Added `SystemCode` and `ErrorCode` enums to indicate types of system and error messages - Added `code()` methods to SystemMsg and ErrorMsg to retrieve the enum value if one exists and equivalent properties in Python #### Bug fixes - Fixed issue where all `SystemMsg` records were logged as gateway heartbeats ## 0.52.0 - 2025-04-15 #### Enhancements - Added new optional `id` field to `SubscriptionRequest` class which will be used for improved error messages - Upgraded `databento-dbn` to 0.32.0: - Fixed `RType` variant names in Python to match `Schema` - Added missing Python type declarations for `RType` variants - Fixed issue with Python `_hidden_fields` definition that caused `KeyError: _reserved1_00` with `CMBP1Msg` and other records with `ConsolidatedBidAskPair` ## 0.51.0 - 2025-04-08 #### Enhancements - Upgraded `databento-dbn` to 0.31.0: - Fixed Python type annotation for `SystemMsg.is_heartbeat()` method that was previously annotated as a property ## 0.50.0 - 2025-03-18 #### Enhancements - Added new venues, datasets, and publishers for ICE Futures US, ICE Europe Financials, Eurex, and European Energy Exchange (EEX) - Added export of the following enums from `databento_dbn` to the root `databento` package: - `Action` - `InstrumentClass` - `MatchAlgorithm` - `RType` - `SecurityUpdateAction` - `Side` - `StatUpdateAction` - `TriState` - `UserDefinedInstrument` - `VersionUpgradePolicy` - Added export of the following constants from `databento_dbn` to the root `databento` package: - `DBN_VERSION` - `FIXED_PRICE_SCALE` - `UNDEF_ORDER_SIZE` - `UNDEF_PRICE` - `UNDEF_STAT_QUANTITY` - `UNDEF_TIMESTAMP` - Added export of `BidAskPair` and `ConsolidatedBidAskPair` from `databento_dbn` to the root `databento` package - Upgraded `databento-dbn` to 0.29.0: - Added `COMMODITY_SPOT` `InstrumentClass` variant - Improved handling of `datetime` and `date` objects in `start` and `end` parameters ## 0.49.0 - 2025-03-04 #### Enhancements - Added new venues, datasets, and publishers for ICE Futures US and for ICE Futures Europe (Financial products) - Added a `keep_zip` parameter to `Historical.batch.download()`. When `True`, and downloading all files, the jobs contents will be saved as a ZIP file - Calling `Live.terminate()` will now attempt to write EOF before aborting the connection to help close the remote end ## 0.48.0 - 2025-01-21 #### Breaking changes - Updated enumerations for unreleased datasets and publishers. #### Enhancements - Added export of `StatusAction` enum from `databento_dbn` to the root `databento` package - Added export of `StatusReason` enum from `databento_dbn` to the root `databento` package - Added export of `TradingEvent` enum from `databento_dbn` to the root `databento` package - Added new dataset `EQUS.MINI` and new publishers `EQUS.MINI.EQUS`, `XNYS.TRADES.EQUS` - Removed upper bound for supported `python` versions; the constraint is now `^3.9` - Upgraded `databento-dbn` to 0.27.0: - Fixed export of `InstrumentDefMsgV3` to Python #### Bug fixes - Fixed an issue where sending a `KeyboardInterrupt` during iteration of the `Live` client could block execution waiting for the connection to close - Fixed an issue with submitting historical metadata requests for a large number of symbols. ## 0.47.0 - 2024-12-17 #### Enhancements - Upgraded `databento-dbn` to 0.25.0: - Added type aliases for `TBBOMsg`, `BBO1SMsg`, `BBO1MMsg`, `TCBBOMsg`, `CBBO1SMsg`, `CBBO1MMsg` in Python - Removed exports for `CBBOMsg` and `BBOMsg` in the root `databento` package in favor of aliased versions from `databento-dbn` ## 0.46.0 - 2024-12-10 #### Enhancements - Removed deprecated `packaging` parameter from `Historical.batch.submit_job`. Job files can be downloaded individually or as zip files after the job completes - Upgraded `databento-dbn` to 0.24.0: - Added handling for `UNDEF_TIMESTAMP` in `pretty_` timestamp getters for Python. They now return `None` in the case of `UNDEF_TIMESTAMP` ## 0.45.0 - 2024-11-12 This release adds support for Python 3.13. #### Enhancements - Added support for Python 3.13 - Added new IntelligentCross venues `ASPN`, `ASMT`, and `ASPI` - Upgraded `databento-dbn` to 0.23.1: - Fixed `pretty_activation` getter in `databento_dbn` returning `expiration` instead - Fixed some `pretty_` getters in `databento_dbn` didn't correctly handle `UNDEF_PRICE` #### Deprecations - Deprecated `packaging` parameter for `Historical.batch.submit_job` which will be removed in a future release ## 0.44.1 - 2024-10-29 #### Enhancements - Improved exception messages emitted by the `Live` client to always include contents of any `ErrorMsg` sent by the gateway #### Bug fixes - Fixed an issue where calling `Live.stop` would not close the connection within a reasonable time ## 0.44.0 - 2024-10-22 #### Enhancements - Removed deprecated `databento.from_dbn`; `databento.read_dbn` can be used instead - Upgraded `databento-dbn` to 0.23.0 #### Bug fixes - Fixed an issue where `DBNStore.request_symbology` could request the wrong end date ## 0.43.1 - 2024-10-15 #### Enhancements - Keyword arguments to `DBNStore.to_parquet` will now allow `where` and `schema` to be specified - Improved record processing time for the `Live` client #### Bug fixes - Fixed an issue where validating the checksum of a batch file loaded the entire file into memory ## 0.43.0 - 2024-10-09 This release drops support for Python 3.8 which has reached end-of-life. #### Enhancements - Added `PriceType` enum for validation of `price_type` parameter in `DBNStore.to_df` - Upgraded `databento-dbn` to 0.22.1 #### Bug fixes - Fixed return type hint for `metadata.get_dataset_condition` #### Breaking changes - Removed support for Python 3.8 due to end of life ## 0.42.0 - 2024-09-23 #### Enhancements - Added `mode` parameter to `DBNStore.to_csv` to control the file writing mode - Added `mode` parameter to `DBNStore.to_json` to control the file writing mode - Added `mode` parameter to `DBNStore.to_parquet` to control the file writing mode - Added `compression` parameter to `DBNStore.to_file` which controls the output compression format - Added new consolidated publisher values for `XNAS.BASIC` and `DBEQ.MAX` - Changed `DBNStore` to be more tolerant of truncated DBN streams #### Breaking changes - Changed default write mode for `DBNStore.to_csv` to overwrite ("w") - Changed default write mode for `DBNStore.to_json` to overwrite ("w") - Changed default write mode for `DBNStore.to_parquet` to overwrite ("w") ## 0.41.0 - 2024-09-03 #### Enhancements - Added `databento.read_dbn` alias - Added `mode` parameter to `DBNStore.to_file` to control the file writing mode #### Breaking changes - Changed default write mode for `DBNStore.to_file` to overwrite ("w") #### Deprecations - Deprecated `databento.from_dbn` and will be removed in a future release, use `databento.read_dbn` instead ## 0.40.0 - 2024-08-27 #### Enhancements - Added `adjustment_factors.get_range(...)` method for `Reference` client - Added `security_master.get_range(...)` method for `Reference` client - Added `security_master.get_last(...)` method for `Reference` client - Upgraded `databento-dbn` to 0.20.1 ## 0.39.3 - 2024-08-20 #### Enhancements - Added new publisher values for `XCIS.BBOTRADES` and `XNYS.BBOTRADES` #### Bug fixes - Fixed an issue receiving multiple DBN v1 `ErrorMsg` in the `Live` client would cause an `InvalidState` error - Fixed an issue where creating `Live` clients in multiple threads could cause a `RuntimeError` upon initialization ## 0.39.2 - 2024-08-13 #### Enhancements - Changed `corporate_actions.get_range(...)` to stream compressed zstd data ## 0.39.1 - 2024-08-13 #### Enhancements - Added new publisher values for `XCIS.BBOTRADES` and `XNYS.BBOTRADES` #### Bug fixes - Fixed an issue where a symbol list which contained a `None` would produce a convoluted exception ## 0.39.0 - 2024-07-30 #### Enhancements - Added new publisher value for `DBEQ.SUMMARY` - Upgraded `databento-dbn` to 0.20.0 ## 0.38.0 - 2024-07-23 This release adds a new feature to the `Live` client for automatically reconnecting when an unexpected disconnection occurs. #### Enhancements - Added `Reference` data client with `corporate_actions.get_range(...)` method - Added `ReconnectPolicy` enumeration - Added `reconnect_policy` parameter to the `Live` client to specify client reconnection behavior - Added `Live.add_reconnect_callback` method for specifying a callback to handle client reconnections - Added platform information to the user agent reported by the `Historical` and `Live` clients - Upgraded `databento-dbn` to 0.19.1 - Added `BBOMsg`, `CBBOMsg`, and `StatusMsg` exports to the root `databento` package #### Breaking changes - Calling `Live.stop` will now clear all user streams and callbacks - Renamed `Session` to `LiveSession` in the `databento.live.session` module ## 0.37.0 - 2024-07-09 #### Enhancements - A disconnected `Live` client can now be reused with a different dataset - Upgraded `databento-dbn` to 0.19.0 ## 0.36.3 - 2024-07-02 #### Enhancements - Added export of `StatType` enum from `databento_dbn` to the root `databento` package ## 0.36.2 - 2024-06-25 #### Enhancements - Upgraded `databento-dbn` to 0.18.2 ## 0.36.1 - 2024-06-18 #### Enhancements - Added type alias `TBBOMsg` for `MBP1Msg` - Added support for `bbo-1s`, `bbo-1m`, and `status` schemas - Instances of the `Live` client will now call `Live.stop` when garbage collected - Added new publisher values for `XNAS.BASIC` and `XNAS.NLS` ## 0.36.0 - 2024-06-11 #### Enhancements - Upgraded `databento-dbn` to 0.18.1 #### Bug fixes - Fixed an issue where `heartbeat_interval_s` was not being sent to the gateway - Fixed an issue where a truncated DBN stream could be written by the `Live` client in the event of an ungraceful disconnect #### Breaking changes - Output streams of the `Live` client added with `Live.add_stream` will now upgrade to the latest DBN version before being written ## 0.35.0 - 2024-06-04 #### Enhancements - Added optional `heartbeat_interval_s` parameter to `Live` client for configuring the interval at which the gateway will send heartbeat records - Upgraded `databento-dbn` to 0.18.0 - Added new off-market publisher values for `IFEU.IMPACT` and `NDEX.IMPACT` #### Breaking changes - Renamed `CbboMsg` to `CBBOMsg` - Renamed `use_snapshot` parameter in `Live.subscribe` function to `snapshot` - All Python exceptions raised by `databento-dbn` have been changed to use the `DBNError` type ## 0.34.1 - 2024-05-21 #### Enhancements - Added `use_snapshot` parameter to `Live.subscribe`, defaults to `False` ## 0.34.0 - 2024-05-14 #### Enhancements - Added `pip-system-certs` dependency for Windows platforms to prevent a connection issue in `requests` when behind a proxy - Iteration of the `Live` client will now automatically call `Live.stop` when the iterator is destroyed, such as when a for loop is escaped with an exception or `break` statement #### Bug fixes - Fixed an issue where `batch.download` and `batch.download_async` would fail if requested files already existed in the output directory - Fixed an issue where `batch.download`, `batch.download_async`, and `timeseries.get_range` could use a lot of memory while streaming data - Fixed an issue where reusing a `Live` client with an open output stream would drop DBN records when received at the same time as the `Metadata` header #### Deprecations - The `start_date` and `end_date` keys in the response from `Historical.metadata.get_dataset_range` will be removed in a future release. Use the new `start` and `end` keys instead, which include time resolution ## 0.33.0 - 2024-04-16 #### Enhancements - The `Historical.batch.download` and `Historical.batch.download_async` methods will now automatically retry the download if a rate limit (HTTP 429) error is received - The `Historical.batch.download` and `Historical.batch.download_async` methods will now retry failed downloads automatically - The `Historical.batch.download` and `Historical.batch.download_async` methods will now download files concurrently - The `output_dir` parameter for `Historical.batch.download` and `Historical.batch.download_async` is now optional and will default to the current working directory if unspecified - Added `pip-system-certs` dependency for Windows platforms to prevent a connection issue in `requests` when behind a VPN #### Breaking changes - The `enable_partial_downloads` parameter for `Historical.batch.download` and `Historical.batch.download_async` has been removed, partial files will always be resumed which was the default behavior - The parameters for `Historical.batch.download` and `Historical.batch.download_async` have been reordered because `output_dir` is now optional, `job_id` now comes first ## 0.32.0 - 2024-04-04 #### Enhancements - Improved exception messages when multiple `ErrorMsg` are received by the `Live` client - Upgraded `databento-dbn` to 0.17.1 #### Bug fixes - Removed live session ID parsing to `int`, that could cause a session to fail when nothing was wrong #### Breaking changes - Renamed publishers from deprecated datasets to their respective sources (`XNAS.NLS` and `XNYS.TRADES` respectively) #### Deprecations - Deprecated dataset values `FINN.NLS` and `FINY.TRADES` ## 0.31.1 - 2024-03-20 #### Enhancements - Increase `Live` session connection and authentication timeouts - Added new `F_TOB` and `F_MAYBE_BAD_BOOK` variants to `RecordFlags` #### Bug fixes - Fixed an issue where calling `Live.subscribe` from a `Live` client callback would cause a deadlock ## 0.31.0 - 2024-03-05 #### Enhancements - Added `DBNStore.insert_symbology_json` convenience method for adding symbology JSON data - Upgraded `databento-dbn` to 0.16.0 ## 0.30.0 - 2024-02-22 #### Enhancements - Changed how `SymbolMappingMsg` objects are ingested by `InstrumentMap` to single source the timestamp parsing from the `databento-dbn` package #### Bug fixes - Fixed an issue where setting a timezone in `DBNStore.to_df` could cause invalid symbol mappings #### Breaking changes - Changed `Live.add_stream` to use the exclusive write mode when handling file paths so existing files won't be overwritten ## 0.29.0 - 2024-02-13 #### Enhancements - Added `tz` parameter to `DBNStore.to_df` which will convert all timestamp fields from UTC to a specified timezone when used with `pretty_ts` - Added new publisher values for consolidated DBEQ.MAX #### Bug fixes - `Live.block_for_close` and `Live.wait_for_close` will now call `Live.stop` when a timeout is reached instead of `Live.terminate` to close the stream more gracefully ## 0.28.0 - 2024-02-01 #### Enhancements - Substantially increased iteration queue size - Added methods `DBNQueue.enable` and `DBNQueue.disable` for controlling queue consumption - Added method `DBNQueue.is_enabled` to signal the queue can accept records - Added method `DBNQueue.is_full` to signal the queue has reached capacity - Added enabled checks to `DBNQueue.put` and `DBNQueue.put_nowait` #### Breaking changes - Iterating a `Live` client after the streaming session has started will now raise a `ValueError`. Calling `Live.start` is not necessary when iterating the `Live` client - Moved constant `databento.live.client.DEFAULT_QUEUE_SIZE` to `databento.live.session.DBN_QUEUE_CAPACITY` - Removed `maxsize` parameter from `DBNQueue` constructor. `DBNQueue` now subclasses `SimpleQueue` instead - Removed property `DBNQueue.enabled`, use `DBNQueue.is_enabled` instead - Removed method `DBNQueue.is_half_full`, use `DBNQueue.is_full` instead #### Bug fixes - Fixed an issue where DBN records could be dropped while iterating - Fixed an issue where async iteration would block the event loop ## 0.27.0 - 2024-01-23 #### Enhancements - Added `Session.session_id` property which will contain the numerical session ID once a live session has been authenticated - Upgraded `databento-dbn` to 0.15.1 #### Breaking changes - Renamed `DatabentoLiveProtocol.started` to `DatabentoLiveProtocol.is_started` which now returns a bool instead of an `asyncio.Event` #### Bug fixes - Fixed an issue where an error message from the live gateway would not properly raise an exception if the connection closed before `Live.start` was called ## 0.26.0 - 2024-01-16 This release adds support for transcoding DBN data into Apache parquet. #### Enhancements - Added `DBNStore.to_parquet` for transcoding DBN data into Apache parquet using `pyarrow` - Upgraded `databento-dbn` to 0.15.0 ## 0.25.0 - 2024-01-09 #### Breaking changes - Removed deprecated `pretty_px` parameter for `DBNStore.to_df`; `price_type` can be used instead #### Bug fixes - Fixed an issue where the `Live` client would not raise an exception when reading an incompatible DBN version - Fixed an issue where sending lots of subscriptions could cause a `BufferError` - Fixed an issue where `Historical.batch.download` was slow - Fixed an issue where `Historical.timeseries.get_range` was slow - Fixed an issue where reading a DBN file with non-empty metadata symbol mappings and mixed `SType` would cause an error when mapping symbols (credit: Jakob Lövhall) ## 0.24.1 - 2023-12-15 #### Enhancements - Added new publisher value for OPRA MIAX Sapphire #### Bug fixes - Fixed issue where a large unreadable symbol subscription message could be sent - Fixed issue where calling `Live.stop` could cause a truncated DBN record to be written to a stream ## 0.24.0 - 2023-11-23 This release adds support for DBN v2 as well as Python 3.12. DBN v2 delivers improvements to the `Metadata` header symbology, new `stype_in` and `stype_out` fields for `SymbolMappingMsg`, and extends the symbol field length for `SymbolMappingMsg` and `InstrumentDefMsg`. The entire change notes are available [here](https://github.com/databento/dbn/releases/tag/v0.14.0). Users who wish to convert DBN v1 files to v2 can use the `dbn-cli` tool available in the [databento-dbn](https://github.com/databento/dbn/) create. On a future date, the Databento live and historical APIs will stop serving DBN v1. This release of `databento-python` is fully compatible with both DBN v1 and v2, so this upgrade should be seamless for most users. In some cases, DBN v1 records will be converted to their v2 counterparts: - When iterating a `DBNStore` and with `DBNStore.replay` - When iterating a `Live` client and records dispatched to callbacks #### Enhancements - Added support for Python 3.12 - Improved the performance for stream writes in the `Live` client - Upgraded `databento-dbn` to 0.14.2 - Added `databento.common.types` module to hold common type annotations #### Bug fixes - Fixed an issue where specifying an OHLCV schema in `DBNStore.to_ndarray` or `DBNStore.to_df` would not properly filter records by their interval - Fixed an issue where `DBNStore.to_ndarray` and `DBNStore.to_df` with a non-zero count could get stuck in a loop if the DBN data did not contain any records #### Breaking Changes - `DBNStore` iteration and `DBNStore.replay` will upgrade DBN version 1 messages to version 2 - `Live` client iteration and callbacks upgrade DBN version 1 messages to version 2 - Moved `DBNRecord`, `RecordCallback`, and `ExceptionCallback` types to them `databento.common.types` module - Moved `AUTH_TIMEOUT_SECONDS` and `CONNECT_TIMEOUT_SECONDS` constants from the `databento.live` module to `databento.live.session` - Moved `INT64_NULL` from the `databento.common.dbnstore` module to `databento.common.constants` - Moved `SCHEMA_STRUCT_MAP` from the `databento.common.data` module to `databento.common.constants` - Removed `schema` parameter from `DataFrameIterator` constructor, `struct_type` is to be used instead - Removed `NON_SCHEMA_RECORD_TYPES` constant as it is no longer used - Removed `DERIV_SCHEMAS` constant as it is no longer used - Removed `SCHEMA_COLUMNS` constant as it is no longer used - Removed `SCHEMA_DTYPES_MAP` constant as it is no longer used - Removed empty `databento.common.data` module ## 0.23.1 - 2023-11-10 #### Enhancements - Added new publishers for consolidated DBEQ.BASIC and DBEQ.PLUS #### Bug fixes - Fixed an issue where `Live.block_for_close` and `Live.wait_for_close` would not flush streams if the timeout was reached - Fixed a performance regression when reading a historical DBN file into a numpy array ## 0.23.0 - 2023-10-26 #### Enhancements - Added `map_symbols_csv` function to the `databento` module for using `symbology.json` files to map a symbol column onto a CSV file - Added `map_symbols_json` function to the `databento` module for using `symbology.json` files to add a symbol key to a file of JSON records - Added new publisher values in preparation for IFEU.IMPACT and NDEX.IMPACT datasets #### Bug fixes - Fixed issue where a large unreadable symbol subscription message could be sent - Fixed an issue where `DBNStore.to_df` with `pretty_ts=True` was very slow ## 0.22.1 - 2023-10-24 #### Bug fixes - Fixed an issue where `DBNStore.to_csv` and `DBNStore.to_json` were mapping symbols even when `map_symbols` was set to `False` - Fixed an issue where empty symbology mappings caused a `ValueError` when loading symbols into the `DBNStore` instrument map ## 0.22.0 - 2023-10-23 #### Enhancements - Added `price_type` argument for `DBNStore.to_df` to specify if price fields should be `fixed`, `float` or `decimal.Decimal` - Added `py.typed` marker file - Upgraded `databento-dbn` to 0.13.0 #### Breaking Changes - Changed outputs of `DBNStore.to_csv` and `DBNStore.to_json` to match the encoding formats from the Databento API #### Deprecations - Deprecated `pretty_px` argument for `DBNStore.to_df` to be removed in a future release; the default `pretty_px=True` is now equivalent to `price_type="float"` and `pretty_px=False` is now equivalent to `price_type="fixed"` ## 0.21.0 - 2023-10-11 #### Enhancements - Added `map_symbols` support for DBN data generated by the `Live` client - Added support for file paths in `Live.add_stream` - Added new publisher values in preparation for DBEQ.PLUS - Upgraded `databento-dbn` to 0.11.1 #### Bug fixes - Fixed an issue where `DBNStore.from_bytes` did not rewind seekable buffers - Fixed an issue where the `DBNStore` would not map symbols with input symbology of `SType.INSTRUMENT_ID` - Fixed an issue with `DBNStore.request_symbology` when the DBN metadata's start date and end date were the same - Fixed an issue where closed streams were not removed from a `Live` client on shutdown ## 0.20.0 - 2023-09-21 This release includes batch customization options which improve data usability for CSV and JSON text encodings, as well as providing a way to split files by raw symbols. #### Enhancements - Added `ARCX.PILLAR.ARCX` publisher - Added `pretty_px` option for `batch.submit_job`, which formats prices to the correct scale using the fixed-precision scalar 1e-9 (available for CSV and JSON text encodings) - Added `pretty_ts` option for `batch.submit_job`, which formats timestamps as ISO 8601 strings (available for CSV and JSON text encodings) - Added `map_symbols` option for `batch.submit_job`, which appends a symbol field to each text-encoded record (available for CSV and JSON text encodings) - Added `split_symbols` option for `batch.submit_job`, which will split files by raw symbol - Upgraded `databento-dbn` to 0.10.2 #### Bug fixes - Fixed an issue where no disconnection exception were raised when iterating the `Live` client. - Fixed an issue where calling `DBNStore.to_df`, `DBNStore.to_json`, or `DBNStore.to_csv` with `map_symbols=True` would cause a `TypeError` #### Breaking changes - Removed `default_value` parameter from `Historical.symbology.resolve` - Swapped the ordering for the `pretty_px` and `pretty_ts` boolean parameters ## 0.19.1 - 2023-09-08 #### Bug fixes - Fixed an issue where the index column was not serialized with `DBNStore.to_json` - Fixed an issue where timestamps serialized by `DBNStore.to_json` had reduced precision ## 0.19.0 - 2023-08-25 This release includes improvements to handling large DBN data and adds support for future datasets. #### Enhancements - Added `count` parameter to `DBNStore.to_df` and `DBNStore.to_ndarray` to help process large files incrementally - Improved memory usage of `DBNStore.to_csv` and `DBNStore.to_json` - Added the `Publisher`, `Venue`, and `Dataset` enums - Replace null prices with `NaN` when `pretty_px=True` in `DBNStore.to_df()` - Upgraded `databento-dbn` to 0.8.3 #### Bug fixes - Fixed issue where exception messages were displaying JSON encoded data - Fixed typo in `BATY.PITCH.BATY` publisher - Reduced floating error when converting prices to floats with `pretty_px=True` #### Breaking changes - `DBNStore.to_df` now always utf-8 decodes string fields ## 0.18.1 - 2023-08-16 #### Bug fixes - Fixed issue where extra `python` key was sent by the `Live` client ## 0.18.0 - 2023-08-14 #### Breaking changes - Renamed the `TimeSeriesHttpAPI` class to `TimeseriesHttpAPI` #### Bug fixes - Fixed an issue where `DBNStore.to_csv()`, `DBNStore.to_df()`, `DBNStore.to_json()`, and `DBNStore.to_ndarray()` would consume large amounts of memory ## 0.17.0 - 2023-08-10 This release includes improvements to the ergonomics of the clients metadata API, you can read more about the changes [here](https://databento.com/blog/api-improvements-august-2023). #### Enhancements - Upgraded `databento-dbn` to 0.8.2 #### Breaking changes - Changed `metadata.list_publishers()` to return a list of publisher details objects - Changed `metadata.list_fields(...)` to return a list of field detail objects for a particular schema and encoding - Changed `metadata.list_fields(...)` to require the `schema` and `encoding` parameters - Changed `metadata.list_unit_prices(...)` to return a list of unit prices for each feed mode and data schema - Changed `metadata.list_unit_prices(...)` to require the `dataset` parameter - Removed `metadata.list_unit_prices(...)` `mode` and `schema` parameters - Removed `metadata.list_fields(...)` `dataset` parameter ## 0.16.1 - 2023-08-03 #### Bug fixes - Fixed an issue where starting a `Live` client before subscribing gave an incorrect error message - Fixed an issue where a `Live` client exception callback would fail when the callback function does not have a `__name__` attribute ## 0.16.0 - 2023-07-25 This release includes updates to the fields in text encodings (CSV and JSON), you can read more about the changes [here](https://databento.com/blog/CSV-JSON-updates-july-2023). #### Enhancements - Added `rtype` field to all schemas that was previously excluded #### Breaking changes - Reordered fields of DataFrame and CSV encoded records to match historical API ## 0.15.2 - 2023-07-19 #### Bug fixes - Fixed an issue where the `end` parameter in `timeseries.get_range_async` did not support a value of `None` - Fixed an issue where `timeseries.get_range` requests would begin with an invalid `path` parameter ## 0.15.1 - 2023-07-06 #### Bug fixes - Fixed an issue with release tests - Fixed an issue with release workflow ## 0.15.0 - 2023-07-05 #### Enhancements - Added `symbology_map` property to `Live` client - Added `optional_symbols_list_to_list` parsing function - Changed `Live.add_callback` and `Live.add_stream` to accept an exception callback - Changed `Live.__iter__()` and `Live.__aiter__()` to send the session start message if the session is connected but not started - Upgraded `databento-dbn` to 0.7.1 - Removed exception chaining from exceptions emitted by the library #### Bug fixes - Fixed issue where a large unreadable symbol subscription message could be sent - Fixed an `ImportError` observed in Python 3.8 #### Breaking changes - Removed `Encoding`, `Compression`, `Schema`, and `SType` enums as they are now exposed by `databento-dbn` - Renamed `func` parameter to `record_callback` for `Live.add_callback` and `Live.add_stream` - Removed `optional_symbols_list_to_string` parsing function ## 0.14.1 - 2023-06-16 #### Bug fixes - Fixed issue where `DBNStore.to_df()` would raise an exception if no records were present - Fixed exception message when creating a `DBNStore` from an empty data source ## 0.14.0 - 2023-06-14 #### Enhancements - Added `DatabentoLiveProtocol` class - Added `metadata` property to `Live` - Added support for emitting warnings in API response headers - Added support for reusing a `Live` client to reconnect - Relaxed 10 minute minimum request time range restriction - Upgraded `aiohttp` to 3.8.3 - Upgraded `numpy` to 1.23.5 - Upgraded `pandas` to 1.5.3 - Upgraded `requests` to 2.28.1 - Upgraded `zstandard` to 0.21.0 #### Breaking changes - Removed support for Python 3.7 - Renamed `symbol` to `raw_symbol` in definition schema when converting to a DataFrame - Changed iteration of `Live` to no longer yield DBN metadata - Changed `Live` callbacks to no longer yield DBN metadata #### Bug fixes - Fixed optional `end` parameter for `batch.submit_job(...)` ## 0.13.0 - 2023-06-02 #### Enhancements - Added support for `statistics` schema - Added batch download support data files (`condition.json` and `symbology.json`) - Renamed `booklevel` MBP field to `levels` for brevity and consistent naming - Upgraded `databento-dbn` to 0.6.0 #### Breaking changes - Changed `flags` field to an unsigned int - Changed default of `ts_out` to `False` for `Live` client - Changed `instrument_class` DataFrame representation to be consistent with other `char` types - Removed `open_interest_qty` and `cleared_volume` fields that were always unset from definition schema - Removed sunset `timeseries.stream` method - Removed support for legacy stypes ## 0.12.0 - 2023-05-01 #### Enhancements - Added `Live` client for connecting to Databento's live service - Added `degraded`, `pending` and `missing` condition variants for `batch.get_dataset_condition` - Added `last_modified_date` field to `batch.get_dataset_condition` response - Upgraded `DBNStore` to support mixed schema types to support live data - Upgraded `databento-dbn` to 0.5.0 #### Breaking changes - Changed iteration `DBNStore` to return record types from `databento-dbn` instead of numpy arrays - Renamed the `cost` field to `cost_usd` for `batch.submit_job` and `batch.list_jobs` (value now expressed as US dollars) - Renamed `product_id` field to `instrument_id` - Renamed `symbol` field in definitions to `raw_symbol` - Removed `dtype` property from `DBNStore` - Removed `record_size` property from `DBNStore` - Removed `bad` condition variant from `batch.get_dataset_condition` - Removed unused `LiveGateway` enum - Removed `STATISTICS` from `Schema` enum - Removed `STATUS` from `Schema` enum - Removed `GATEWAY_ERROR` from `Schema` enum - Removed `SYMBOL_MAPPING` from `Schema` enum #### Deprecations - Deprecated `SType.PRODUCT_ID` to `SType.INSTRUMENT_ID` - Deprecated `SType.NATIVE` to `SType.RAW_SYMBOL` - Deprecated `SType.SMART` to `SType.PARENT` and `SType.CONTINUOUS` ## 0.11.0 - 2023-04-13 #### Bug fixes - Changed `end` and `end_date` to optional to support new forward-fill behaviour - Upgraded `zstandard` to 0.20.0 ## 0.10.0 - 2023-04-07 #### Enhancements - Added support for `imbalance` schema - Added `instrument_class`, `strike_price`, and `strike_price_currency` to definition schema - Changed parsing of `end` and `end_date` params throughout the API - Improved exception messages for server and client timeouts - Upgraded `databento-dbn` to 0.4.3 #### Breaking changes - Renamed `Bento` class to `DBNStore` - Removed `metadata.list_compressions` (redundant with docs) - Removed `metadata.list_encodings` (redundant with docs) - Removed optional `start` and `end` params from `metadata.list_schemas` (redundant) - Removed `related` and `related_security_id` from definition schema ## 0.9.0 - 2023-03-10 #### Enhancements - Improved use of the logging module #### Breaking changes - Removed `record_count` property from Bento class - Changed `metadata.get_dataset_condition` response to a list of condition per date #### Bug fixes - Fixed bug in `Bento` where invalid metadata would prevent iteration ## 0.8.1 - 2023-03-05 #### Enhancements - Added `from_dbn` convenience alias for loading DBN files #### Bug fixes - Fixed bug in `Bento` iteration where multiple readers were created ## 0.8.0 - 2023-03-03 #### Enhancements - Added `batch.list_files(...)` method - Added `batch.download(...)` method - Added `batch.download_async(...)` method - Integrated DBN encoding 0.3.2 #### Breaking changes - Dropped support for DBZ encoding - Renamed `timeseries.stream` to `timeseries.get_range` - Renamed `timeseries.stream_async` to `timeseries.get_range_async` - Changed `.to_df(...)` `pretty_ts` default argument to `True` - Changed `.to_df(...)` `pretty_px` default argument to `True` - Changed `.to_df(...)` `map_symbols` default argument to `True` #### Deprecations - Deprecated `timeseries.stream(...)` method - Deprecated `timeseries.stream_async(...)` method ## 0.7.0 - 2023-01-10 - Added support for `definition` schema - Updated `Flags` enum - Upgraded `dbz-python` to 0.2.1 - Upgraded `zstandard` to 0.19.0 ## 0.6.0 - 2022-12-02 - Added `metadata.get_dataset_condition` method to `Historical` client - Upgraded `dbz-python` to 0.2.0 ## 0.5.0 - 2022-11-07 #### Enhancements - Updated `quickstart.ipynb` jupyter notebook #### Bug fixes - Fixed dataframe columns for derived data schemas (dropped `channel_id`) - Fixed `batch.submit_job` requests for `dbz` encoding - Updated `quickstart.ipynb` jupyter notebook ## 0.4.0 - 2022-09-14 - Added `map_symbols` option for `.to_df()` (experimental) - Upgraded `dbz-python` to 0.1.5 ## 0.3.0 - 2022-08-30 - Initial release --- Source: https://databento.com/docs/release-notes/release-notes-http # HTTP API ## 0.36.0 - TBD #### Enhancements - Added support for `year` split_duration for batch job file splitting - Added new venues, datasets, and publishers for US Equities Securities Information Processors - Added a new endpoint `/batch.get_job_details` which returns the full details for a specified batch job - Added `short` parameter to `/batch.list_jobs` to allow early migration to the upcoming short response format - Added an `allocate_isins` parameter to `/adjustment_factors.get_range`, `/corporate_actions.get_range`, `/security_master.get_range`, and `/security_master.get_last` to control whether new ISINs are allocated for plans that are ISIN-limited - Added an optional `dataset` parameter to `/metadata.list_fields`. If not provided, the returned fields are for the latest DBN encoding version, which may not match a specific dataset's schema ## 0.35.0 - 2025-08-19 #### Breaking changes - Removed `bill_id` field from the response of `/batch.submit_job` and `/batch.list_jobs` endpoints #### Enhancements - Added per-schema availability ranges to the response of the `/metadata.get_dataset_range` endpoint ## 0.34.1 - 2025-06-17 #### Enhancements - Changed the following Venue, Publisher, and Dataset descriptions: - "ICE Futures Europe (Financials)" renamed to "ICE Europe Financials" - "ICE Futures Europe (Commodities)" renamed to "ICE Europe Commodities" ## 0.34.0 - 2025-06-09 #### Breaking changes - Updated the names of several subfields in the `/corporate_actions.get_range` response, under the `date_info`, `event_info`, and `rate_info` fields. The following subfields were renamed: - `decl_currency` renamed to `declared_currency` - `decl_gross_amount` renamed to `declared_gross_amount` - `f_x_rate` renamed to `fx_rate` - `iss_new_name` renamed to `issuer_new_name` - `iss_old_name` renamed to `issuer_old_name` - `new_bbg_company_id` renamed to `new_bbg_comp_id` - `new_bbg_company_tk` renamed to `new_bbg_comp_ticker` - `new_bbg_exh_id` renamed to `new_figi` - `new_bbg_exh_tk` renamed to `new_figi_ticker` - `new_min_tra_qty` renamed to `new_min_trading_qty` - `new_mktsg_id` renamed to `new_market_segment_id` - `new_reg_s144_a` renamed to `new_reg_s144a` - `new_unit_sec_id` renamed to `new_unit_security_id` - `offeree_iss_id` renamed to `offeree_issuer_id` - `offeror_iss_id` renamed to `offeror_issuer_id` - `old_bbg_company_id` renamed to `old_bbg_comp_id` - `old_bbg_company_tk` renamed to `old_bbg_comp_ticker` - `old_bbg_exh_id` renamed to `old_figi` - `old_bbg_exh_tk` renamed to `old_figi_ticker` - `old_min_tra_qty` renamed to `old_min_trading_qty` - `old_mktsg_id` renamed to `old_market_segment_id` - `old_reg_s144_a` renamed to `old_reg_s144a` - `old_unit_sec_id` renamed to `old_unit_security_id` - `pp_sec_id` renamed to `pp_security_id` - `poolfactor` renamed to `pool_factor` - `pre_offer_q_ty` renamed to `pre_offer_qty` - `qual_st_cap_gains` renamed to `qual_short_term_cap_gains` - `redem_percentage` renamed to `redemption_percentage` - `st_cap_gains` renamed to `short_term_cap_gains` - `sec_new_name` renamed to `security_new_name` - `sec_old_name` renamed to `security_old_name` - `section199_a_foreign_tax_paid` renamed to `section199a_foreign_tax_paid` - `section199_a_inc_div` renamed to `section199a_inc_div` - `section199_a_st_cap_gain` renamed to `section199a_short_term_cap_gain` - `tra_isin` renamed to `trading_isin` - `tra_sec_id` renamed to `trading_security_id` - `us_deposit_receiptateto_currency` renamed to `usd_rate_to_currency` - `cashbak` renamed to `cash_back` - `companyulsory_acq_date` renamed to `compulsory_acq_date` - `frankdiv` renamed to `franked_div` - `lead_plntiff_deadline_date` renamed to `lead_plaintiff_deadline_date` - `maxprice` renamed to `max_price` - `minprice` renamed to `min_price` - `redem_premium` renamed to `redemption_premium` - `redem_price` renamed to `redemption_price` - `unit_frankdiv` renamed to `unfranked_div` ## 0.33.0 - 2024-12-10 #### Enhancements - Removed deprecated `packaging` parameter from `batch.submit_job`. Job files can be downloaded individually or as zip files after the job completes ## 0.32.0 - 2024-11-26 #### Enhancements - Introduced POST variants of the following endpoints to complement their existing GET versions: - POST `/metadata.get_record_count(...)` - POST `/metadata.get_billable_size(...)` - POST `/metadata.get_cost(...)` ## 0.31.0 - 2024-11-12 #### Deprecations - The `packaging` parameter for `batch.submit_job` is deprecated and will be removed on 2024-12-10. To download all batched files as a dynamic ZIP from the `batch/download` endpoint, use `'{job_id}.zip'` as the filename instead. ## 0.30.0 - 2024-09-24 This release introduces the corporate actions and adjustment factors datasets for the reference data API, both with global coverage. The corporate actions dataset provides point-in-time (PIT) corporate action events, while the adjustment factors dataset provides end-of-day price adjustments for capital events, supporting multiple currencies for the same event. You can read more about the release [here](https://databento.com/blog/corporate-actions). #### Enhancements - Added `corporate_actions.get_range(...)` endpoint - Added `adjustment_factors.get_range(...)` endpoint ## 0.29.0 - 2024-09-03 #### Deprecations - The TAR `packaging` option (`tar`) for `batch.submit_job` is deprecated and will be removed at a future date. We recommend you switch to the ZIP `packaging` option (`zip`), which will remain available. ## 0.28.0 - 2024-06-25 #### Enhancements - Added support for the `status` schema for the `GLBX.MDP3` dataset ## 0.27.0 - 2024-06-04 #### Breaking changes - Removed the `start_date` and `end_date` keys in the response from `/metadata.get_dataset_range` ## 0.26.0 - 2024-05-14 #### Enhancements - Added `start` and `end` keys to the response from `/metadata.get_dataset_range` to provide time resolution and an exclusive end time #### Deprecations - The `start_date` and `end_date` keys in the response from `/metadata.get_dataset_range` will be removed in a future release. Use the new `start` and `end` keys instead, which include time resolution ## 0.25.0 - 2024-03-26 #### Breaking changes - Removed the forward-fill behavior of `end` parameters with unspecified months, days, hours, minutes, or seconds ## 0.24.0 - 2024-03-06 #### Deprecations - The forward-fill behavior of `end` parameters with unspecified months, days, hours, minutes, or seconds will be removed. To ensure consistent time ranges between old and new behavior: specify an explicit `end` as an ISO 8601 string up to seconds resolution, or a UNIX nanoseconds integer. See https://to.dbn.to/api-change-2024-03-27 ## 0.23.0 - 2024-02-15 #### Enhancements - A `/timeseries.get_range` request which contains one or more unresolved symbols will return a warning message in the response's warnings header - A `/timeseries.get_range` request will now return a `206 Partial Content` status code if one or more symbols were not resolved ## 0.22.0 - 2024-02-06 #### Enhancements - A `/timeseries.get_range` request with a date range that contains one or more days with a condition other than available will return a warning message in the response's warnings header ## 0.21.0 - 2024-01-30 #### Breaking changes - Reduced `/batch.submit_job` rate limit down from 20/second to 20/minute ## 0.20.0 - 2024-01-18 This release introduces live and historical ICE Europe Commodities iMpact (`IFEU.IMPACT`) and ICE Endex iMpact (`NDEX.IMPACT`) data. The datasets cover European futures and options for all energy and commodities products, including all listed outrights, spreads, and options combinations. You can read more about the release [here](https://databento.com/blog/ice-futures-market-data). #### Enhancements - Added the `IFEU.IMPACT` and `NDEX.IMPACT` datasets, does not yet include our `mbo`, `mbp-10`, `mbp-1`, and `tbbo` schemas ## 0.19.0 - 2023-10-17 #### Breaking changes - Standardized timestamps in responses to ISO 8601 format with a `T` separator and `Z` suffix for UTC - Renamed `mappings` field in the `symbology.json` to `result` to match the `/symbology.resolve` endpoint response ## 0.18.0 - 2023-10-11 #### Bug fixes - Fixed an issue where `/timeseries.get_range` requests for OHLCV schemas and `stype_in=parent` with an intraday start would always return no data ## 0.17.0 - 2023-10-04 This release introduces the Databento Equities Basic dataset (`DBEQ.BASIC`), our first bundle of live and historical US equities data. The bundle includes five proprietary feeds--NYSE Chicago Integrated, IEX TOPS, NYSE National Trades, NYSE National BBO, and MIAX Pearl Equities Depth of Market--that can be licensed for distribution, display, and non-display applications at no cost. You can read more about the release [here](https://databento.com/blog/dbeq-basic). #### Enhancements - Added the `DBEQ.BASIC` dataset ## 0.16.0 - 2023-09-26 #### Breaking changes - Symbology mappings no longer include unresolved symbols, this change affects DBN metadata and `/symbology.resolve` responses - Changed batch job `symbology.csv` schema (now a row per date) #### Bug fixes - Fixed symbology resolution on weekends (now uses mappings from previous Friday) - Fixed consistent streaming of CSV header when no data is found for a request ## 0.15.0 - 2023-09-19 This release includes customization options which improve data usability for CSV and JSON text encodings, as well as providing a way to split files by raw symbols for batch jobs. #### Enhancements - Added `pretty_px` option for `/timeseries.get_range` and `/batch.submit_job`, which formats prices to the correct scale using the fixed-precision scalar 1e-9 (available for CSV and JSON text encodings) - Added `pretty_ts` option for `/timeseries.get_range` and `/batch.submit_job`, which formats timestamps as ISO 8601 strings (available for CSV and JSON text encodings) - Added `map_symbols` option for `/timeseries.get_range` and `/batch.submit_job`, which appends a symbol field to each text-encoded record (available for CSV and JSON text encodings) - Added `split_symbols` option for `/batch.submit_job`, which will split files by raw symbol ## 0.14.0 - 2023-08-29 This release introduces the OPRA (Options Price Reporting Authority) dataset (`OPRA.PILLAR`) for the historical and live APIs, covering all 16 national exchanges. This not only includes options on single name equities, but also on indices, e.g., Cboe VIX options, SPX options, and certain volatility benchmark indices like SPIKE and VSPKE. You can read more about the release [here](https://databento.com/blog/opra-data). #### Enhancements - Added the `OPRA.PILLAR` dataset, does not yet include Global Trading Hours (GTH) ## 0.13.0 - 2023-08-23 #### Enhancements - Added a `symbology.csv` to batch download support files for the `csv` encoding - Changed `/symbology.resolve` to support input symbology of `instrument_id` and output of `raw_symbol` ## 0.12.0 - 2023-08-10 This release includes improvements to the ergonomics of our metadata API, you can read more about the changes [here](https://databento.com/blog/api-improvements-august-2023). Note that the `/timeseries.get_range` and `/symbology.resolve` endpoints both accept `POST` and `GET` requests, with `POST` being the primary and recommended method. #### Breaking changes - Changed `/batch.submit_job` to now require form-encoded data - Changed `/metadata.list_publishers` response to an ordered list of publisher details objects - Changed `/metadata.list_fields` response to an ordered list of field detail objects for a particular schema and encoding - Changed `/metadata.list_fields` to now require the `schema` and `encoding` query parameters - Changed `/metadata.list_unit_prices` response to a list of unit prices for each feed mode and data schema - Changed `/metadata.list_unit_prices` to now require the `dataset` query parameter - Removed `/metadata.list_unit_prices` `mode` and `schema` query parameters - Removed `/metadata.list_field` `dataset` query parameter ## 0.11.0 - 2023-07-25 This release includes updates to the fields in text encodings (CSV and JSON), you can read more about the changes [here](https://databento.com/blog/CSV-JSON-updates-july-2023). #### Breaking changes - Changed CSV encoding column ordering for most schemas - Changed JSON encoding to serialize header fields in nested `hd` object (consistency with how `levels` are serialized) - Changed JSON encoding to encode all 64-bit integers as strings to avoid loss of precision when converting to double-precision floating point - Added `rtype` field/column to MBP schemas that was previously excluded - Added `depth` field/column to MBP schemas that was previously excluded - Renamed JSON fields `bid_oq_*` and `ask_oq_*` to `bid_ct_*` and `ask_ct_*` respectively - Renamed `security_group` to `group` for the `definition` schema ## 0.10.0 - 2023-07-06 #### Enhancements - Added warnings header for API responses - Added POST request support for `timeseries.get_range` and `symbology.resolve` endpoints - Improved behavioral consistency of query params `start`, `end`, `start_date` and `end_date` across the API (exclusive end) - Relaxed 10 minute minimum request time range restriction - Added `rtype` to encoded CSV and JSON #### Deprecations - Deprecated `symbology` section in the `metadata.json` for batch downloads (will be removed in a future version after July 3rd) #### Bug fixes - Fixed handling of certain errors involving invalid `parent` or `continuous` symbols (no longer a HTTP 500 error) ## 0.9.0 - 2023-06-01 #### Enhancements - Added additional batch download support data files (`condition.json` and `symbology.json`) #### Breaking changes - Removed support for sunset `/timeseries.stream` endpoint - Removed support for legacy stypes ## 0.8.0 - 2023-05-01 #### Enhancements - Added additional field `last_modified_date` to `/metadata.get_dataset_condition` response - Added `degraded`, `pending` and `missing` condition variants for `/metadata.get_dataset_condition` - Added `last_modified_date` field to `batch.get_dataset_condition` response #### Breaking changes - Renamed `product_id` field to `instrument_id` - Renamed `symbol` field in definitions to `raw_symbol` - Renamed the `cost` field to `cost_usd` for `/batch.submit_job` and `/batch.list_jobs` (value now expressed as US dollars) - Removed `bad` condition variant from `batch.get_dataset_condition` #### Deprecations - Deprecated stype `product_id` in favor of `instrument_id` - Deprecated stype `native` in favor of `raw_symbol` - Deprecated stype `smart` in favor of separate `parent` and `continuous` stypes ## 0.7.0 - 2023-04-07 #### Breaking changes - Removed `/metadata.list_compressions` endpoint (redundant with docs) - Removed `/metadata.list_encodings` endpoint (redundant with docs) - Removed optional `start` and `end` params from `/metadata.list_schemas` endpoint (redundant) - Changed parsing of `end` and `end_date` params throughout the API ## 0.6.0 - 2023-03-10 #### Breaking changes - Changed `metadata.get_dataset_condition` response to an array of conditions per date ## 0.5.0 - 2023-03-03 #### Enhancements - Added `/batch.list_files` endpoint for listing batch job file details - Integrated DBN encoding 0.3.2 #### Breaking changes - Renamed `/timeseries.stream` to `/timeseries.get_range` ## 0.4.0 - 2022-12-02 - Added `/metadata.get_dataset_condition` endpoint to discover availability and quality - Improved smart symbology resolution ## 0.3.0 - 2022-08-30 - Added Databento Binary Encoding (DBZ) streaming and batch downloads - Added initial smart symbology support - Further improved consistency of parameter and field names ## 0.2.0 - 2021-12-10 - Added historical APIs for metadata, time series and symbology - Added streaming and batch download support - Added native product ID and symbol support - Added sizing and pricing functionality - Improved error handling, size limits and rate limits - Improved consistency of field names between schemas ## 0.1.0 - 2021-08-30 - Added support for server-streaming RPC over HTTP - Added basic schemas and encoding (binary, CSV, JSON) support - Added new datasets: CME Globex MDP 3.0 --- Source: https://databento.com/docs/release-notes/release-notes-raw # Raw API ## 0.9.4 - Upcoming #### Bug fixes - Fixed an unclear error when the data for a replay subscription was no longer retained - Sessions configured to skip records now resume from the earliest retained records; other sessions receive a fatal `ErrorMsg` with the new `ReplayDataAgedOut` code and the earliest available `start` time ## 0.9.3 - 2026-08-01 #### Bug fixes - Fixed an issue where heartbeats could be suppressed while buffered data had not yet reached the client ## 0.9.2 - 2026-07-18 #### Bug fixes - Fixed an issue with delayed delivery of imbalance for some US equities datasets - Fixed missing incremental symbol mappings for mapping changes inside the replay window of subscriptions with a non-zero `start` time ## 0.9.1 - 2026-06-27 #### Enhancements - Improved time to first record when subscribing with a non-zero `start` time - Added new venues, datasets, and publishers for US Equities Securities Information Processors - Added `StatType` variants `MwcbLevel1`, `MwcbLevel2`, and `MwcbLevel3` ## 0.9.0 - 2026-04-26 #### Enhancements - Made symbol resolution failures non-fatal: unresolved symbols now result in `ErrorMsg` records with `code=SymbolResolutionFailed` instead of terminating the session, while successfully resolved symbols in the same subscription continue to stream normally #### Breaking changes - Changed the default `slow_reader_behavior` from `warn` to `skip` for sessions that don't set it explicitly #### Bug fixes - Fixed symbol resolution for instruments using valid historical timestamp for `start_ts` ## 0.7.5 - 2026-02-28 #### Bug fixes - Stopped accepting subscriptions with `start` time in the future ## 0.7.4 - 2026-02-15 #### Enhancements - Added support for new `slow_reader_behavior` session parameter ## 0.7.3 - 2026-02-10 #### Enhancements - Started setting `start_ts` from source definition and `end_ts` for replay subscriptions to when the mapping ended - Improved reliability of sending error records before tearing down socket #### Bug fixes - Fixed end-of-interval notifications sent for timestamps before subscription `start` ## 0.7.2 - 2025-12-14 #### Enhancements - Improved the performance of symbol resolution and raised the subscription rate limit ## 0.7.1 - 2025-11-09 #### Enhancements - Included `security_type=IRS` (interest rate swaps) under `FUT` parent symbology ## 0.7.0 - 2025-10-26 #### Enhancements - Improved symbology resolution before the publisher has sent all definitions: this primarily affects CME Globex MDP 3.0 on Sundays and OPRA Pillar on weekday mornings - Changed `OPT` parent symbology to include `MLEG` instruments - Added `.SPOT` parent symbology for spot instruments #### Breaking changes - Removed support for deprecated `stype_in=smart` ## 0.6.4 - 2025-09-28 #### Enhancements - Improved error message for sending empty command message - Added end-of-interval notification `SystemMsg` records ## 0.6.3 - 2025-09-07 #### Enhancements - Improved the error message for sending a command message with a byte order mark (BOM) ## 0.6.2 - 2025-08-02 #### Bug fixes - Fixed issue where subscriptions with more than 1000 symbols were only partially handled - Updated continuous symbology logic for open interest and volume ranking to fall back on expiration to match the behavior of the historical API ## 0.6.1 - 2025-06-29 #### Enhancements - Added [replay completed notification `SystemMsg` records](https://roadmap.databento.com/b/n0o5prm6/feature-ideas/add-replay-finished-notification-to-live-api) - Added throttling to subscription handling so one client sending lots of subscriptions can't degrade other users' sessions ## 0.6.0 - 2025-05-24 #### Enhancements - Added [subscription acknowledgement `SystemMsg` records](https://roadmap.databento.com/b/n0o5prm6/feature-ideas/add-a-subscription-acknowledgement-to-the-live-api) - Added a new optional `is_last` field to subscription requests to indicate the request is part of a larger subscription - Began setting `code` field in DBN version 2+ `ErrorMsg` and `SystemMsg` records - Added slow reader warnings in the form of `SystemMsg` records ## 0.5.6 - 2025-04-06 #### Enhancements - Added support for continuous open interest (`.n.`) and volume (`.v.`) symbology for ICE Europe Commodities and ICE Endex ## 0.5.5 - 2024-12-01 #### Enhancements - Added support for continuous calendar symbology for ICE Europe Commodities and ICE Endex ## 0.5.4 - 2024-10-02 #### Enhancements - Subscriptions with `stype_in`=`instrument_id` will now return SymbolMapping records ## 0.5.3 - 2024-10-02 #### Bug fixes - Fixed issue where an instrument definition with `security_update_action` Delete could cause `ALL_SYMBOLS` subscriptions to drop records - Subscription with `stype_in`=`instrument_id` will now fail on unknown instrument ids ## 0.5.1 - 2024-07-24 #### Bug fixes - Fixed issue where start times falling between trading sessions could be rejected ## 2024-07-20 #### Bug fixes - Fixed bug where sessions could be closed with "Internal error" ## 2024-06-25 #### Enhancements - Added subscription option `snapshot=...` to start live MBO data with an order book snapshot ## 0.5.0 - 2024-05-25 #### Enhancements - Added support for optional `snapshot` field in subscription control message to request a snapshot of stateful schemas like MBO - Added support for optional `heartbeat_interval_s` field in authentication control message to configure the interval when heartbeat messages will be sent - Added support for optional `id` field in subscription control message ## 0.4.6 - 2024-04-13 #### Bug fixes - Disabled continuous symbology for ICE Endex and ICE Europe Commodities until it can be implemented correctly ## 0.4.5 - 2024-03-25 #### Bug fixes - Fixed inadvertent change to session ID format after successfully authenticating with live gateway ## 0.4.4 - 2024-03-23 #### Enhancements - Added functionality to send `SymbolMappingMsg` records when subscribing to `ALL_SYMBOLS` #### Bug fixes - Fixed sending `SymbolMappingMsg` for all instruments in parent symbol when subscribing to a continuous symbol ## 0.4.3 - 2024-02-13 #### Enhancements - Changed rate limiter to have more predictable behavior ## 0.4.2 - 2024-01-06 #### Enhancements - Improved error messages #### Bug fixes - Fixed command truncation ## 0.4.0 - 2023-11-08 #### Breaking changes - Added support to respond with DBNv2. DBN version is indicated in the metadata header ## 0.3.0 - 2023-10-20 #### Enhancements - When a subscription has symbols that fail to resolve, a separate `ErrorMsg` record will be sent for each symbol - Added `pretty_px` session parameter which formats prices to the correct scale using the fixed-precision scalar 1e-9 (available for JSON encoding) - Added `pretty_ts` session parameter which formats timestamps as ISO 8601 strings (available for JSON encoding) ## 0.2.0 - 2023-07-23 #### Breaking changes - Updated JSON encoding: - Moved header fields (`rtype`, `publisher_id`, `instrument_id`, and `ts_event`) to nested object under the key `hd` in JSON encoding to match structure definitions - Added `channel_id` to MBO schema - Changed to always include `rtype` to differentiate between schemas, making `details` session parameter redundant - Rename `security_group` in definitions schema to `group` to match struct definitions - All 64-bit integers are now serialized as strings to avoid loss of precision #### Deprecations - Deprecated `details` session parameter ## 0.1.0 - 2023-05-01 - Initial support for real-time and intraday data --- Source: https://databento.com/docs/release-notes/release-notes-rust # Rust ## 0.64.0 - Upcoming #### Enhancements - Added `LiveClient::unsubscribe()` and the `Unsubscription` type for removing symbols from a live session's subscriptions - Added `LiveClient::unsubscriptions()` and `LiveClient::unsubscriptions_mut()` - Added `Subscription::to_unsubscription()` and `From<&Subscription>` for `Unsubscription` for reversing a subscription - Modified `LiveClient::resubscribe()` to resubscribe only to active subscriptions #### Breaking changes - Added `sent_at` field to `Subscription` ## 0.63.0 - 2026-09-29 #### Enhancements - Optimized `LiveClient::next_record()` and `LiveClient::try_next_record()` to decode records in batches - Upgraded DBN version to 0.71.0 - Upgraded zstd to 0.14 #### Deprecations - From DBN: - Deprecated `dbn::decode::dbn::async_decode_record_ref_with_fsm`; `DbnFsm` can be used directly ## 0.62.0 - 2026-09-21 #### Enhancements - Upgraded DBN version to 0.70.0: - Sped up `decode_ref()` and `decode_record_ref()` on the sync and async record decoders, which now decode records in batches - Added `DbnFsm::process_batch()` for decoding a batch of records in a single call, along with `has_buffered_record()` and `next_buffered_record()` for draining the batch - Added new venue, dataset, and publishers for Databento Core Indices #### Breaking changes - From DBN: - Replaced `Record::header()` with a method per header field: `raw_rtype()`, `publisher_id()`, `instrument_id()`, and `raw_ts_event()`, joining the existing `record_size()`, `rtype()`, and `publisher()`. `RecordRef`, `RecordRefMut`, `RecordBuf`, `RecordEnum`, and `RecordRefEnum` keep a `header()` method of their own, and concrete record types keep their `hd` field - `Record::raw_rtype()` returns `u16` and `Record::instrument_id()` returns `u64`, wider than the corresponding `RecordHeader` fields, which are unchanged - `RType::try_into_schema()` takes a `u16` and returns `None` for values outside the range of the `RecordHeader` `rtype` field #### Bug fixes - From DBN: - Fixed `DbnFsm::process_all()` and `DbnFsm::process_many()` returning `Record` with no records instead of `ReadMore` when the buffer held no complete record - Fixed unsound `RecordRefMut` accessors, which returned references outliving the borrow of `self` and let safe code hold two mutable references to the same record, or a shared reference aliasing a mutable one. `get_mut()` and `get_mut_unchecked()` now take `&mut self`, and `header()` returns a reference borrowing `self` - Fixed undefined behavior when mutating a record through a `RecordRefMut`, where the write went through a pointer that permitted only reads, or that did not span the whole record - Fixed `DbnFsm::process_all()` and `DbnFsm::process_many()` discarding the records decoded before an invalid record. They now return those records and the error on the following call, matching `DbnFsm::process()` ## 0.61.0 - 2026-09-01 #### Enhancements - Upgraded DBN version to 0.69.0: - Added `UnsubscribeAck` to the `SystemCode` enum for acknowledging unsubscribe requests #### Bug fixes - From DBN: - Fixed unsound `AsRef<[u8]>` for `RecordBuf`, which produced a slice longer than the buffer when the header `length` written through `raw_buf_mut()` exceeded the capacity - Fixed incorrect count in DBN metadata causing decoding to read into the start of records ## 0.60.0 - 2026-08-25 #### Enhancements - Upgraded DBN version to 0.68.0: - Added `MetadataEncoder::encoded_len()` for the number of bytes `encode()` will write - Improved documentation for the `min_price_increment_amount` and `inst_attrib_value` fields #### Breaking changes - Changed `BatchClient::list_jobs()` to request the new short response format of the `batch.list_jobs` endpoint and return `Vec`, which contains only the `id`, `state`, and `ts_received` fields for each job. Use `BatchClient::get_job_details()` to fetch the complete details of an individual job #### Deprecations - Deprecated `BatchClient::list_jobs_full()`, which returns the full job details like `list_jobs()` did previously. The `batch.list_jobs` endpoint will stop returning full job details at a future date, at which point this method will be removed ## 0.59.0 - 2026-08-18 #### Enhancements - Added a `dataset` field to `ListFieldsParams` for `MetadataClient::list_fields()`. If not provided, the returned fields are for the latest DBN encoding version, which may not match a specific dataset's schema - Upgraded DBN version to 0.67.0: - Added `SType::ListingId`, `SType::IssuerId`, and `SType::SecurityId` variants, for use with the reference data API - Upgraded `sha2` version to 0.11 #### Breaking changes - Updated the minimum supported `tokio` version to 1.47, which was released one year ago ## 0.58.0 - 2026-08-11 #### Enhancements - Upgraded DBN version to 0.66.0: - Improved performance of record field accessors that convert a raw value to an enum, such as `side()`, `action()` and `rtype()`, by no longer copying the description when constructing the conversion error - Improved performance of `Record::raw_index_ts()` for records with an unrecognized `rtype` - Relaxed the bound on `From<&R> for RecordRef` from `R: HasRType` to `R: Record`, so a `RecordRef` can be constructed from any record type, including `RecordRef` itself ## 0.57.0 - 2026-08-04 #### Enhancements - Added an `allocate_isins` field to `GetRangeParams` and `GetLastParams` for `AdjustmentFactorsClient`, `CorporateActionsClient`, and `SecurityMasterClient` to control whether new ISINs are allocated for plans that are ISIN-limited - Upgraded DBN version to 0.65.0: - Added a `ReplayDataAgedOut` variant to `ErrorCode` for when the data for a replay subscription is no longer retained and the schema is incompatible with skipping records - Added new publisher values for JPX - Upgraded `time` version to 0.3.55 ## 0.56.0 - 2026-07-28 #### Enhancements - Added `CorporateActionsClient::list_events()` and `list_enums()` for fetching documentation on supported corporate action event types and enum values - Upgraded DBN version to 0.64.0: - Added a `finish()` method to `DynWriter` to flush data and finalize the output stream. For compressed data this will write the zstandard end-of-frame block - Upgraded `time` version to 0.3.54 #### Bug fixes - From DBN: - Fixed `AsRef<[u8]>` for concrete record types encoding only `size_of::()` bytes instead of the full `record_size()` from the header length, which dropped appended `ts_out` bytes and corrupted output when re-encoding a typed record decoded from a stream with `ts_out` ## 0.55.0 - 2026-07-14 #### Enhancements - Upgraded DBN version to 0.63.0: - Added an integer representation for `VersionUpgradePolicy` ## 0.54.0 - 2026-07-07 #### Enhancements - Upgraded DBN version to 0.62.0: - Added `RecordDecoder::seek_to` for absolute byte-offset seeking - Added `is_compressed` helpers for dynamic DBN readers - Added `MatchAlgorithm` variant `Allocation` - Added new publisher values for Cboe Titanium Cboe Global Indices Feed - Added `StatType` variants `MwcbLevel1`, `MwcbLevel2`, and `MwcbLevel3` - Added `StatType` variants `AuctionCollarReferencePrice`, `AuctionCollarUpperPrice`, and `AuctionCollarLowerPrice` - Upgraded `time` version to 0.3.53 #### Breaking changes - Breaking changes from DBN: - Renamed the following Venue, Dataset, and Publisher: - `CGIF` to `MAIN` - `CGIF.TITANIUM` to `MAIN.CGIF` - `CGIF.TITANIUM.CGIF` to `MAIN.CGIF.MAIN` - Changed the default `size` field to 0 in `MboMsg`, `TradeMsg`, `Mbp1Msg`, `Mbp10Msg`, `BboMsg`, `Cmbp1Msg`, and `CbboMsg` ## 0.53.0 - 2026-06-02 #### Enhancements - Added `BatchClient::get_job_details()` to access the full details of a batch job - Upgraded DBN version to 0.59.0: - Added `encode_record_ref_with_ts_out` and `encode_record_refs_with_ts_out` methods to the `EncodeRecordRef` and `AsyncEncodeRecordRef` traits. The DBN encoder uses vectored I/O to append the `ts_out` timestamp without copying the record body, while CSV and JSON encoders serialize from a cloned struct as usual - Added new venues, datasets, and publishers for US Equities Securities Information Processors ## 0.52.0 - 2026-05-26 #### Enhancements - Added logging for gateway `ErrorMsg` and `SystemMsg` records in the live `Client`. Suppress via the `tracing` subscriber's level filter ## 0.51.0 - 2026-05-12 #### Enhancements - Added `case` and `payload` fields to `ApiError` to expose the machine-readable error identifier and structured context (dataset, requested range, available range, etc.) returned by the historical API - Included the error `case` in `ApiError`'s `Display` output - Upgraded DBN version to 0.58.0: - Relaxed `TsSymbolMap::insert` to accept `impl Into>` for the `symbol` parameter ## 0.50.0 - 2026-05-05 #### Enhancements - Upgraded DBN version to 0.57.0: - Added `INDEX` `InstrumentClass` variant - Added `StatType` variants `INDICATIVE_CLOSE_PRICE` and `VENUE_SPECIFIC_PRICE_1` ## 0.49.0 - 2026-04-29 #### Enhancements - Added `SplitDuration::Year` for historical batch job submission - Upgraded DBN version to 0.56.0: - Added `DecodeRecordRef::decode_buf_iter()` for lazily iterating owned `RecordBuf`s via a standard `Iterator` - Added `AsyncDecodeRecordRef::decode_stream()` for lazily decoding owned `RecordBuf`s via a `futures::Stream` - Improved `DbnFsm` decode throughput by making `AlignedBuffer` shifts lazy, deferring the internal move to buffer refill boundaries - Added new publisher values for Cboe Titanium Cboe Global Indices Feed - Added support for splitting files by year to `TimeSplitter` ## 0.48.0 - 2026-04-21 #### Enhancements - Upgraded DBN version to 0.55.0: - Added new publisher values for OPRA MEMX MX2 Options and IEX Options - Improved `DbnFsm` decode throughput on current-version data and `AsIs` workloads #### Bug fixes - Removed unsound `Copy` and `Clone` implementations for `RecordRefMut` ## 0.47.0 - 2026-04-15 #### Enhancements - Replaced `typed-builder` dependency with `bon` for all parameter builder structs - Added `maybe_` prefixed setters for all `Option` fields on parameter builders, allowing callers to pass `Option` directly (e.g. `maybe_start(Some(datetime))` or `maybe_start(None)`) - Upgraded DBN version to 0.54.0: - Added `RecordBuf`, an owned stack-allocated buffer for holding a DBN record of dynamic type. Complements `RecordRef` (borrowed, dynamic) and concrete types (owned, static). Supports `get()`, `try_get()`, `set()`, `upgrade()`, and cross-capacity `PartialEq` - Added `RecordRefMut` for mutable non-owning references to DBN records - Added `RecordRef::to_owned()` and `RecordRefMut::to_owned()` for converting borrowed records to an owned `RecordBuf` - Added `Hash`, `PartialEq`, and `Eq` for `RecordRef`, including cross-type equality between `RecordBuf` and `RecordRef` - Changed `RecordHeader` to be `Copy` #### Breaking changes - Changed `use_snapshot()` setter on `Subscription` builder to require an explicit `bool` argument: `.use_snapshot()` becomes `.use_snapshot(true)` - Changed `map_symbols` field in `SubmitJobParams` from `bool` to `Option` - Changed `limit()` setter on `GetRangeParams`, `GetRangeToFileParams`, `GetQueryParams`, and `SubmitJobParams` builders to accept `NonZeroU64` instead of `Option`. Use `maybe_limit()` for the previous behavior of passing an `Option` directly #### Bug fixes - Removed `#[doc(hidden)]` from `Subscription::use_snapshot` ## 0.46.0 - 2026-04-07 #### Enhancements - Added `TimeoutConf` struct and `timeout_conf()` builder method for configuring connect and auth timeouts on the Live client (defaults to 10s and 30s) - Added `ConnectTimeout` and `AuthTimeout` error variants - Upgraded DBN version to 0.53.0: - `FlagSet` setter methods are now `const`, making it easier to define constants #### Breaking changes - Changed `split_duration` field in `SubmitJobParams` and `BatchJob` from `Option` to `SplitDuration`. Use `SplitDuration::None` instead of `None` #### Bug fixes - Added missing `None` variant to `SplitDuration` enum for explicitly requesting no time-based splitting ## 0.45.0 - 2026-03-31 #### Enhancements - Upgraded DBN version to 0.52.0 - Added `HeartbeatTimeout` error variant: `next_record` and `fill_buf` return `Error::HeartbeatTimeout` if no data is received for `heartbeat_interval` + 5 seconds (defaults to 35 seconds) - Changed `SlowReaderBehavior::Skip` to send "skip" instead of "drop" to the gateway ## 0.44.0 - 2026-03-17 #### Enhancements - Added support for `progress` field in `BatchJob` response ## 0.43.0 - 2026-03-04 #### Enhancements - Upgraded DBN version to 0.51.0: - Added logic to set `code` when upgrading version 1 `ErrorMsg` to newer versions - Upgraded `async-compression` dependency version to 0.4.41 - Upgraded `time` version to 0.3.47 - Fixed miri issue with potential for unaligned data when decoding DBN through `DbnFsm` ## 0.42.0 - 2026-02-24 #### Enhancements - Upgraded DBN version to 0.50.0: - Added `SkippedRecordsAfterSlowReading` to the `ErrorCode` enum for gateway errors originating from slow client catch-up - Upgraded reqwest version to 0.13 ## 0.41.0 - 2026-02-18 #### Enhancements - Added three new methods to the Live client for more control over when I/O occurs. Together they provide an alternative to `next_record()`: - `fill_buf()`: reads available data from the socket into the internal buffer without processing records - `try_next_record()`: returns the next record from the internal buffer without performing any I/O. Returns `Ok(None)` if no complete records are available in the buffer. - `is_closed()`: returns whether the connection has been closed - Added `SlowReaderBehavior` enum and `slow_reader_behavior()` setter on the live client builder to configure gateway behavior when client falls behind - Added `slow_reader_behavior()` getter to `LiveClient` - Upgraded DBN version to 0.49.0: - Added `encode_records` to `EncodeRecord` and `AsyncEncodeRecord` traits for more efficient batch encoding - Added `encode_record_refs` to `EncodeRecordRef` and `AsyncEncodeRecordRef` traits with more efficient vectored I/O implementation for DBN #### Breaking changes - From DBN: - Moved `encode_records` method from the `EncodeDbn` trait to `EncodeRecord` ## 0.40.0 - 2026-01-27 #### Enhancements - Added support for using compression in the live API: - Added `compression()` setter on the live client builder - Added `compression()` getter on the live client - A new `Unknown(String)` variant was added to the reference data enums `Country`, `Currency`, `Event`, and `EventSubType`. This allows for forward compatibility when new variants are added - Upgraded DBN version to 0.48.0: - Added initial support for splitting DBN files - Added new publisher for Blue Ocean ATS (`OCEA_MEMOIR_OCEA`) - Added `http_client_builder` method to the reference and historical client builders to allow more customization over the HTTP client used - Improved logging in Historical and Reference APIs - The following reference data enums now implement `Display`: - `Country` - `Currency` - `Event` - `EventSubType` - `Frequency` - `OutturnStyle` - `SecurityType` #### Breaking changes - Refactor `live::protocol` messages to implement a new `RawApiMsg` trait - Move optional session parameters into `SessionOptions` struct which implements `Default` - As part of adding the new variant to the four reference data enums: - `Country`, `Currency`, `Event`, and `EventSubType` are no longer `Copy` - The `Error` type of the `FromStr` implementation was changed to `Infallible`: any unknown string will be parsed to the `Unknown` variant - The `as_str` method on these enums was removed: use `as_ref` instead - `Event` has been marked non-exhaustive - `historical::ClientBuilder` and `reference::ClientBuilder` are no longer `Clone` #### Bug fixes - From DBN: - Fixed issue where `AsyncDynReader` instances created from `with_buffer()` would only decode the first frame of multi-frame Zstandard files ## 0.39.0 - 2026-01-20 #### Enhancements - Upgraded DBN version to 0.47.0: - Added new off-market publisher for Cboe Futures Exchange (`XCBF_PITCH_XOFF`) - Added new `StatType` variants to be used by `XCBF.PITCH` dataset: - `UpperPriceLimit` - `LowerPriceLimit` - `BlockVolume` - `VenueSpecificVolume1` - Upgraded `time` version to 0.3.45 - Improved documentation of heartbeat intervals ## 0.38.0 - 2025-12-16 #### Breaking changes - Changed `Error::BadArgument::param_name` from a `String` to a `&'static str` #### Bug fixes - Fixed panic on failed write during authentication ## 0.37.0 - 2025-12-09 #### Enhancements - Added optional `chrono` feature for converting `chrono` date and datetime types to `DateRange`, `DateTimeRange`, and `Subscription::start` (credit: William Nelson) - Upgraded DBN version to 0.45.0: - Added new venue, dataset, and publisher for Cboe Futures Exchange (`XCBF.PITCH`) #### Breaking changes - Changed the default to `true` for the `map_symbols` parameter in `SubmitJobParams::builder()` for JSON and CSV encodings - Moved `DateTimeLike` trait to top level #### Bug fixes - Fixed checksum verification for resumed downloads in `batch().download()`. Previously, it would erroneously fail ## 0.36.0 - 2025-11-19 #### Enhancements - Added support for using built-in range syntax for specifying date and datetime ranges in the historical API, e.g. `date!(2020-01-01)..=date!(2020-12-31)` - Changed `metadata().get_dataset_range()`, `metadata().list_schemas()`, and `metadata().list_unit_prices()` to accept any `impl AsRef` as the dataset parameter - Upgraded DBN version to 0.44.0: - Added logic to set `code` when upgrading version 1 `SystemMsg` to newer versions - Added `Dataset::publishers()` method to retrieve all `Publisher` values for a dataset #### Breaking changes - Updated the minimum supported `tokio` version to 1.41, which was released one year ago - Made reference enums `Country`, `Currency`, and `EventSubType` non-exhaustive - Removed variants from `Country` and `Currency` that were not present in any data - Added more variants to `EventSubType` ## 0.35.0 - 2025-10-22 #### Enhancements - Initial release with Reference API support - Upgraded DBN version to 0.43.0: - Added explicit `Unset` variant for `SystemCode` and `ErrorCode` - Added `Default` implementation for `SystemCode` and `ErrorCode` ## 0.34.1 - 2025-09-30 #### Enhancements - Upgraded DBN version to 0.42.0 #### Bug fixes - Fixed visibility of fields in `DateRange` and `DateTimeRange` ## 0.34.0 - 2025-09-23 #### Enhancements - Added batch download retry, resumption, and checksum verification - Changed setter for `batch::DownloadParams` to accept any `impl ToString` for `filename_to_download` #### Breaking changes - Changed `sha2` to a required dependency ## 0.33.1 - 2025-08-26 #### Enhancements - Upgraded DBN version to 0.41.0: - Added `interval` method to `RType` and `Schema` to get the duration for subsampled schemas like `Ohlcv1H` and `Cbbo1S` - Changed the default value for `channel_id` to be `u8::MAX` in `MboMsg` and `u16::MAX` elsewhere since 0 is a valid channel ID ## 0.33.0 - 2025-08-19 #### Enhancements - Upgraded DBN version to 0.40.0: - Added `DbnVersion` new type #### Breaking changes - Removed `bill_id` field from `BatchJob` struct - Breaking changes from DBN: - Marked `ErrorCode` and `SystemCode` non-exhaustive to allow adding future variants without a breaking change - Added `EndOfInterval` variant to `SystemCode` to notify when all OHLCV bars and subsampled BBO records have been published for a time interval ## 0.32.0 - 2025-08-12 #### Enhancements - Upgraded DBN version to 0.39.1: - Added `PUBLISHER_SPECIFIC` flag - Improved JSON encoding performance - Added a `Default` implementation for `SecurityUpdateAction` #### Breaking changes - Removed unused `Received` variant from `JobState` enum ## 0.31.0 - 2025-07-30 #### Enhancements - Changed `timeseries.get_range_to_file()` to return a concrete type instead of an impl trait type - Upgraded DBN version to 0.39.0: - Added `side()` and `unpaired_side()` methods to `ImbalanceMsg` that convert the fields of the same name to the `Side` enum - Added `pretty_auction_time` property in Python for `ImbalanceMsg` - Added `Default` implementation for `StatUpdateAction` - Added warnings to the floating-point getter methods' docstrings - Added `action` and `ts_in_delta` getters to `BboMsg` - Added `ts_recv` getter to `StatusMsg` - Added missing floating-point price getters to `InstrumentDefMsg` record types from all DBN versions - Added more floating-point price getters to `ImbalanceMsg` - Added floating-point price getter to `StatMsg` and `v1::StatMsg` #### Breaking changes - Breaking changes from DBN: - Changed `SystemMsg::code()` and `ErrorMsg::code()` methods to return a `Result` instead of an `Option` to be consistent with other enum conversion methods - Changed `auction_time` field in `ImbalanceMsg` to be formatted as a timestamp #### Bug fixes - Removed unused `S3` and `Disk` variants from `Delivery` enum ## 0.30.0 - 2025-07-22 #### Breaking changes - Upgraded DBN version to 0.38.0: - Renamed `Compression::ZStd` to `Zstd` for consistency ## 0.29.0 - 2025-07-15 #### Breaking changes - Upgraded DBN version to 0.37.0: - Renamed the following Venue, Dataset, and Publishers: - `XEER` to `XEEE` - `XEER.EOBI` to `XEEE.EOBI` - `XEER.EOBI.XEER` to `XEEE.EOBI.XEEE` - `XEER.EOBI.XOFF` to `XEEE.EOBI.XOFF` - Renamed `CMBP1` constant to `CMBP_1` for consistency with `MBP_1` and `MBP_10` #### Bug fixes - Removed `Intraday` variant from `DatasetCondition` enum ## 0.28.0 - 2025-07-01 #### Enhancements - Added operating system info to the user agent to aid troubleshooting - Standardized `client` info sent by live clients to match historical - Added methods to the client builders to extend the user agents with a custom string #### Deprecations - Deprecated `Historical::with_url()`: use the builder to override the base URL - Deprecated the `upgrade_policy` parameters for `timseries().get_range()` and `timeseries().get_range_to_file()`: use the `Historical` client parameter instead ## 0.27.1 - 2025-06-25 #### Enhancements - Added `range_by_schema` field to `DatasetRange` struct #### Bug fixes - Changed type of `last_modified_date` in `DatasetConditionDetail` to `Option` to support missing dates ## 0.27.0 - 2025-06-10 #### Enhancements - Made the buffer size used by the live client when reading from the TCP socket configurable through the `LiveBuilder::buffer_size()` method - Added support for using `rustls` without pulling in OpenSSL. `reqwest` with OpenSSL is still the default - Upgraded DBN version to 0.36.0: - Added support for width, fill, and padding when formatting `pretty::Ts` - Added support for sign, precision, width, fill, and padding when formatting `pretty::Px` - Optimized pretty formatting of prices and timestamps #### Breaking changes - Changed type of `split_duration` to `Option` to support setting no split duration - Breaking changes from DBN: - Moved core async decoding and encoding functionality to new traits to match the sync interface and present a standardized interface - Decoding: `AsyncDecodeRecordRef` and `AsyncDecodeRecord` - Encoding: `AsyncEncodeRecord`, `AsyncEncodeRecordRef`, and `AsyncEncodeRecordTextExt` #### Deprecations - Deprecated `LiveClient::connect` and `LiveClient::connect_with_addr` methods in favor of using the builder so additional optional parameters can be added without a breaking change #### Bug fixes - Fixed bug with deserializing `null` `split_duration` in historical `batch().list_jobs()` ## 0.26.2 - 2025-06-03 #### Enhancements - Improved performance of live client by removing redundant state - Upgraded DBN version to 0.35.1 #### Bug fixes - Fixed handling of `VersionUpgradePolicy` in `timeseries().get_range()` and `get_range_to_file()` - Bug fixes from DBN: - Fixed behavior where encoding metadata could lower the `version` - Changed `DbnFsm::data()` to exclude all processed data - Fixed `Metadata::upgrade()` behavior with `UpgradeToV2` ## 0.26.1 - 2025-05-30 #### Bug fixes - Fixed handling of `VersionUpgradePolicy` in live client - Fixed default upgrade policies to `UpgradeToV3` to match announcement for version 0.26.0 ## 0.26.0 - 2025-05-28 This version marks the release of DBN version 3 (DBNv3), which is the new default. API methods and decoders support decoding all versions of DBN, but now default to upgrading data to version 3. #### Enhancements - Added `From` conversion for `DateTimeRange` - Added `is_last` field to live subscription requests which will be used to improve the handling of split subscription requests - Upgraded DBN version to 0.35.0: - Version 1 and 2 structs can be converted to version 3 structs with the `From` trait - Implemented conversion from `RecordRef` to `IoSlice` for use with `Write::write_vectored` #### Breaking changes - Breaking changes from DBN: - Definition schema: - Updated `InstrumentDefMsg` with new `leg_` fields to support multi-leg strategy definitions. - Expanded `asset` to 11 bytes and `ASSET_CSTR_LEN` to match - Expanded `raw_instrument_id` to 64 bits to support more venues. Like other 64-bit integer fields, its value will now be quoted in JSON - Removed `trading_reference_date`, `trading_reference_price`, and `settl_price_type` fields which will be normalized in the statistics schema - Removed `md_security_trading_status` better served by the status schema - Statistics schema: - Updated `StatMsg` has an expanded 64-bit `quantity` field. Like other 64-bit integer fields, its value will now be quoted in JSON - The previous `StatMsg` has been moved to `v2::StatMsg` or `StatMsgV2` - Changed the default `VersionUpgradePolicy` to `UpgradeToV3` - Updated the minimum supported `tokio` version to 1.38, which was released one year ago ## 0.25.0 - 2025-05-13 #### Enhancements - Increased live subscription symbol chunking size - Upgraded DBN version to 0.34.0: - Added a `v3::StatMsg` record with an expanded 64-bit `quantity` field - Added `with_compression_level` methods to `DynWriter`, `AsyncDynWriter`, and `AsyncDynBufWriter` - Added `DBN_VERSION` constants to each version module: `v1`, `v2`, and `v3` - Added `UNDEF_STAT_QUANTITY` constants to each version module - Added statistics compatibility trait `StatRec` for generalizing across different versions of the statistics record - Added `AsRef<[u8]>` implementations for `RecordEnum` and `RecordRefEnum` - Added new off-market publishers for Eurex, and European Energy Exchange (EEX) #### Breaking changes - From DBN: - Made `Record` a subtrait of `AsRef<[u8]>` as all records should be convertible to bytes ## 0.24.0 - 2025-04-22 #### Enhancements - Upgraded DBN version to 0.33.0: - Added `SystemCode` and `ErrorCode` enums to indicate types of system and error messages - Added `code()` methods to `SystemMsg` and `ErrorMsg` to retrieve the enum value if one exists and equivalent properties in Python - Converting a `v1::SystemMsg` to a `v2::SystemMsg` now sets to `code` to the heartbeat value - Added `ASSET_CSTR_LEN` constants for the size of `asset` field in `InstrumentDefMsg` in different DBN versions - Added `encode_record_with_sym()` method to `AsyncJsonEncoder` which encodes a record along with its text symbol to match the sync encoder #### Breaking changes - Breaking changes from DBN: - Added `code` parameter to `SystemCode::new()` and `ErrorMsg::new()` - Updated the `rtype_dispatch` and `schema_dispatch` macro invocations to look more like function invocation - Increased the size of `asset` field in `v3::InstrumentDefMsg` from 7 to 11. The `InstrumentDefMsgV3` message size remains 520 bytes. ## 0.23.0 - 2025-04-15 #### Enhancements - Added `subscriptions` to `LiveClient` `Debug` implementation - Upgraded DBN version to 0.32.0: - Added `SystemCode` and `ErrorCode` enums to indicate types of system and error messages - Added `code()` methods to `SystemMsg` and `ErrorMsg` to retrieve the enum value if one exists and equivalent properties in Python - Converting a `v1::SystemMsg` to a `v2::SystemMsg` now sets to `code` to the heartbeat value - Added `Ord` and `PartialOrd` implementations for all enums and `FlagSet` to allow for use in ordered containers like `BTreeMap` - Added `decode_records()` method to `AsyncDbnDecoder` and `AsyncDbnRecordDecoder` which is similar to the sync decoder methods of the same name - Upgraded `pyo3` version to 0.24.1 - Upgraded `time` version to 0.3.41 #### Breaking changes - Added new `id` field to live `Subscription`, which will be used for improved error messages - Added new `id` parameter to `live::protocol::SubRequest::new()` method - Breaking changes from DBN: - Added `code` parameter to `SystemCode::new()` and `ErrorMsg::new()` - Updated the `rtype_dispatch` and `schema_dispatch` macro invocations to look more like function invocation - Removed deprecated `dataset` module. The top-level `Dataset` enum and its `const` `as_str()` method provide the same functionality for all datasets - Removed deprecated `SymbolIndex::get_for_rec_ref()` method ## 0.22.0 - 2025-04-01 #### Enhancements - Added an implementation `From` for `DateRange` and `DateTimeRange` to make it simpler to request a single full day's worth of data - Added conversions between `DateRange` and `DateTimeRange` - Added conversions from `timeseries::GetRangeParams`, `timeseries::GetRangeToFileParams`, and `dbn::Metadata` to `symbology::ResolveParams` - Upgraded DBN version to 0.30.0: - Added support for mapping symbols from instrument definitions to `PitSymbolMap` with a new `on_instrument_def()` method - Added instrument definition compatibility trait `InstrumentDefRec` for generalizing across different versions of the instrument definition record - Added `Ord` and `PartialOrd` implementations for all enums and `FlagSet` to allow for use in ordered containers like `BTreeMap` - Added `decode_records()` method to `AsyncDbnDecoder` and `AsyncDbnRecordDecoder` which is similar to the sync decoder methods of the same name - Removed deprecated `dataset` module. The top-level `Dataset` enum and its `const` `as_str()` method provide the same functionality for all datasets - Removed deprecated `SymbolIndex::get_for_rec_ref()` method ## 0.21.0 - 2025-03-18 #### Enhancements - Improved error when calling `LiveClient::next_record()` on an instance that hasn't been started - Improved error when calling `LiveClient::start()` on an instance that has already been started - Upgraded DBN version to 0.29.0: - Added new venues, datasets, and publishers for ICE Futures US, ICE Europe Financials products, Eurex, and European Energy Exchange (EEX) - Added new `SkipBytes` and `AsyncSkipBytes` traits which are a subset of the `Seek` and `AsyncSeek` traits respectively, only supporting seeking forward from the current position - Deprecated `AsyncRecordDecoder::get_mut()` and `AsyncDecoder::get_mut()` as modifying the inner reader after decoding any records could lead to a corrupted stream and decoding errors ## 0.20.0 - 2025-02-12 #### Enhancements - Added `LiveClient::reconnect()` and `LiveClient::resubscribe()` methods to make it easier to resume a live session after losing the connection to the live gateway - Added `subscriptions()` and `subscriptions_mut()` getters to `LiveClient` for getting all active subscriptions - Added `shutdown()` method to `live::protocol` to clean up the active session - Downgraded to tracing span level on `LiveClient::next_record()` to "debug" to reduce performance impact - Added `From<&[&str]>` and `From<[str; N]>` implementations for `Symbols` #### Breaking changes - Changed `LiveClient::close()` to take `&mut self` rather than an owned value to `self` now that clients can be reused through the `reconnect()` method - Changed `LiveClient::subscribe()` to take a `Subscription` parameter rather than a `&Subscription` because it will now store the `Subscription` struct internally - Upgraded DBN version to 0.28.0: - Added `CommoditySpot` `InstrumentClass` variant and made `InstrumentClass` non-exhaustive to allow future additions without breaking changes ## 0.19.0 - 2025-01-21 #### Enhancements - Upgraded DBN version to 0.27.0: - Updated enumerations for unreleased US equities datasets and publishers - Added new venue `EQUS` for consolidated US equities - Added new dataset `EQUS.MINI` and new publishers `EQUS.MINI.EQUS` and `XNYS.TRADES.EQUS` #### Bug fixes - Changed historical metadata methods with `symbols` parameter to use a `POST` request to allow for requesting supported maximum of 2000 symbols ## 0.18.0 - 2025-01-08 #### Enhancements - Upgraded DBN version to 0.26.0: - Added `v3` namespace in preparation for future DBN version 3 release. DBN version 2 remains the current and default version - Added `v3::InstrumentDefMsg` record with new fields to support normalizing multi-leg strategy definitions - Removal of statistics-schema related fields `trading_reference_price`, `trading_reference_date`, and `settl_price_type` - Removal of the status-schema related field `md_security_trading_status` - Added initial support for merging DBN: - Decoding streams: `MergeDecoder` and `MergeRecordDecoder` structs - Metadata: `MergeDecoder` struct and `Metadata::merge()` method - In the CLI: specify more than one input file to initiate a merge - Eliminated `unsafe` in `From` implementations for record structs from different versions ## 0.17.0 - 2024-12-17 #### Enhancements - Upgraded DBN version to 0.25.0: - Added `v1` and `v2` namespaces in DBN to allow unambiguously referring to the record types for a given DBN version regardless of whether the record type has changed - Changed `dataset()` method on `MetadataBuilder` to accept an `impl ToString` so now `Dataset` and `&str` can be passed directly - Changed async DBN decoding to return `Ok(None)` when an incomplete record remains in the stream. This matches the existing behavior of sync DBN decoding - Upgraded `thiserror` version to 2.0 #### Breaking changes - Removed deprecated `Packaging` enum and `packaging` field that's no longer supported by the API - As part of the DBN version upgrade: - `VersionUpgradePolicy::Upgrade` was renamed to `UpgradeToV2` - Changed async DBN decoding to return `Ok(None)` when an incomplete record remains in the stream ## 0.16.0 - 2024-11-12 #### Enhancements - Upgraded DBN version to 0.23.1: - Added floating-point getters for price fields - Added new IntelligentCross venues `ASPN`, `ASMT`, and `ASPI` - Upgraded `thiserror` version to 2.0 #### Deprecations - Deprecated `Packaging` enum and `packaging` field on `SubmitJobParams` and `BatchJob`. These will be removed in a future version. All files from a batch job can be downloaded with the `batch().download()` method on the historical client ## 0.15.0 - 2024-10-22 #### Enhancements - Upgraded DBN version to 0.23.0: - Added new `None` `Action` variant that will be gradually rolled out to historical and live `GLBX.MDP3` data - Added consistent escaping of non-printable and non-ASCII values when text encoding `c_char` fields - Implemented `Default` for `Action` and `Side` - Implemented missing `Serialize` for (with `serde` feature enabled) for `Venue`, `Dataset`, `Publisher`, `Compression`, `SType`, `Schema`, and `Encoding` ## 0.14.1 - 2024-10-08 #### Enhancements - Upgraded DBN version to 0.22.1: - Fixed buffer overrun in `c_chars_to_str` on non-null terminated input - Combined `_reserved3` and `reserved4` fields in `CbboMsg` ## 0.14.0 - 2024-10-01 #### Enhancements - Made several previously internal functions public to allow advanced users more customization and piecemeal usage of the live API: - `ApiKey` - `Symbols::to_chunked_api_string()` - `live::protocol` module containing implementations of the raw API messages - Changed from `log` crate to `tracing` for better diagnostics ## 0.13.0 - 2024-09-25 #### Deprecations - Deprecated `Packaging::Tar`. Users should switch to `Packaging::Zip`. This variant will be removed in a future version when it is no longer supported by the API ## 0.12.1 - 2024-08-27 #### Enhancements - Added `Intraday` variant to `DatasetCondition` in preparation for intraday data being available from the historical API - Upgraded DBN version to 0.20.1 for new publisher values for `XCIS.BBOTRADES` and `XNYS.BBOTRADES` ## 0.12.0 - 2024-07-30 #### Breaking changes - Upgraded DBN version to 0.20.0: - Renamed `SType::Nasdaq` variant to `SType::NasdaqSymbol` - Renamed `SType::Cms` variant to `SType::CmsSymbol` ## 0.11.4 - 2024-07-16 #### Enhancements - Upgraded DBN version to 0.19.1 with fixes for `BBOMsg` record struct ## 0.11.3 - 2024-07-09 #### Enhancements - Upgraded DBN version to 0.19.0 with new `BBOMsg` record struct ## 0.11.2 - 2024-06-25 #### Enhancements - Added `historical::timeseries::get_range_to_file` method to persist the data stream to a given path before returning an `AsyncDbnDecoder` ## 0.11.1 - 2024-06-11 #### Enhancements - Added getter for `heartbeat_interval` to `LiveClient` #### Bug fixes - Fixed potential incorrect DNS resolution when overriding the live gateway address with `live::Builder::addr` ## 0.11.0 - 2024-06-04 #### Enhancements - Added configurable `heartbeat_interval` parameter for live client that determines the timeout before heartbeat `SystemMsg` records will be sent. It can be configured via the `heartbeat_interval` and `heartbeat_interval_s` methods of the `live::ClientBuilder` - Added `addr` function to `live::ClientBuilder` for configuring a custom gateway address without using `LiveClient::connect_with_addr` directly - Upgraded DBN version to 0.18.1 #### Breaking changes - Added `heartbeat_interval` parameter to `LiveClient::connect` and `LiveClient::connect_with_addr` - Removed deprecated `start_date` and `end_date` fields from `DatasetRange` struct ## 0.10.0 - 2024-05-22 #### Enhancements - Added `use_snapshot` attribute to `Subscription`, defaults to false - Upgraded reqwest version to 0.12 #### Breaking changes - Upgraded DBN version to 0.18.0 - Changed type of `flags` in `MboMsg`, `TradeMsg`, `Mbp1Msg`, `Mbp10Msg`, and `CbboMsg` from `u8` to a new `FlagSet` type with predicate methods for the various bit flags as well as setters. The `u8` value can still be obtained by calling the `raw()` method - Improved `Debug` formatting - Switched `DecodeStream` from `streaming_iterator` crate to `fallible_streaming_iterator` to allow better notification of errors - Changed default value for `stype_in` and `stype_out` in `SymbolMappingMsg` to `u8::MAX` to match C++ client and to reflect an unknown value. This also changes the value of these fields when upgrading a `SymbolMappingMsgV1` to DBNv2 ## 0.9.1 - 2024-05-15 #### Bug fixes - Fixed build when only `live` feature is enabled ## 0.9.0 - 2024-05-14 #### Enhancements - Added `start` and `end` fields to the `DatasetRange` struct which are instances of `OffsetDateTime` and provide time resolution and an exclusive end - Upgraded DBN version to 0.17.1 #### Deprecations - The `start_date` and `end_date` fields of the `DatasetRange` struct are deprecated and will be removed in a future release ## 0.8.0 - 2024-04-01 #### Enhancements - Upgraded DBN version to 0.17.0 - Added new record types and schema variants for consolidated BBO and subsampled BBO - Added `Volatility` and `Delta` `StatType` variants #### Breaking changes - Removed previously-deprecated `live::SymbolMap`. Please use `databento::dbn::PitSymbolMap` instead ## 0.7.1 - 2024-03-05 #### Enhancements - Improve error handling when a historical HTTP error response is not in the expected JSON format ## 0.7.0 - 2024-03-01 #### Enhancements - Documented cancellation safety of `LiveClient` methods - Documented `live::Subscription::start` is based on `ts_event` - Allow constructing a `DateRange` and `DateTimeRange` with an `end` based on a `time::Duration` - Implemented `Debug` for `LiveClient`, `live::ClientBuilder`, `HistoricalClient`, `historical::ClientBuilder`, `BatchClient`, `MetadataClient`, `SymbologyClient`, and `TimeseriesClient` - Derived `Clone` for `live::ClientBuilder` and `historical::ClientBuilder` - Added `ApiKey` type for safely deriving `Debug` for types containing an API key #### Breaking changes - Changed default `upgrade_policy` in `LiveBuilder` and `GetRangeParams` to `Upgrade` so by default the primary record types can always be used - Simplified `DateRange` and `DateTimeRange` by removing `FwdFill` variant that didn't work correctly - Upgraded DBN version to 0.16.0 - Updated `StatusMsg` in preparation for status schema release - Fixed handling of `ts_out` when upgrading DBNv1 records to version 2 - Fixed handling of `ErrorMsgV1` and `SystemMsgV1` in `rtype` dispatch macros ## 0.6.0 - 2024-01-16 #### Enhancements - Relaxed version requirements for `tokio`, `tokio-util`, and `thiserror` #### Breaking changes - Upgraded DBN version to 0.15.0 - Added support for larger `SystemMsg` and `ErrorMsg` records - Improved `Debug` implementations for records and `RecordRef` - Improved panic messages for `RecordRef::get` - Upgraded `typed-builder` to 0.18 #### Bug fixes - Fixed documentation for `end` in `DateRange::Closed` and `DateTimeRange::Closed` ## 0.5.0 - 2023-11-23 This release adds support for DBN v2. DBN v2 delivers improvements to the `Metadata` header symbology, new `stype_in` and `stype_out` fields for `SymbolMappingMsg`, and extends the symbol field length for `SymbolMappingMsg` and `InstrumentDefMsg`. The entire change notes are available [here](https://github.com/databento/dbn/releases/tag/v0.14.0). Users who wish to convert DBN v1 files to v2 can use the `dbn-cli` tool available in the [databento-dbn](https://github.com/databento/dbn/) crate. On a future date, the Databento live and historical APIs will stop serving DBN v1. This release is fully compatible with both DBN v1 and v2, and so should be seamless for most users. #### Enhancements - Made `LiveClient::next_record`, `dbn::decode::AsyncDbnDecoder::decode_record` and `decode_record_ref`, and `dbn::decode::AsyncRecordDecoder::decode` and `decode_ref` cancel safe. This makes them safe to use within a [`tokio::select!`](https://docs.rs/tokio/latest/tokio/macro.select.html) statement - Improved error reporting for `HistoricalClient` when receiving an error from Databento's API - Improved error messages around API keys - Improved performance of CSV and JSON encoding - Added support for emitting warnings from historical API response headers, such as for future deprecations - Added `symbol_map` method to the `Resolution` struct returned by `symbology::resolve` that returns a `TsSymbolMap` - Added `PartialEq` and `Eq` implementations for parameter builder classes - Added `upgrade_policy` setter to the `LiveClient` builder and a getter to the `LiveClient` - Added `upgrade_policy` optional setter to the `timeseries::GetRangeParams` builder #### Breaking changes - Upgraded `dbn` to 0.14.2. There are several breaking changes in this release as we begin migrating to DBN encoding version 2 (DBNv2) in order to support the ICE exchange: - Renamed `dbn::InstrumentDefMsg` to `dbn::compat::InstrumentDefMsgV1` and added a new `dbn::InstrumentDefMsg` with a longer `raw_symbol` field - Renamed `dbn::SymbolMappingMsg` to `dbn::compat::SymbolMappingMsgV1` and added a new `dbn::SymbolMappingMsg` with longer symbol fields and new `stype_in` and `stype_out` fields - Added `symbol_cstr_len` field to `dbn::Metadata` - Made `Error` non-exhaustive, meaning it no longer be exhaustively matched against, and new variants can be added in the future without a breaking change - Added an `upgrade_policy` parameter to `LiveClient::connect` and `connect_with_addr`. The builder provides a more stable API since new parameters are usually introduced as optional #### Deprecations - Deprecated `live::SymbolMap` in favor of `databento::dbn::PitSymbolMap` ## 0.4.2 - 2023-10-23 #### Enhancements - Upgraded `dbn` to 0.13.0 for improvements to symbology helpers - Upgraded `tokio` to 1.33 - Upgraded `typed-builder` to 0.17 #### Bug fixes - Fixed panic in `LiveClient` when gateway returned an auth response without the `success` key ## 0.4.1 - 2023-10-06 #### Enhancements - Added support for changing datetime format used in batch job responses - Upgraded `dbn` to 0.11.1 ## 0.4.0 - 2023-09-21 #### Enhancements - Added `pretty_px` option for `batch::submit_job`, which formats prices to the correct scale using the fixed-precision scalar 1e-9 (available for CSV and JSON text encodings) - Added `pretty_ts` option for `batch::submit_job`, which formats timestamps as ISO 8601 strings (available for CSV and JSON text encodings) - Added `map_symbols` option to `batch::submit_job`, which appends the raw symbol to every record (available for CSV and JSON text encodings) reducing the need to look at the `symbology.json` file - Added `split_symbols` option for `batch::submit_job`, which will split files by raw symbol - Added `encoding` option to `batch::submit_job` to allow requesting non-DBN encoded data through the client - Added `map_symbols`, `pretty_px`, and `pretty_ts` to `BatchJob` response ## 0.3.0 - 2023-09-13 #### Enhancements - Added `SymbolMap` type to help maintain up-to-date symbol mappings with live data - Added chunking to handle subscribing to many symbols for the Live client - Upgraded DBN version to 0.10.2 for easier historical symbology ## 0.2.1 - 2023-08-25 #### Enhancements - Upgraded DBN version to 0.9.0 for publisher improvements to support OPRA ## 0.2.0 - 2023-08-10 #### Breaking changes - Changed `metadata::list_publishers` to return a `Vec` - `metadata::list_fields`: - Changed return type to `Vec` - Made `encoding` and `schema` parameters required - Removed `dataset` parameter - `metadata::list_unit_prices`: - Changed return type to `Vec` - Made `dataset` parameter required - Removed `mode` and `schema` parameters ## 0.1.0 - 2023-08-02 - Initial release with support for historical and live data --- Source: https://databento.com/docs/release-notes/release-notes-data # Data ## 2026-10-13 Fixed an issue where NDEX.IMPACT had no options data through 2023-03-17. ## 2026-09-15 The latest regeneration of CME Globex data prior to May 2017 includes several data quality fixes. #### Bug fixes - Fixed [missing final settlement prices on 2016-03-18](https://issues.databento.com/b/6vrl98vl/feature-ideas/cme-globex-mdp2-missing-final-settlement-prices-on-2016-03-18) by normalizing settlement prices from MDP2 security definitions into the statistics schema - Fixed [CME Globex MDP2 requests for part of a UTC date returning truncated or empty data](https://issues.databento.com/b/6vrl98vl/feature-ideas/cme-globex-mdp2-requests-for-part-of-a-utc-date-return-truncated-or-empty) - Fixed [incomplete OHLCV bars in CME Globex MDP2 data](https://issues.databento.com/b/6vrl98vl/feature-ideas/cme-globex-mdp2-data-has-incomplete-bars-for-many-days) where most of a day consolidated into a single bar - Fixed `min_price_increment_amount` in MDP2 definitions, which incorrectly held the `min_price_increment` value ## 2026-08-08 #### Enhancements - Upgraded GLBX.MDP3 to DBN version 3: - [Definition schema now includes one record per strategy leg](https://roadmap.databento.com/b/n0o5prm6/feature-ideas/expose-metadata-of-every-underlying-leg-in-multi-leg-futures-and-options) - Added `N` (None) `action`, which can carry `F_LAST` flag - Settlement prices from security definitions move to the statistics schema - Price limits are added to the statistics schema #### Bug fixes - GLBX.MDP3 fixes: - Fixed [issue where legacy CME data is missing `side` for trades](https://issues.databento.com/roadmap/legacy-cme-fixfast-mdp2-data-is-missing-trade-side) - Fixed [incorrect mapping of instrument security trading status for Pre-Cross](https://issues.databento.com/b/6vrl98vl/feature-ideas/glbxmdp3-incorrect-mapping-of-instrument-security-trading-status-for-pre-cross) - Fixed issue where publication of a Status record could cause MBO records to incorrectly have `F_LAST` set, and also trigger a premature MBP record before the end of the CME event ## 2026-07-19 #### Bug fixes - Fixed [issue with `ts_recv` jitter in `EQUS.MINI` where timestamps could be non-monotonic](https://issues.databento.com/roadmap/equsmini-live-data-occasional-out-of-order-ts-recv-within-the-same-instrument) This fix will be propagated to already-generated historical data at a later date. ## 2026-05-31 #### Bug fixes - Fixed [issue with incorrect off-market publisher attribution for trades](https://issues.databento.com/b/6vrl98vl/feature-ideas/ifeuifusifllndeximpact-some-off-market-trades-marked-as-on-exchange) for live and historical data starting on 2026-05-31. - Fixed [issue with truncated contract symbols](https://issues.databento.com/b/6vrl98vl/feature-ideas/fix-contractsymbolextra-normalization-for-ice-option-strategies) for live and historical data starting on 2026-05-31 These fixes will be propagated to already-generated historical data at a later date. ## 2026-05-09 #### Enhancements - Added channel 333 (Event Contracts II) to `GLBX.MDP3` data - Moved the weekly gateway restart from Sunday 09:30 UTC to Saturday 02:15 CT ## 2026-04-22 #### Bug fixes - Fixed `instrument_id` inconsistency in Blue Ocean ATS historical data from 2026-03-22 through 2026-04-21 ## 2026-04-15 This release introduces live and historical CFE (`XCBF.PITCH`) data. CFE is the exclusive venue for futures on the Cboe Volatility Index (VIX), a widely used benchmark for expected U.S. equity market volatility. Its coverage also extends to corporate bond index, equity index and cryptocurrency contracts. You can read more about the release [here](https://databento.com/blog/introducing-cboe-futures-exchange-cfe). ## 2026-04-13 #### Bug fixes - Fixed Blue Ocean ATS historical data from 2026-02-10 through 2026-02-13 that was degraded by a packet handling bug following a network gap affecting 2026-02-08 through 2026-02-09. Data from 2026-02-08 through 2026-02-09 remains degraded. ## 2026-04-08 This release introduces live and historical Blue Ocean ATS MEMOIR Depth (`OCEA.MEMOIR`) data. Blue Ocean ATS is a leading venue for after-hours US equities trading, supporting 24-hour access to US equity markets. You can read more about the release [here](https://databento.com/blog/blue-ocean-ats-now-available). #### Enhancements - Added support for the `OCEA.MEMOIR` dataset ## 2025-11-09 #### Bug fixes - Fixed default values for new imbalance data to match other schemas - Fixed `auction_time` to be in UTC for NYSE datasets ## 2025-11-04 This release introduces live and historical Eurex EOBI (`XEUR.EOBI`) data. Eurex is Europe's largest derivatives exchange and a major trading venue for equity index and fixed income derivatives, as well as single stock options. You can read more about the release [here](https://databento.com/blog/eurex-now-available). #### Enhancements - Added support for the `XEUR.EOBI` dataset ## 2025-09-23 #### Bug fixes - Fixed [missing `ts_ref` in CME Globex settlement price statistics prior to 2015](https://issues.databento.com/roadmap/mdp2-statistics-missing-ts-ref-for-settlement-prices) - Fixed a symbology issue in the `XCIS.TRADESBBO` dataset starting on 2025-02-21. ## 2025-08-26 #### Enhancements - Added preliminary end-of-day statistics to EQUS.SUMMARY. More details available on the [dataset supplement](https://databento.com/docs/datasets/equs-summary.md#end-of-day-summary). ## 2025-08-05 #### Enhancements - Added historical data for delisted market types in ICE datasets, including [EUA futures when they traded on IFEU prior to 2021](https://roadmap.databento.com/roadmap/extend-coverage-for-ice-eua-futures) #### Bug fixes - Fixed [missing options book updates for ICE datasets](https://issues.databento.com/b/6vrl98vl/feature-ideas/ifeuifllifusndeximpact-book-updates-missing-for-options) ## 2025-07-25 The latest regeneration of CME Globex data prior to May 2017 includes several data quality fixes and improvements. #### Enhancements - Added [`status` schema data for CME Globex prior to May 2017](https://issues.databento.com/b/6vrl98vl/feature-ideas/glbxmdp3-status-schema-unavilabile-for-extended-mdp2-history) #### Bug fixes - Fixed [incorrect `activation` and `expiration` from 2015-11-22 to 2017-05-21](https://issues.databento.com/b/6vrl98vl/feature-ideas/glbxmdp3-incorrect-activationexpiration-from-2015-11-22-to-2017-05-21) - Fixed [incorrect price levels in the MBP-10 schema for products with reduced `market_depth` and implied books on MDP2 CME Globex data](https://issues.databento.com/b/6vrl98vl/feature-ideas/incorrect-price-levels-on-mbp-10-for-products-with-reduced-market-depth-or-implied-books-on-mdp2-cme-data) - Fixed [missing `ts_ref` in CME Globex data from 2015-11-20 to 2017-05-21](https://issues.databento.com/b/6vrl98vl/feature-ideas/missing-ts-ref-in-cme-mdp2-after-2015-11-20) - Fixed [empty CME Globex data for 2010-09-13, 2010-11-17, 2012-07-19, and 2017-03-13](https://issues.databento.com/b/6vrl98vl/feature-ideas/glbxmdp3-empty-data-for-2010-09-13-2010-11-17-2012-07-19-and-2017-03-13) - Fixed [out-of-order statistics records in pre-2017 CME Globex data](https://issues.databento.com/b/6vrl98vl/feature-ideas/records-with-unordered-ts-recv-in-pre-2017-glbxmdp3-data) - Fixed [missing `SettlPriceType` normalization in MDP2 CME Globex data](https://issues.databento.com/b/6vrl98vl/feature-ideas/missing-settlpricetype-normalization-for-mdp2-cme-data) ## 2025-07-06 #### Bug fixes - Fixed [missing symbol mappings in live EQUS.MINI when subscribing between 09:30Z on Sunday and 06:00Z on Monday](https://issues.databento.com/b/6vrl98vl/feature-ideas/equsmini-live-data-symbol-mappings-are-not-relayed-to-the-client-when-starting-the-connection-between-930z-on-sunday-and-0600z-on-monday) ## 2025-07-01 #### Bug fixes - Fixed [incomplete ICE Futures US data prior to 2023-03-19](https://issues.databento.com/b/6vrl98vl/feature-ideas/incomplete-ice-futures-us-prior-to-2023-03-19) - Fixed [broken symbology for `BRN` futures and options in ICE Europe Commodities data on 2024-01-01 and 2024-01-02](https://issues.databento.com/b/6vrl98vl/feature-ideas/broken-symbology-for-some-brn-futures-and-options-on-2024-01-01-and-2024-01-02) ## 2025-06-27 #### Bug fixes - Fixed [misaligned and incorrect values for some instruments in CME Globex from 2012-02-05 through 2012-02-10](https://issues.databento.com/b/6vrl98vl/feature-ideas/instrument-definition-issues-on-glbxmdp3-from-2012-02-05-to-2012-02-10) ## 2025-06-17 This release introduces live and historical ICE Futures US iMpact (`IFUS.IMPACT`) and ICE Europe Financials iMpact (`IFLL.IMPACT`) data. The datasets cover US futures and options for all softs, metals, and financial products, as well as European equity and interest rate derivatives. You can read more about the release [here](https://databento.com/blog/introducing-ice-futures-us). #### Enhancements - Changed the following Venue, Publisher, and Dataset descriptions: - "ICE Futures Europe (Financials)" renamed to "ICE Europe Financials" - "ICE Futures Europe (Commodities)" renamed to "ICE Europe Commodities" ## 2025-06-10 #### Enhancements - Extended [OPRA Pillar history back to 2013](https://roadmap.databento.com/b/n0o5prm6/feature-ideas/extend-opra-dataset-history). You can read more about the release [here](https://databento.com/blog/opra-improvements-coming-soon). ## 2025-05-20 #### Bug fixes - Fixed [degraded dates in CME Globex prior to 2024-12-09 could have stuck orders](https://issues.databento.com/b/6vrl98vl/feature-ideas/glbxmdp3-degraded-dates-prior-to-2024-12-09-can-have-stuck-orders) - Fixed [incorrect `underlying` for options created mid-week in CME Globex prior to 2017-05-21](https://issues.databento.com/b/6vrl98vl/feature-ideas/glbxmdp3-options-created-mid-week-have-incorrect-underlying-in-pre-2017-glbxmdp3-data) ## 2025-05-07 #### Enhancements - Extended US equities coverage to Cboe EDGX Depth, Cboe EDGA Depth, Cboe BZX Depth, and Cboe BYX Depth. You can read more about the release [here](https://databento.com/blog/cboe-us-equities-depth-now-available). - Added [normalization for the official opening and closing cross prices in Nasdaq data](https://roadmap.databento.com/b/n0o5prm6/feature-ideas/official-opening-and-closing-cross-prices) ## 2025-04-05 #### Enhancements - Added support for [open interest and volume-based continuous symbology in the live API](https://roadmap.databento.com/b/n0o5prm6/feature-ideas/support-full-continuous-symbology-for-live-data) ## 2025-04-01 #### Bug fixes - Fixed [incorrect strike prices for some options in CME Globex definitions prior to 2017-05-21](https://issues.databento.com/b/6vrl98vl/feature-ideas/incorrect-strike-prices-for-some-options-on-cme-mdp2-data) ## 2025-03-13 #### Enhancements - Extended US equities coverage to MEMX MEMOIR Depth. You can read more about the release [here](https://databento.com/blog/memx-memoir-depth-now-available). ## 2025-02-26 #### Bug fixes - Regenerated [degraded EPRL.DOM historical data from 2025-02-03 through 2025-02-19 (inclusive)](https://issues.databento.com/b/6vrl98vl/feature-ideas/eprldom-historical-data-degraded-from-2025-02-03-to-2025-02-19-inclusive) ## 2025-02-01 #### Enhancements - Added the Databento US Equities Mini dataset, which provides cost-effective real-time US equities data. You can read more about this release [here](https://databento.com/blog/databento-us-equities-mini-now-available). ## 2025-01-15 #### Breaking changes - Rolled out [changes to live connection limits](https://databento.com/blog/changes-to-live-connection-limits) ## 2024-12-14 #### Bug fixes - Fixed [degraded CME Globex data for 2024-11-08](https://issues.databento.com/b/6vrl98vl/feature-ideas/glbxmdp3-historical-data-is-degraded-for-2024-11-08) ## 2024-12-03 #### Enhancements - Extended [CME Globex history back to 2010](https://databento.com/blog/CME-history-extended-to-2010) ## 2024-12-02 This release features several normalization and data quality improvements to ICE Europe Commodities and ICE Endex. #### Enhancements - Improved ICE normalization to use `ContractSymbol` for `raw_symbol` when available. `ContractSymbol` provides more human-readable symbols than the `StrategySymbol` fallback. - Updated ICE normalization to normalize off-market trades under a separate publisher #### Bug fixes - Fixed [an issue](https://issues.databento.com/b/6vrl98vl/feature-ideas/ifeundex-system-priced-leg-trades-are-being-counted-towards-ohlcv-bars) with ICE Europe Commodities and ICE Endex where System Priced leg trades were counted towards OHLCV bars ## 2024-10-22 #### Bug fixes - Fixed [an issue](https://issues.databento.com/b/6vrl98vl/feature-ideas/incomplete-and-missing-data-on-some-fridays-in-2020-in-ice-endex-and-ice-futures-europe) where several Fridays in 2020 had no data for ICE Europe Commodities and ICE Endex ## 2024-10-24 #### Enhancements - Added adjustment factors to our reference data API. You can read about this release [here](https://databento.com/blog/adjustment-factors). ## 2024-07-05 #### Enhancements - Added an MBP snapshot record at 00:00 UTC for historical data. This is generated along with the existing MBO snapshot and will have the `F_SNAPSHOT` flag set. The MBP snapshot will only be sent for instruments that actually have a quote. ## 2024-06-25 #### Enhancements - Added the `status` schema for the `GLBX.MDP3` dataset ## 2024-06-18 #### Bug fixes - Changed normalization for CME Globex MDP 3.0 statistics to use tag 34-MsgSeqNum for `sequence` to match other schemas instead of tag 83-RptSeq - Fixed scaling of `strike_price` for metal options for CME Globex MDP 3.0 - Added [truncated digit](https://www.cmegroup.com/confluence/display/EPICSANDBOX/Options+with+Truncated+Strike+Prices) of `strike_price` for MDP 3.0 2-Year T-Note options (OZT) for CME Globex MDP 3.0. `strike_price` now includes the full strike price - Fixed scaling of `unit_of_measure_qty` in older CME Globex MDP 3.0. Previously, the newer 9-digit precision scaling was erroneously applied to pre-late 2018 data when CME used 7-digit precision - Changed `stat_flags` to always be 0 when it is unused or there is no source field to normalize to it ## 2024-05-07 #### Bug fixes - Changed normalization for ICE Europe Commodities iMpact and ICE Endex iMpact to no longer set `ts_ref` when a valuation date time or trading date isn't specified - Added missing normalization to statistics schema for Option Open Interest Message and Option Settlement Price Message in ICE Europe Commodities iMpact and ICE Endex iMpact ## 2024-01-18 This release introduces live and historical ICE Europe Commodities iMpact (`IFEU.IMPACT`) and ICE Endex iMpact (`NDEX.IMPACT`) data. The datasets cover European futures and options for all energy and commodities products, including all listed outrights, spreads, and options combinations. You can read more about the release [here](https://databento.com/blog/ice-futures-market-data). #### Enhancements - Added the `IFEU.IMPACT` and `NDEX.IMPACT` datasets, does not yet include our `mbo`, `mbp-10`, `mbp-1`, and `tbbo` schemas ## 2023-11-17 #### Enhancements This release adds support for Nasdaq TotalView-ITCH (`XNAS.ITCH`) live data. This includes detailed information on every order in the book, captured and distributed from our Equinix NY4 colo, offering more insights into Nasdaq market activity than data sourced from the SIPs. You can read more about this release [here](https://databento.com/blog/nasdaq-totalview-live). ## 2023-10-04 This release introduces the Databento Equities Basic dataset (`DBEQ.BASIC`), our first bundle of live and historical US equities data. The bundle includes five proprietary feeds--NYSE Chicago Integrated, IEX TOPS, NYSE National Trades, NYSE National BBO, and MIAX Pearl Equities Depth of Market--that can be licensed for distribution, display, and non-display applications at no cost. You can read more about the release [here](https://databento.com/blog/dbeq-basic). #### Enhancements - Added the `DBEQ.BASIC` dataset, does not include live data for the `mbo` schema #### Breaking changes - Changed `XNAS.ITCH` Cross Trade messages when no cross occurred to have a price of `UNDEF_PRICE` instead of the previous price of 0 ## 2023-08-29 This release introduces the OPRA (Options Price Reporting Authority) dataset (`OPRA.PILLAR`) for the historical and live APIs, covering all 16 national exchanges. This not only includes options on single name equities, but also on indices, e.g., Cboe VIX options, SPX options, and certain volatility benchmark indices like SPIKE and VSPKE. You can read more about the release [here](https://databento.com/blog/opra-data). #### Enhancements - Added the `OPRA.PILLAR` dataset, does not yet include live data for the `mbp-1` schema, or Global Trading Hours (GTH) ## 2023-07-23 - Updated CSV encoding: - Reordered columns in every schema to better reflect relative importance - Added `channel_id` to MBO schema - Always include `rtype` to differentiate between schemas - Rename `security_group` in definitions schema to `group` to match struct definitions - Updated JSON encoding: - Moved header fields (`rtype`, `publisher_id`, `instrument_id`, and `ts_event`) to - Added `channel_id` to MBO schema - Changed to always include `rtype` to differentiate between schemas nested object under the key `hd` in JSON encoding to match structure definitions - Rename `security_group` in definitions schema to `group` to match struct definitions - All 64-bit integers are now serialized as strings to avoid loss of precision ## 2023-05-01 - Renamed `product_id` to `instrument_id` in common header - Renamed `symbol` to `raw_symbol` in definitions schema ## 2023-04-28 - Added `strike_price` field to definitions schema - Added `strike_price_currency` field to definitions schema - Added `instrument_class` field to definitions schema - Deprecated `related` and `related_security_id` from definitions schema ## 2023-03-07 - Fixed an [issue](https://issues.databento.com/roadmap/duplicate-order-ids-at-start-of-day-affecting-cme-globex-mdp-30) causing duplicate add order snapshots on CME Globex MDP 3.0. --- Source: https://databento.com/docs/schemas # Schemas and data formats Learn about the schemas (data formats) supported by Databento and get a list of fields (data dictionary) for each. | Title | Subtitle | Image | Supported languages | | [What's a schema?](https://databento.com/docs/schemas/whats-a-schema.md) | Compare Databento's 15 schemas, e.g., MBO, MBP, TBBO, definitions, and more. | ![schemas image](https://databento.com/docs/what-is-a-schema.svg.md) | Terminology, Data formats | | [Market by order (MBO)](https://databento.com/docs/schemas/mbo.md) | Full order book data, including all buy and sell orders at every price level. | ![Market by order bar chart image](https://databento.com/docs/mbo.svg.md) | L3, List of fields, Data formats | | [Imbalance](https://databento.com/docs/schemas/imbalance.md) | Auction imbalance data like paired quantity, total quantity, and auction status. | ![Imbalance line graph image](https://databento.com/docs/imbalance.svg.md) | L3, List of fields, Data formats | | [Market by price (MBP-10)](https://databento.com/docs/schemas/mbp-10.md) | Aggregated book depth with every trade and book update at top ten price levels. | ![Market by price bar chart image](https://databento.com/docs/mbp-10.svg.md) | L2, List of fields, Data formats | | [Market by price (MBP-1)](https://databento.com/docs/schemas/mbp-1.md) | Aggregated book depth with every trade and book update at BBO. | ![Market by price bar chart image](https://databento.com/docs/mbp-1.svg.md) | L1, List of fields, Data formats | | [BBO on trade (TBBO)](https://databento.com/docs/schemas/tbbo.md) | Every trade event alongside the BBO immediately before each trade. | ![Trades with best bid offer image](https://databento.com/docs/tbbo.svg.md) | L1, List of fields, Data formats | | [BBO on interval (BBO)](https://databento.com/docs/schemas/bbo.md) | Last best bid, best offer, and sale on 1-second or 1-minute intervals. | ![Best bid and offer image](https://databento.com/docs/bbo.svg.md) | L1, List of fields, Data formats | | [Trades](https://databento.com/docs/schemas/trades.md) | Every trade event, tick-by-tick. | ![Trades table image](https://databento.com/docs/trades.svg.md) | L1, List of fields, Data formats | | [Aggregate bars (OHLCV)](https://databento.com/docs/schemas/ohlcv.md) | Open, high, low, close prices with volume in second, minute, hour, or daily intervals. | ![OHLCV candles image](https://databento.com/docs/ohlcv.svg.md) | L0, List of fields, Data formats | | [Instrument definitions](https://databento.com/docs/schemas/instrument-definitions.md) | Properties like symbol, instrument name, expiration date, tick size, and strike price. | ![Instrument definitions image](https://databento.com/docs/definitions.svg.md) | L0, List of fields, Data formats | | [Statistics](https://databento.com/docs/schemas/statistics.md) | Daily instrument statistics like open interest and official settlement prices. | ![Statistics table image](https://databento.com/docs/statistics.svg.md) | L0, List of fields, Data formats | | [Status](https://databento.com/docs/schemas/status.md) | Trading session updates like halts, pauses, short-selling restrictions, and auction start. | ![Status line graph image](https://databento.com/docs/status.svg.md) | L0, List of fields, Data formats | | [Corporate actions](https://databento.com/docs/schemas/corporate-actions.md) | Corporate actions events with global coverage. | ![Corporate actions icon](https://databento.com/docs/corporate-actions-schema.svg.md) | Reference, List of fields | | [Adjustment factors](https://databento.com/docs/schemas/adjustment-factors.md) | End-of-day price adjustment factors for capital events with global coverage. | ![Adjustment factors icon](https://databento.com/docs/adjustment-factors-schema.svg.md) | Reference, List of fields | | [Security master](https://databento.com/docs/schemas/security-master.md) | Latest or point-in-time (PIT) static information on global market securities. | ![Security master icon](https://databento.com/docs/security-master-schema.svg.md) | Reference, List of fields | --- Source: https://databento.com/docs/schemas/whats-a-schema # What's a schema? Databento supports over 15 different data formats of market data. When you make a request from Databento, you must usually specify which data format--also called a **schema**--to receive your data in. ### Supported schemas and their fields A schema represents a collection of data fields. The following is a summary of schemas supported by Databento. Schemas are grouped into four levels, which are also how Databento [prices data](https://databento.com/pricing): L0, L1 (top of book), L2 (market depth), and L3 (full order book). Click on any schema below for its details, the fields included, and a data dictionary that defines each field. | Level | Schema | Schema IDs | Common names used by third parties | | L3: Full order book | [MBO](https://databento.com/docs/schemas/mbo.md) | `mbo` | Market by order, full order book, tick data | | L3: Full order book | [Imbalance](https://databento.com/docs/schemas/imbalance.md) | `imbalance` | Auction imbalance, order imbalance, NOII | | L2: Market depth | [MBP-10](https://databento.com/docs/schemas/mbp-10.md) | `mbp-10` | Market by price, market depth | | L1: Top of book | [MBP-1](https://databento.com/docs/schemas/mbp-1.md) / [CMBP-1](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1) | `mbp-1` / `cmbp-1` | Top of book, trades and quotes | | L1: Top of book | [BBO](https://databento.com/docs/schemas/bbo.md) / [CBBO](https://databento.com/docs/schemas/bbo.md#consolidated-bbo-on-interval-cbbo) | `bbo-1s`, `bbo-1m` / `cbbo-1s`, `cbbo-1m` | Top of book sampled in time space, subsampled BBO and trades | | L1: Top of book | [TBBO](https://databento.com/docs/schemas/tbbo.md) / [TCBBO](https://databento.com/docs/schemas/tbbo.md#consolidated-bbo-on-trade-tcbbo) | `tbbo` / `tcbbo` | Top of book sampled in trade space | | L1: Top of book | [Trades](https://databento.com/docs/schemas/trades.md) | `trades` | Last sale, time and sales, tick-by-tick trades | | L0 | [OHLCV](https://databento.com/docs/schemas/ohlcv.md) | `ohlcv-1s`, `ohlcv-1m`, `ohlcv-1h`, `ohlcv-1d` | OHLCV bars, aggregates | | L0 | [Definition](https://databento.com/docs/schemas/instrument-definitions.md) | `definition` | Security definitions, reference data, symbol list | | L0 | [Statistics](https://databento.com/docs/schemas/statistics.md) | `statistics` | Session or daily statistics, end-of-day summary, open interest | | L0 | [Status](https://databento.com/docs/schemas/status.md) | `status` | Market or trading state/status | **Market by order (MBO)** provides every order book event across every price level, keyed by its order ID. This allows you to determine the queue position of each order and provides the highest level of granularity available. **Imbalance** provides auction imbalance data such as paired quantity, total quantity, and auction status. **Market by price (MBP-10)** provides every order book event across the top ten price levels, keyed by price. This includes every trade and changes to aggregate market depth, alongside total size and order count at the top ten price levels. **Market by price (MBP-1)** provides every order book event that updates the top price level, also known as the best bid and offer (BBO). This includes every trade and changes to book depth, alongside total size and order count at the BBO. **Consolidated market by price (CMBP-1)** provides every order book event that updates the top price level across all venues in the dataset, also known as the consolidated best bid and offer (CBBO). This includes every trade and changes to book depth, alongside total size and publisher attribution at the CBBO. **BBO on interval (BBO)** provides the last best bid, best offer, and sale at 1-second or 1-minute intervals. This is a subset of [MBP-1](https://databento.com/docs/schemas/mbp-1.md). **Consolidated BBO on interval (CBBO)** provides the consolidated last best bid, best offer, and sale at 1-second or 1-minute intervals. This is a subset of [CMBP-1](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1). **BBO on trade (TBBO)** provides every trade event alongside the BBO immediately *before* the effect of each trade. This is a subset of [MBP-1](https://databento.com/docs/schemas/mbp-1.md). **Consolidated BBO on trade (TCBBO)** provides every trade event alongside the consolidated BBO immediately before the effect of each trade. This is a subset of [CMBP-1](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1). **Trades** provides every trade event. This is a subset of [MBO](https://databento.com/docs/schemas/mbo.md). **Aggregate bars (OHLCV)** provide open, high, low, and close prices and total volume aggregated from trades at 1-second, 1-minute, 1-hour, or 1-day intervals. **Instrument definitions** provide reference information about each instrument, which includes properties like symbol, instrument name, expiration date, listing date, tick size, and strike price. **Statistics** provides official summary statistics of each instrument that's published by the venue. This generally includes properties like daily volume, open interest, preliminary and final settlement prices, and official open, high, and low prices. **Status** provides updates about the trading session, such as halts, pauses, short-selling restrictions, auction start, and other matching engine statuses. The granularity and frequency of these updates vary by publisher and dataset. > **See also** > > The MBP-1, BBO and TBBO schemas, as well as the CMBP-1, CBBO, and TCBBO schemas, all provide top of book data with different update space and sampling intervals. Learn more about their differences in our [MBP-1 vs. BBO vs. TBBO schemas guide](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). ### Why are Databento's naming conventions different from third parties? Databento avoids terms like Level 1 (L1) or Level 2 (L2) due to their inconsistent application. For example, some vendors refer to both MBO and MBP data as L2, while others strictly refer to MBP data as L2. More misleadingly, many vendors refer to MBO data as L3, even though this doesn't follow any major trading venue's naming convention. Likewise, the term **tick** originates from the concept of a ticker tape and only refers to trades, _not_ resting limit orders. This becomes a source of confusion when vendors use the term **tick data** to refer to either MBO or MBP data when it should be strictly reserved for trades data. > **See also** > > For more Databento naming conventions and key terminology, visit our [FAQs](https://databento.com/docs/faqs.md). ### Special cases Our MBO, MBP-1 and MBP-10 schemas adopt the following conventions in these special cases: - **Combining MBO with trades feed**: Typically, MBO data provides the highest granularity, but certain venues enhance their trades feed with additional information like trades not reflected in the MBO feed, consolidated NBBO, and more. In these cases, we document the exception in our [Venues and datasets](https://databento.com/docs/datasets.md) section and recommend that you request _both_ our MBO and trades schemas if you need the highest level of granularity. ### Deriving one schema from another Databento captures market data directly from the source and is only subscribed to the most granular feed(s) available from each [publisher](https://databento.com/docs/datasets.md). Order book feeds are usually [normalized](https://databento.com/docs/conventions/normalization.md) into our MBO schema and top-of-book feeds are usually normalized into our MBP-1 schema. To ensure consistency between schemas, Databento doesn't source the less granular schemas from separate feeds. Instead, Databento derives all of the less granular schemas starting from the most granular schema available. As you may have noticed from our schemas' descriptions, the majority of them--MBP, BBO, TBBO, trades, and OHLCV--are simply derived from MBO data. Likewise, you can also derive one schema from another losslessly on the client side, and you should expect your self-derived data to be consistent with ours. For example: - MBP-1, BBO, and Trades can be derived from MBP-10. - BBO, TBBO, and Trades can be derived from MBP-1. - Trades and OHLCV can be derived from TBBO. - OHLCV can be derived from Trades. Deriving your own schema is useful for various reasons: - The data needs to be defined differently for your application. - Our derivation differs from those of another vendor and you want transparency. - You can cut down the number of API requests made to Databento by getting the most granular schema that you need and deriving the rest yourself. - Databento provides MBP-10 merely as a convenience feature. You can reduce bandwidth requirements, latency, and transfer time significantly by deriving MBP-10 yourself from MBO. This is especially relevant for OHLCV, which can vary depending on how trade breaks or market halts are managed, how the start and end of each time interval are determined, and how illiquid instruments are handled if there are no trades over a given time interval. If these considerations are trivial for your use case, Databento offers OHLCV data in multiple time intervals (seconds, minutes, hours, and daily) for your convenience. The table below summarizes which schemas can be derived from another. Each row represents the original schema, and each column represents schemas that you can derive from the original schema. | Schema | MBO | MBP-10 | MBP-1 | CMBP-1 | TBBO | TCBBO | BBO-1s | BBO-1m | CBBO-1s | CBBO-1m | Trades | OHLCV-1s | OHLCV-1m | OHLCV-1h | OHLCV-1d | | MBO | ✓ | ✓ | ✓ | | ✓ | | ✓ | ✓ | | | ✓ | ✓ | ✓ | ✓ | ✓ | | MBP-10 | | ✓ | ✓ | | ✓ | | ✓ | ✓ | | | ✓ | ✓ | ✓ | ✓ | ✓ | | MBP-1 | | | ✓ | | ✓ | | ✓ | ✓ | | | ✓ | ✓ | ✓ | ✓ | ✓ | | CMBP-1 | | | | ✓ | | ✓ | | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | TBBO | | | | | ✓ | | | | | | ✓ | ✓ | ✓ | ✓ | ✓ | | TCBBO | | | | | | ✓ | | | | | ✓ | ✓ | ✓ | ✓ | ✓ | | BBO-1s | | | | | | | ✓ | ✓ | | | | | | | | | BBO-1m | | | | | | | | ✓ | | | | | | | | | CBBO-1s | | | | | | | | | ✓ | ✓ | | | | | | | CBBO-1m | | | | | | | | | | ✓ | | | | | | | Trades | | | | | | | | | | | ✓ | ✓ | ✓ | ✓ | ✓ | | OHLCV-1s | | | | | | | | | | | | ✓ | ✓ | ✓ | ✓ | | OHLCV-1m | | | | | | | | | | | | | ✓ | ✓ | ✓ | | OHLCV-1h | | | | | | | | | | | | | | ✓ | ✓ | | OHLCV-1d | | | | | | | | | | | | | | | ✓ | > **See also** > > Learn how to resample trades data to other intervals, such as 5-minute intervals, from our [bar aggregation tutorial](https://databento.com/docs/examples/basics-historical/custom-ohlcv.md). > > You can also learn how to generate MBP-10 from MBO data using an order book, as seen in our [limit order book construction tutorial](https://databento.com/docs/examples/order-book/limit-order-book.md). --- Source: https://databento.com/docs/schemas/mbo # Market by order (MBO) **Market by order (MBO)** provides every order book event across every price level, keyed by its order ID. This allows you to determine the queue position of each order and provides the highest level of granularity available. MBO data includes all trades, fills, adds, cancels, modifies (or replaces), book clear events, and, depending on the venue and dataset, other special order events. It is often called "L3 data". Databento groups this under [full order book](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields) (L3). ### Fields (`mbo`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 160 in the MBO schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | char | The event action. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, **T**rade, **F**ill, or **N**one. See [Action](https://databento.com/docs/conventions/common-fields-enums-types.md#action). | | `side` | char | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `price` | int64\_t | The order price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The order quantity. | | `channel_id` | uint8\_t | The channel ID assigned by Databento as an incrementing integer starting at zero. | | `order_id` | uint64\_t | The order ID assigned by the venue. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `sequence` | uint32\_t | The message sequence number assigned at the venue. | ### Snapshots For the convenience of managing state and recovery, Databento provides a synthetic snapshot of the order book at the start of each UTC day in our historical MBO data and periodic book snapshots in our real-time MBO data. The mechanics of these snapshots are detailed [here](https://databento.com/docs/conventions/mbo-snapshot.md). > **See also** > > Learn more about the different action types and how to manage order state with respect to each action from our [State management of resting orders](https://databento.com/docs/examples/order-book/order-tracking.md) tutorial. > > Also learn how to construct a limit order book from MBO data from our [limit order book construction](https://databento.com/docs/examples/order-book/limit-order-book.md) tutorial. > > MBO data normalization differs slightly from one venue or dataset to another. > Edge cases and differences are documented separately for each venue in the [Venues and datasets](https://databento.com/docs/datasets.md) section. --- Source: https://databento.com/docs/schemas/imbalance # Imbalance **Imbalance** provides auction imbalance data such as paired quantity, total quantity, and auction status. Databento groups this under [full order book](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields) (L3). > **See also** > > Imbalance data normalization differs slightly from one venue or dataset to another. > Edge cases and differences are documented separately for each venue in the [Venues and datasets](https://databento.com/docs/datasets.md) section. ### Fields (`imbalance`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 20 in the imbalance schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `ref_price` | int64\_t | The price at which the imbalance shares are calculated, where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `auction_time` | uint64\_t | Projected auction timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `cont_book_clr_price` | int64\_t | The hypothetical auction-clearing price for both cross and continuous orders, where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `auct_interest_clr_price` | int64\_t | The hypothetical auction-clearing price for cross orders only, where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ssr_filling_price` | int64\_t | The price at which sell short interest will be filled if a sell short restriction is in effect, where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ind_match_price` | int64\_t | The price at which the highest number of shares would trade, subject to auction collars, where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `upper_collar` | int64\_t | Upper limit of the auction collar, where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `lower_collar` | int64\_t | Lower limit of the auction collar, where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `paired_qty` | uint32\_t | The quantity of shares that are eligible to be matched at `ref_price`. | | `total_imbalance_qty` | uint32\_t | The quantity of shares that are not paired at `ref_price`. | | `market_imbalance_qty` | uint32\_t | The total market order imbalance quantity at the `ref_price`. | | `unpaired_qty` | uint32\_t | During the Closing Auction, the number of unpaired shares priced at or better than the `ref_price`. | | `auction_type` | char | Venue-specific character code indicating the auction type. | | `side` | char | The market side of the `total_imbalance_qty`. Can be **A**sk, **B**id, or **N**one. | | `auction_status` | uint8\_t | Venue-specific status code. Refer to [the venue-specific documentation](https://databento.com/docs/datasets.md). | | `freeze_status` | uint8\_t | Venue-specific status code. Refer to [the venue-specific documentation](https://databento.com/docs/datasets.md). | | `num_extensions` | uint8\_t | The number of times the halt period has been extended. | | `unpaired_side` | char | The side of the `unpaired_qty`. | | `significant_imbalance` | char | Venue-specific status code. Refer to [the venue-specific documentation](https://databento.com/docs/datasets.md). | --- Source: https://databento.com/docs/schemas/mbp-10 # Market by price (MBP-10) **MBP-10 (market by price)** provides every order book event across the top ten price levels, keyed by price. This includes every trade and changes to aggregate market depth, alongside total size and order count at the top ten price levels. This is often called "L2 data". Databento groups this under [market depth](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields) (L2). ### Fields (`mbp-10`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 10 in the MBP-10 schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | char | The event action. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, or **T**rade. See [Action](https://databento.com/docs/conventions/common-fields-enums-types.md#action). | | `side` | char | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `depth` | uint8\_t | The book level where the update event occurred. | | `price` | int64\_t | The order price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The order quantity. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `sequence` | uint32\_t | The message sequence number assigned at the venue. | | `bid_px_N` | int64\_t | The bid price at level N (top level if N = 00) where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ask_px_N` | int64\_t | The ask price at level N (top level if N = 00) where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `bid_sz_N` | uint32\_t | The bid size at level N (top level if N = 00). | | `ask_sz_N` | uint32\_t | The ask size at level N (top level if N = 00). | | `bid_ct_N` | uint32\_t | The bid order count at level N (top level if N = 00). | | `ask_ct_N` | uint32\_t | The ask order count at level N (top level if N = 00). | ### Implementation differences between clients and encodings Bid and ask depth messages (fields starting with `bid_` and `ask_`) are structured differently in the **C++ and Rust clients**, **the Python record interface**, and **JSON data**. Instead of using the `_N` suffix, they're stored in an array of structures named levels, with the top-of-book at index `0`. For example, in C++, `levels[5].bid_px` corresponds to `bid_px_05` in the Python DataFrame API and CSV format. > **See also** > > It is possible to construct MBP-10 yourself from MBO data if you want more price levels or prefer to reduce your bandwidth use. Learn how to construct a limit order book from MBO data from our [limit order book construction](https://databento.com/docs/examples/order-book/limit-order-book.md) tutorial. --- Source: https://databento.com/docs/schemas/mbp-1 # Market by price (MBP-1) **MBP-1 (market by price)** provides every order book event that updates the top price level, also known as the best bid and offer (BBO). This includes every trade and changes to book depth, alongside total size and order count at the BBO. This is often called "L1 data". Databento groups this under [top of book](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields) (L1). ### Fields (`mbp-1`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 1 in the MBP-1 schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | char | The event action. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, or **T**rade. See [Action](https://databento.com/docs/conventions/common-fields-enums-types.md#action). | | `side` | char | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `depth` | uint8\_t | The book level where the update event occurred. | | `price` | int64\_t | The order price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The order quantity. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `sequence` | uint32\_t | The message sequence number assigned at the venue. | | `bid_px_00` | int64\_t | The bid price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ask_px_00` | int64\_t | The ask price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `bid_sz_00` | uint32\_t | The bid size at the top level. | | `ask_sz_00` | uint32\_t | The ask size at the top level. | | `bid_ct_00` | uint32\_t | The bid order count at the top level. | | `ask_ct_00` | uint32\_t | The ask order count at the top level. | > **See also** > > Some venues or data feeds may publish two-sided BBO changes in a single update. In cases like this where the `side` is indeterminate, we'll also use the side code `N`. Exceptions and edge cases like this can be found in our [Venues and datasets](https://databento.com/docs/datasets.md) section. ### Implementation differences between clients and encodings Bid and ask depth messages (fields starting with `bid_` and `ask_`) are structured differently in the **C++ and Rust clients**, **the Python record interface**, and **JSON data**. Instead of using the `_N` suffix, they're stored in an array of structures named levels, with the top-of-book at index `0`. For example, in C++, `levels[5].bid_px` corresponds to `bid_px_05` in the Python DataFrame API and CSV format. > **See also** > > MBP-1 has many similarities to the TBBO and BBO schemas. The main distinction is that MBP-1 is in book update space, while TBBO is in trade space, and BBO is in time space. Learn about the differences between each in our [MBP-1 vs. TBBO vs. BBO schemas guide](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). ##### Consolidated market by price (CMBP-1) **CMBP-1 (consolidated market by price)** provides every order book event that updates the top price level across all venues in the dataset, also known as the consolidated best bid and offer (CBBO). This includes every trade and changes to book depth, alongside total size and publisher attribution at the CBBO. This is often called "L1 data". ### Fields (`cmbp-1`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 177 in the CMBP-1 schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID of the event. Refer to the [CMBP-1 publisher](https://databento.com/docs/schemas/mbp-1.md#cmbp-1-publisher) section below for variants. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | char | The event action. Can be **A**dd, **C**ancel, **M**odify, clea**R** book, or **T**rade. See [Action](https://databento.com/docs/conventions/common-fields-enums-types.md#action). | | `side` | char | The side that initiates the event. Can be **A**sk for a sell order (or sell aggressor in a trade), **B**id for a buy order (or buy aggressor in a trade), or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `price` | int64\_t | The order price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The order quantity. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `bid_px_00` | int64\_t | The bid price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ask_px_00` | int64\_t | The ask price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `bid_sz_00` | uint32\_t | The bid size at the top level. | | `ask_sz_00` | uint32\_t | The ask size at the top level. | | `bid_pb_00` | uint16\_t | The publisher ID indicating the venue containing the best bid. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `ask_pb_00` | uint16\_t | The publisher ID indicating the venue containing the best ask. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | > **See also** > > Some venues or data feeds may publish two-sided CBBO changes in a single update. In cases like this where the `side` is indeterminate, we'll also use the side code `N`. Exceptions and edge cases like this can be found in our [Venues and datasets](https://databento.com/docs/datasets.md) section. ### CMBP-1 publisher The value used to populate the `publisher_id` field will vary depending on the `action`. For **T**rades, `publisher_id` will correspond to the venue the trade executed on. For all other actions, `publisher_id` will be the consolidated publisher ID for the dataset. In all scenarios, `bid_pb_00` and `ask_pb_00` will represent the venues showing the NBBO. ### Implementation differences between clients and encodings Bid and ask depth messages (fields starting with `bid_` and `ask_`) are structured differently in the **C++ and Rust clients**, **the Python record interface**, and **JSON data**. Instead of using the `_N` suffix, they're stored in an array of structures named levels, with the top-of-book at index `0`. For example, in C++, `levels[5].bid_px` corresponds to `bid_px_05` in the Python DataFrame API and CSV format. > **See also** > > CMBP-1 has many similarities to the TCBBO and CBBO schemas. The main distinction is that CMBP-1 is in book update space, while TCBBO is in trade space, and CBBO is in time space. Learn about the differences between each in our [CMBP-1 vs. TCBBO vs. CBBO schemas guide](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). --- Source: https://databento.com/docs/schemas/tbbo # BBO on trade (TBBO) **BBO on trade (TBBO)** provides every trade event alongside the BBO immediately *before* the effect of each trade. This is a subset of [MBP-1](https://databento.com/docs/schemas/mbp-1.md). Databento groups this under [top of book](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields) (L1). ### Fields (`tbbo`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 1 in the TBBO schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | char | The event action. Always **T**rade in the TBBO schema. See [Action](https://databento.com/docs/conventions/common-fields-enums-types.md#action). | | `side` | char | The side that initiates the trade. Can be **A**sk for a sell aggressor in a trade, **B**id for a buy aggressor in a trade, or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `depth` | uint8\_t | The book level where the update event occurred. | | `price` | int64\_t | The order price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The order quantity. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `sequence` | uint32\_t | The message sequence number assigned at the venue. | | `bid_px_00` | int64\_t | The bid price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ask_px_00` | int64\_t | The ask price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `bid_sz_00` | uint32\_t | The bid size at the top level. | | `ask_sz_00` | uint32\_t | The ask size at the top level. | | `bid_ct_00` | uint32\_t | The bid order count at the top level. | | `ask_ct_00` | uint32\_t | The ask order count at the top level. | ### Implementation differences between clients and encodings Bid and ask depth messages (fields starting with `bid_` and `ask_`) are structured differently in the **C++ and Rust clients**, **the Python record interface**, and **JSON data**. Instead of using the `_N` suffix, they're stored in an array of structures named levels, with the top-of-book at index `0`. For example, in C++, `levels[5].bid_px` corresponds to `bid_px_05` in the Python DataFrame API and CSV format. > **See also** > > TBBO has many similarities to the MBP-1 and BBO schemas. The main distinction is that TBBO is in trade space, while MBP-1 is in book update space, and BBO is in time space. In particular, the `action` type of TBBO is always **T**rade. Learn about the differences between each in our [MBP-1 vs. TBBO vs. BBO schemas guide](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). ##### Consolidated BBO on trade (TCBBO) **Consolidated BBO on trade (TCBBO)** provides every trade event alongside the consolidated BBO immediately *before* the effect of each trade. This is a subset of [CMBP-1](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1). ### Fields (`tcbbo`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 194 in the TCBBO schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID indicating the venue where the trade executed on. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | char | The event action. Always **T**rade in the TCBBO schema. See [Action](https://databento.com/docs/conventions/common-fields-enums-types.md#action). | | `side` | char | The side that initiates the event. Can be **A**sk for a sell aggressor in a trade, **B**id for a buy aggressor in a trade, or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `price` | int64\_t | The order price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The order quantity. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `bid_px_00` | int64\_t | The bid price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ask_px_00` | int64\_t | The ask price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `bid_sz_00` | uint32\_t | The bid size at the top level. | | `ask_sz_00` | uint32\_t | The ask size at the top level. | | `bid_pb_00` | uint16\_t | The publisher ID indicating the venue containing the best bid. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `ask_pb_00` | uint16\_t | The publisher ID indicating the venue containing the best ask. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | ### TCBBO publisher The `publisher_id` field will correspond to the venue the trade executed on. `bid_pb_00` and `ask_pb_00` will represent the individual venues showing the NBBO. ### Implementation differences between clients and encodings Bid and ask depth messages (fields starting with `bid_` and `ask_`) are structured differently in the **C++ and Rust clients**, **the Python record interface**, and **JSON data**. Instead of using the `_N` suffix, they're stored in an array of structures named levels, with the top-of-book at index `0`. For example, in C++, `levels[5].bid_px` corresponds to `bid_px_05` in the Python DataFrame API and CSV format. > **See also** > > TCBBO has many similarities to the CMBP-1 and CBBO schemas. The main distinction is that TCBBO is in trade space, while CMBP-1 is in book update space, and CBBO is in time space. In particular, the `action` type of TCBBO is always **T**rade. Learn about the differences between each in our [CMBP-1 vs. TCBBO vs. CBBO schemas guide](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). --- Source: https://databento.com/docs/schemas/bbo # BBO on interval (BBO) **BBO on interval (BBO)** provides the last best bid, best offer, and sale at 1-second or 1-minute intervals. This is a subset of [MBP-1](https://databento.com/docs/schemas/mbp-1.md). Databento groups this under [top of book](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields) (L1). Databento adopts the following convention for BBO: - The time interval is indicated by the schema ID's suffix: `-1s` for 1-second and `-1m` for 1-minute. - If no trade or BBO update occurs within the interval, no record is printed. - If a BBO update occurs but no trade takes place within the interval, the last sale information is forward-filled from the previous interval. - If no BBO update occurs but a trade takes place within the interval, the BBO information is forward-filled from the previous interval. ### Fields (`bbo-1s` and `bbo-1m`) | Field | Type | Description | | `ts_recv` | uint64\_t | The end timestamp of the interval, clamped to the second/minute boundary, expressed as the number of nanoseconds since the UNIX epoch. | | `ts_event` | uint64\_t | The matching-engine-received timestamp of the last trade for the instrument expressed as the number of nanoseconds since the UNIX epoch. Will be `UNDEF_TIMESTAMP` in cases where there was no last trade in the session. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Can be either 195 (`BBO-1s`) or 196 (`BBO-1m`). See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `side` | char | The side that initiated the last trade. Can be **A**sk for a sell aggressor in a trade, **B**id for a buy aggressor in a trade, or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `price` | int64\_t | The last trade price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. Will be `UNDEF_PRICE` if there was no last trade in the session. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The last trade quantity. Will be 0 if there was no last trade in the session. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `sequence` | uint32\_t | The message sequence number assigned at the venue of the last update. | | `bid_px_00` | int64\_t | The bid price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ask_px_00` | int64\_t | The ask price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `bid_sz_00` | uint32\_t | The bid size at the top level. | | `ask_sz_00` | uint32\_t | The ask size at the top level. | | `bid_ct_00` | uint32\_t | The bid order count at the top level. | | `ask_ct_00` | uint32\_t | The ask order count at the top level. | ##### Consolidated BBO on interval (CBBO) **Consolidated BBO on interval (CBBO)** provides the consolidated last best bid, best offer, and sale at 1-second or 1-minute intervals. This is a subset of [CMBP-1](https://databento.com/docs/schemas/mbp-1.md#consolidated-market-by-price-cmbp-1). Databento adopts the following convention for CBBO: - The time interval is indicated by the schema ID's suffix: `-1s` for 1-second and `-1m` for 1-minute. - If no trade or CBBO update occurs within the interval, no record is printed. - If a CBBO update occurs but no trade takes place within the interval, the last sale information is forward-filled from the previous interval. - If no CBBO update occurs but a trade takes place within the interval, the CBBO information is forward-filled from the previous interval. ### Fields (`cbbo-1s` and `cbbo-1m`) | Field | Type | Description | | `ts_recv` | uint64\_t | The end timestamp of the interval, clamped to the second/minute boundary. | | `ts_event` | uint64\_t | The matching-engine-received timestamp of the last trade for the instrument expressed as the number of nanoseconds since the UNIX epoch. Will be `UNDEF_TIMESTAMP` in cases where there was no last trade in the session. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Can be either 192 (`CBBO-1s`) or 193 (`CBBO-1m`). See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The consolidated publisher ID. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `side` | char | The side that initiated the last trade. Can be **A**sk for a sell aggressor in a trade, **B**id for a buy aggressor in a trade, or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `price` | int64\_t | The last trade price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. Will be `UNDEF_PRICE` if there was no last trade in the session. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The last trade quantity. Will be 0 if there was no last trade in the session. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `bid_px_00` | int64\_t | The bid price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `ask_px_00` | int64\_t | The ask price at the top level where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `bid_sz_00` | uint32\_t | The bid size at the top level. | | `ask_sz_00` | uint32\_t | The ask size at the top level. | | `bid_pb_00` | uint16\_t | The publisher ID indicating the venue containing the best bid. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `ask_pb_00` | uint16\_t | The publisher ID indicating the venue containing the best ask. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | ### CBBO publisher The `publisher_id` field will always be the consolidated publisher ID for the dataset. `bid_pb_00` and `ask_pb_00` will represent the individual venues showing the NBBO at the end of the interval. ### Implementation differences between clients and encodings Bid and ask depth messages (fields starting with `bid_` and `ask_`) are structured differently in the **C++ and Rust clients**, **the Python record interface**, and **JSON data**. Instead of using the `_N` suffix, they're stored in an array of structures named levels, with the top-of-book at index `0`. For example, in C++, `levels[5].bid_px` corresponds to `bid_px_05` in the Python DataFrame API and CSV format. > **See also** > > CBBO has many similarities to the CMBP-1 and TCBBO schemas. The main distinction is that CBBO is in time space, while CMBP-1 is in book update space, and TCBBO is in trade space. Learn about the differences between each in our [CMBP-1 vs. TCBBO vs. CBBO schemas guide](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). --- Source: https://databento.com/docs/schemas/trades # Trades **Trades** provides every trade event. This is a subset of [MBO](https://databento.com/docs/schemas/mbo.md). This is often referred to as "time and sales", "last sale," or "tick data." Databento groups this under [top of book](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields) (L1). ### Fields (`trades`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 0 in the trades schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | char | The event action. Always **T**rade in the trades schema. See [Action](https://databento.com/docs/conventions/common-fields-enums-types.md#action). | | `side` | char | The side that initiates the trade. Can be **A**sk for a sell aggressor in a trade, **B**id for a buy aggressor in a trade, or **N**one where no side is specified. See [Side](https://databento.com/docs/conventions/common-fields-enums-types.md#side). | | `depth` | uint8\_t | The book level where the update event occurred. | | `price` | int64\_t | The order price where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `size` | uint32\_t | The order quantity. | | `flags` | uint8\_t | A bit field indicating event end, message characteristics, and data quality. See [Flags](https://databento.com/docs/conventions/common-fields-enums-types.md#flags). | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `sequence` | uint32\_t | The message sequence number assigned at the venue. | --- Source: https://databento.com/docs/schemas/ohlcv # Aggregate bars (OHLCV) **Aggregate bars (OHLCV)** provide open, high, low, and close prices and total volume aggregated from trades at 1-second, 1-minute, 1-hour, or 1-day intervals. Databento groups this under [L0](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields). Databento adopts the following convention for OHLCV: - The time interval is indicated by the schema ID's suffix: `-1s` for 1-second, `-1m` for 1-minute, `-1h` for 1-hour, and `-1d` for 1-day. - The `ts_event` timestamp marks the start of each interval. - If no trade occurs within the interval, no record is printed. ### Fields (`ohlcv-1s`, `ohlcv-1m`, `ohlcv-1h`, `ohlcv-1d`) | Field | Type | Description | | `ts_event` | uint64\_t | The inclusive start time of the bar aggregation period based on the `ts_recv` from trade messages expressed as the number of nanoseconds since the UNIX epoch. | | `rtype` | uint8\_t | A sentinel value indicating the record type. Can be 32 (`OHLCV-1s`), 33 (`OHLCV-1m`), 34 (`OHLCV-1h`), or 35 (`OHLCV-1d`). See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `open` | int64\_t | The open price for the bar where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `high` | int64\_t | The high price for the bar where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `low` | int64\_t | The low price for the bar where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `close` | int64\_t | The close price for the bar where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `volume` | uint64\_t | The total volume traded during the aggregation period. | ### Other sampling intervals If you need other sampling intervals, we recommend that you construct OHLCV aggregates from `trades` data or subsample the OHLCV schema with the nearest resolution on client side. For example, if you need 5-minute OHLCV aggregates, you can subsample this from `ohlcv-1m`. ### Best practices It's recommended that you construct OHLCV aggregates yourself from `trades` data if your application can handle the larger volume of data. There will always be slight implementation differences in how a vendor constructs their OHLCV aggregates. These differences include how trade conditions and retroactive trade breaks are handled, which timestamp is used for the event, and how precisely a vendor closes or publishes the aggregate bar after the end of the interval. Daily OHLCV aggregates may also differ by whether best bid or offer prices are used to compute the highs and lows, and whether the volumes should include block trades. Constructing OHLCV aggregates yourself ensures additional transparency into how the aggregates are computed and consistency within your data. Our `ohlcv-1d` schema is based on UTC dates. If you are interested in daily data based on exchange session hours, you may need to request data for a more granular OHLCV schema and aggregate the data yourself. > **See also** > > `ohlcv-1d` is based entirely on prices and volumes from the electronic trading session and will be consistent with prices and volumes > observed from aggregating our `trades` data. However, this may differ from official settlement prices and volumes published by the venue, > which could include block trades, OTC transactions, and other effects. For official settlement data and more details about this, see > our [statistics](https://databento.com/docs/schemas/statistics.md) schema. --- Source: https://databento.com/docs/schemas/instrument-definitions # Instrument definitions **Instrument definitions** provide point-in-time reference information about each instrument, which includes properties like symbol, instrument name, expiration date, listing date, tick size, and strike price. Databento groups this under [L0](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields). ### Fields (`definition`) | Field [//]: # (column: sortable) | Type [//]: # (column: sortable) | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 19 in the definition schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. | | `instrument_id` | uint32\_t | The numeric instrument ID. | | `raw_symbol` | char\[symbol\_cstr\_len\] | The instrument name (symbol) provided by the publisher, where `symbol_cstr_len` is specified in the [Metadata](https://databento.com/docs/conventions/databento-binary-encoding.md#metadata). | | `security_update_action` | char | Indicates if the instrument definition has been **A**dded, **M**odified, or **D**eleted. | | `instrument_class` | char | The classification of the instrument. See [Instrument class](https://databento.com/docs/schemas/instrument-definitions.md#instrument-class). | | `min_price_increment` | int64\_t | The minimum constant tick for the instrument in units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `display_factor` | int64\_t | The multiplier to convert the venue's display price to the conventional price in units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `expiration` | uint64\_t | The last eligible trade time expressed as a number of nanoseconds since the UNIX epoch. May only have date precision depending on the publisher. | | `activation` | uint64\_t | The time of instrument activation expressed as a number of nanoseconds since the UNIX epoch. May only have date precision depending on the publisher. | | `high_limit_price` | int64\_t | The allowable high limit price for the trading day in units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `low_limit_price` | int64\_t | The allowable low limit price for the trading day in units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `max_price_variation` | int64\_t | The differential value for price banding in units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `unit_of_measure_qty` | int64\_t | The contract size for each instrument, in combination with `unit_of_measure`, in units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `min_price_increment_amount` | int64\_t | The minimum price increment amount disseminated by the venue. Converted to units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `price_ratio` | int64\_t | The value used for price calculation in spread and leg pricing in units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `inst_attrib_value` | int32\_t | Venue-specific instrument attributes. Refer to [the venue-specific documentation](https://databento.com/docs/datasets.md). | | `underlying_id` | uint32\_t | The `instrument_id` of the first underlying instrument. | | `raw_instrument_id` | uint64\_t | The instrument ID assigned by the publisher. May be the same as `instrument_id`. | | `market_depth_implied` | int32\_t | The implied book depth on the price level data feed. | | `market_depth` | int32\_t | The (outright) book depth on the price level data feed. | | `market_segment_id` | uint32\_t | The market segment of the instrument. | | `max_trade_vol` | uint32\_t | The maximum trading volume for the instrument. | | `min_lot_size` | int32\_t | The minimum order entry quantity for the instrument. | | `min_lot_size_block` | int32\_t | The minimum quantity required for a block trade of the instrument. | | `min_lot_size_round_lot` | int32\_t | The minimum quantity required for a round lot of the instrument. Multiples of this quantity are also round lots. | | `min_trade_vol` | uint32\_t | The minimum trading volume for the instrument. | | `contract_multiplier` | int32\_t | The number of deliverables per instrument, i.e. peak days. | | `decay_quantity` | int32\_t | The quantity that a contract will decay daily, after `decay_start_date` has been reached. | | `original_contract_size` | int32\_t | The fixed contract value assigned to each instrument. | | `appl_id` | int16\_t | The channel ID assigned at the venue. | | `maturity_year` | uint16\_t | The calendar year reflected in the instrument symbol. | | `decay_start_date` | uint16\_t | The date at which a contract will begin to decay. | | `channel_id` | uint16\_t | The channel ID assigned by Databento as an incrementing integer starting at zero. | | `currency` | char\[4\] | The currency used for price fields. | | `settl_currency` | char\[4\] | The currency used for settlement, if different from `currency`. | | `secsubtype` | char\[6\] | The strategy type of the spread. | | `group` | char\[21\] | The security group code of the instrument. | | `exchange` | char\[5\] | The exchange used to identify the instrument. | | `asset` | char\[11\] | The underlying asset code (product code) of the instrument. | | `cfi` | char\[7\] | The ISO standard instrument categorization code. | | `security_type` | char\[7\] | The type of the instrument, e.g. FUT for future or future spread. | | `unit_of_measure` | char\[31\] | The unit of measure for the instrument's original contract size, e.g. USD or LBS. | | `underlying` | char\[21\] | The symbol of the first underlying instrument. | | `strike_price_currency` | char\[4\] | The currency used for `strike_price`. | | `strike_price` | int64\_t | The exercise price if the instrument is an option. Converted to units of 1e-9, i.e. 1/1,000,000,000 or 0.000000001. | | `match_algorithm` | char | The matching algorithm used for the instrument, typically **F**IFO. See [Matching algorithm](https://databento.com/docs/schemas/instrument-definitions.md#matching-algorithm). | | `main_fraction` | uint8\_t | The price denominator of the main fraction. | | `price_display_format` | uint8\_t | The number of digits to the right of the tick mark, to display fractional prices. | | `sub_fraction` | uint8\_t | The price denominator of the sub fraction. | | `underlying_product` | uint8\_t | The product complex of the instrument. | | `maturity_month` | uint8\_t | The calendar month reflected in the instrument symbol. | | `maturity_day` | uint8\_t | The calendar day reflected in the instrument symbol, or 0. | | `maturity_week` | uint8\_t | The calendar week reflected in the instrument symbol, or 0. | | `user_defined_instrument` | char | Indicates if the instrument is user defined: **Y**es or **N**o. | | `contract_multiplier_unit` | int8\_t | The type of `contract_multiplier`. Either `1` for hours, or `2` for days. | | `flow_schedule_type` | int8\_t | The schedule for delivering electricity. | | `tick_rule` | uint8\_t | The tick rule of the spread. | | `leg_count` | uint16\_t | The number of legs in the strategy or spread. Will be 0 for outrights. | | `leg_index` | uint16\_t | The 0-based index of the leg (if any). | | `leg_instrument_id` | uint32\_t | The numeric ID assigned to the leg instrument. | | `leg_raw_symbol` | char\[symbol\_cstr\_len\] | The leg instrument's raw symbol assigned by the publisher, where `symbol_cstr_len` is specified in the [Metadata](https://databento.com/docs/conventions/databento-binary-encoding.md#metadata). | | `leg_instrument_class` | char | The leg instrument's classification. See [Instrument class](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-class). | | `leg_side` | char | The side taken for the leg when buying the spread or strategy. | | `leg_price` | int64\_t | The tied price (if any) of the leg. Used for hedged strategies. | | `leg_delta` | int64\_t | The associated delta (if any) of the leg. Used for hedged strategies. | | `leg_ratio_price_numerator` | int32\_t | The numerator of the price ratio of the leg within the spread. | | `leg_ratio_price_denominator` | int32\_t | The denominator of the price ratio of the leg within the spread. | | `leg_ratio_qty_numerator` | int32\_t | The numerator of the quantity ratio of the leg within the spread. | | `leg_ratio_qty_denominator` | int32\_t | The denominator of the quantity ratio of the leg within the spread. | | `leg_underlying_id` | uint32\_t | The numeric ID of the leg instrument's underlying instrument. | ### Instrument class The `instrument_class` field groups instruments into different classes of securities and derivatives. | Name | Value | Description | | Bond | `B` | A bond. | | Call | `C` | A call option. | | Future | `F` | A future. | | Stock | `K` | A stock. | | Mixed spread | `M` | A spread with legs of multiple instrument classes. | | Put | `P` | A put option. | | Future spread | `S` | A spread with future legs. | | Option spread | `T` | A spread with options. | | FX spot | `X` | A foreign exchange spot. | | Commodity spot | `Y` | A commodity being traded for immediate delivery. | | Index | `I` | An index. | ### Security type The `security_type` field classifies the type of instrument. | Name | Value | Description | | Equity | `EQUITY` | An equity. | | Future | `FUT` | A future or a spread on futures. | | Interest rate swap | `IRS` | An interest rate swap security. | | Spreads | `MLEG` | A spread with legs of multiple instrument classes. | | Option on equity | `OPT` | An option on equity or a spread with options on equities. | | Option on future | `OOF` | An option on future or a spread with options on futures. | | Spot | `SPOT` | An FX spot or a commodity spot. | ### Matching algorithm The `matching_algorithm` field defines the type of matching algorithm used for the instrument. | Name | Value | Description | | Undefined | ` ` | A matching algorithm was not specified. | | FIFO | `F` | First-in-first-out matching. | | Configurable | `K` | A configurable match algorithm. | | Pro-Rata | `C` | Trade quantity is allocated to resting orders based on a pro-rata percentage: resting order quantity divided by total quantity. | | FIFO with LMM | `T` | Like FIFO, but with LMM allocations prior to FIFO allocations. | | Threshold Pro-Rata | `O` | Like Pro-Rata, but includes a configurable allocation to the first order that improves the market. Minimum order thresholds may exist. | | FIFO with Top Order and LMM | `S` | Like FIFO with LMM, but includes a configurable allocation to the first order that improves the market. | | Threshold Pro-Rata with LMM | `Q` | Like Threshold Pro-Rata, but includes a special priority to LMMs. | | Eurodollar Futures | `Y` | Special variant used only for Eurodollar futures on CME. | | Time Pro-Rata | `P` | Trade quantity is shared between all orders at the best price. Orders with the highest time priority receive a higher matched quantity. | | Institutional Prioritization | `V` | A two-pass FIFO algorithm. The first pass fills the Institutional Group the aggressing order is associated with. The second pass matches orders without an Institutional Group association. | | Allocation | `A` | Like Pro-Rata, but includes a configurable allocation to the first order that improves the market. | ### Point-in-time instrument definitions An important distinction of Databento's instrument definition data is that we usually provide it in **point-in-time** format like the rest of our market data. In other words, the instrument definitions are timestamped to the nanosecond and treated as a time series, allowing you to simulate the sequence in which they arrived in real time. This is useful as it avoids lookahead and survivorship bias, and also allows you to backtest any strategy that trades instruments immediately as they become available intraday, such as new options strikes and new listings from IPOs. If you're switching to Databento from another data provider, you may find this behavior different, as many data providers only give you a cumulative list of all instruments or daily updates on instrument definitions. ### Snapshots For publishers that do not provide a full list of instrument definitions at least daily, the historical API provides a snapshot of all active definitions at UTC midnight Monday through Friday. Therefore, any request for definitions for a 24-hour period beginning at UTC midnight is guaranteed to include definitions for all instruments active during that time period. ### Instrument definitions on time series endpoints vs. reference data endpoints Databento also provides other forms of reference information and static data, such as corporate actions, cross-symbology mappings, and security master data. However, these additional forms of reference data may be found on API endpoints other than our time series endpoints. The reason that these are separated into different endpoints is that instrument definitions on our time series endpoints are those included with the primary source of data, which is usually a direct feed from a trading venue. On the other hand, other forms of reference data may originate from different secondary sources or need to be licensed separately from such direct feeds. Databento generally provides ways for you to join and merge instrument definitions and other forms of reference data. ### Strategies One definition record is created for each leg of a strategy or spread. The primary fields will be the same for each of these records, while the fields beginning with `leg_` will have information about the particular leg. Some publishers send the strategy definition before the definitions of its legs. Where this happens, the `leg_` fields taken from the leg instrument's own definition are populated by a later record with `security_update_action` set to **M**odify. --- Source: https://databento.com/docs/schemas/statistics # Statistics **Statistics** provides official summary statistics of each instrument that's published by the venue. This generally includes properties like daily volume, open interest, preliminary and final settlement prices, and official open, high, and low prices. Databento groups this under [L0](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields). ### Fields (`statistics`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 24 in the statistics schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `ts_ref` | uint64\_t | The reference timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `price` | int64\_t | The value for price statistics where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. See [Prices](https://databento.com/docs/conventions/common-fields-enums-types.md#prices). | | `quantity` | int64\_t | The value for non-price statistics. Will be `INT64_MAX` when unused. | | `sequence` | uint32\_t | The message sequence number assigned at the venue. | | `ts_in_delta` | int32\_t | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See [ts_in_delta](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-in-delta). | | `stat_type` | uint16\_t | The type of statistic value contained in the message. See [Types of statistics](https://databento.com/docs/schemas/statistics.md#types-of-statistics) table below. | | `channel_id` | uint16\_t | The channel ID within the venue. | | `update_action` | uint8\_t | Indicates if the statistic is newly added (`1`) or deleted (`2`). (Deleted is only used with some stat_types) | | `stat_flags` | uint8\_t | Additional flags associated with certain stat types and datasets. Refer to the [Venues and datasets](https://databento.com/docs/datasets.md) section for details. | Some fields are not applicable depending on the type of statistic. Null, invalid or inapplicable values are represented by the maximum value of the field's type. For example, null is represented by `2^64-1` for `ts_event`, which has an unsigned 64-bit integer type. In cases where the value is actually zero or null has the same meaning as zero, zero is used instead. ### Types of statistics | Type | `stat_type` | `price` | `quantity` | `stat_flags` | `ts_ref` | Description | | Opening price | `1` | ✓ | ✓ | | | The price of the first trade of an instrument. The quantity is set when provided by the venue. | | Indicative opening price | `2` | ✓ | ✓ | | | The probable price of the first trade of an instrument published during pre-open. | | Settlement price | `3` | ✓ | | ✓ | ✓ | The settlement price of an instrument. Flags will indicate whether the price is final or preliminary and actual or theoretical. | | Trading session low price | `4` | ✓ | | | | The lowest trade price of an instrument during the trading session. | | Trading session high price | `5` | ✓ | | | | The highest trade price of an instrument during the trading session. | | Cleared volume | `6` | | ✓ | | ✓ | The number of contracts cleared for an instrument on the previous trading date. | | Lowest offer | `7` | ✓ | | | | The lowest offer price for an instrument during the trading session. | | Highest bid | `8` | ✓ | | | | The highest bid price for an instrument during the trading session. | | Open interest | `9` | | ✓ | | ✓ | The current number of outstanding contracts of an instrument. | | Fixing price | `10` | ✓ | | | ✓ | The volume-weighted average price (VWAP) for a fixing period. | | Close price | `11` | ✓ | ✓ | | | The last trade price and quantity during a trading session. | | Net change | `12` | ✓ | | | | The change in price from the close price of the previous session to the most recent close price. | | Volume-weighted average price | `13` | ✓ | ✓ | | | The volume-weighted average price (VWAP) during the trading session. | | Volatility | `14` | ✓ | | | ✓ | The implied volatility associated with the settlement price. | | Delta | `15` | ✓ | | | ✓ | The options delta associated with the settlement price. | | Uncrossing price | `16` | ✓ | ✓ | | | The auction uncrossing price and quantity. This is used for auctions that are neither the official opening auction nor the official closing auction. | | Upper price limit | `17` | ✓ | | | | The upper price limit for an instrument for the trading session. | | Lower price limit | `18` | ✓ | | | | The lower price limit for an instrument for the trading session. | | Block volume | `19` | | ✓ | | | The total number of block contracts traded for the day. | | Indicative close price | `20` | ✓ | | | | The probable price of the last trade of an instrument published during the trading session. | | MWCB Level 1 | `21` | ✓ | | | | The Market-Wide Circuit Breaker (MWCB) Level 1 threshold (7%), expressed as S&P 500 index points. | | MWCB Level 2 | `22` | ✓ | | | | The Market-Wide Circuit Breaker (MWCB) Level 2 threshold (13%), expressed as S&P 500 index points. | | MWCB Level 3 | `23` | ✓ | | | | The Market-Wide Circuit Breaker (MWCB) Level 3 threshold (20%), expressed as S&P 500 index points. | | Auction collar reference price | `24` | ✓ | | | | The reference price used to derive the auction collar bounds. | | Auction collar upper price | `25` | ✓ | | | | The upper bound of the auction collar. | | Auction collar lower price | `26` | ✓ | | | | The lower bound of the auction collar. | | Venue-specific volume 1 | `10001` | | ✓ | | | A venue-specific volume field. Refer to venue documentation for details. | | Venue-specific price 1 | `10002` | ✓ | | | | A venue-specific price field. Refer to venue documentation for details. | ### Types of statistics by dataset The table below shows which statistics are available by dataset ID. See the table above for a list of `stat_type` variants. Statistic types that are not shown as columns are not currently published on any of these datasets. | `stat_type` | `1` | `2` | `3` | `4` | `5` | `6` | `7` | `8` | `9` | `10` | `11` | `12` | `13` | `14` | `15` | `16` | `17` | `18` | `19` | `20` | `21` | `22` | `23` | `24` | `25` | `26` | `10001` | | [ARCX.PILLAR](https://databento.com/docs/datasets/arcx-pillar.md) | ✓ | | | | | | | | | | ✓ | | | | | ✓ | | | | | | | | | | | | | [BATS.PITCH](https://databento.com/docs/datasets/bats-pitch.md) | ✓ | | | | | | | | | | ✓ | | | | | | | | | | | | | | | | | | [EQUS.SUMMARY](https://databento.com/docs/datasets/equs-summary.md) | ✓ | | | ✓ | ✓ | ✓ | | | | | ✓ | | | | | | | | | | | | | | | | | | [GLBX.MDP3](https://databento.com/docs/datasets/glbx-mdp3.md) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | | | | | ✓ | ✓ | | | | | | | | | | | [IFEU.IMPACT](https://databento.com/docs/datasets/ifeu-impact.md) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | ✓ | | ✓ | ✓ | ✓ | | | | | | | | | | | | | | [IFLL.IMPACT](https://databento.com/docs/datasets/ifll-impact.md) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | ✓ | | ✓ | ✓ | ✓ | | | | | | | | | | | | | | [IFUS.IMPACT](https://databento.com/docs/datasets/ifus-impact.md) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | ✓ | | ✓ | ✓ | ✓ | | | | | | | | | | | | | | [NDEX.IMPACT](https://databento.com/docs/datasets/ndex-impact.md) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | | ✓ | | ✓ | | ✓ | ✓ | ✓ | | | | | | | | | | | | | | [OPRA.PILLAR](https://databento.com/docs/datasets/opra-pillar.md) | ✓ | | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | | | | | | | | | | | | | | | | | [XASE.PILLAR](https://databento.com/docs/datasets/xase-pillar.md) | ✓ | | | | | | | | | | ✓ | | | | | ✓ | | | | | | | | | | | | | [XCBF.PITCH](https://databento.com/docs/datasets/xcbf-pitch.md) | ✓ | | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | | | | | ✓ | ✓ | ✓ | | | | | | | | ✓ | | [XCHI.PILLAR](https://databento.com/docs/datasets/xchi-pillar.md) | ✓ | | | | | | | | | | ✓ | | | | | ✓ | | | | | | | | | | | | | [XEEE.EOBI](https://databento.com/docs/datasets/xeee-eobi.md) | ✓ | | ✓ | ✓ | ✓ | ✓ | | | ✓ | | ✓ | | | | | | | | | | | | | | | | | | [XEUR.EOBI](https://databento.com/docs/datasets/xeur-eobi.md) | ✓ | | ✓ | ✓ | ✓ | ✓ | | | ✓ | | ✓ | | | | | | | | | | | | | | | | | | [XNAS.BASIC](https://databento.com/docs/datasets/xnas-basic.md) | ✓ | | | | | | | | | | ✓ | | | | | | | | | | | | | | | | | | [XNAS.ITCH](https://databento.com/docs/datasets/xnas-itch.md) | ✓ | | | | | | | | | | ✓ | | | | | ✓ | | | | | | | | | | | | | [XNYS.PILLAR](https://databento.com/docs/datasets/xnys-pillar.md) | ✓ | | | | | | | | | | ✓ | | | | | ✓ | | | | | | | | | | | | ### Official vs. Databento summary statistics The key distinction of the `statistics` schema is these are official summary statistics provided by the venue--Databento doesn't compute these statistics. On most venues, Databento also provides separate [OHLCV](https://databento.com/docs/schemas/ohlcv.md) data that could be used in place of official open, high, low, and settlement prices, and volume from the `statistics` schema. These schemas are different and may vary for a few intentional reasons: - Often, official statistics have opaque methodology or are difficult to replicate because they're tallied by hand - Some venues include volumes and open interest from open outcry, auction, block trades, RFQs, or other events that may not be disseminated in the electronic trading session or public feeds - Some venues double count volumes The main purpose of Databento's summary data is that it provides more consistency: we derive it systematically from the tick data or full order book data during the electronic trading session and ensures consistency with our tick data; we publish at more deterministic times, and we normalize across venues using a UTC midnight cutoff universally. For electronic trading and research applications, this consistency may be more important, whereas for back office and administrative applications, official statistics may be preferable. --- Source: https://databento.com/docs/schemas/status # Status **Status** provides updates about the trading session, such as halts, pauses, short-selling restrictions, auction start, and other matching engine statuses. The granularity and frequency of these updates vary by publisher and dataset. Databento groups this under [L0](https://databento.com/docs/schemas/whats-a-schema.md#supported-schemas-and-their-fields). ### Fields (`status`) | Field | Type | Description | | `ts_recv` | uint64\_t | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv). | | `ts_event` | uint64\_t | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. See [ts_event](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-event). | | `rtype` | uint8\_t | A sentinel value indicating the record type. Always 18 in the status schema. See [Rtype](https://databento.com/docs/conventions/common-fields-enums-types.md#rtype). | | `publisher_id` | uint16\_t | The publisher ID assigned by Databento, which denotes the dataset and venue. See [Publishers](https://databento.com/docs/conventions/common-fields-enums-types.md#publishers-datasets-and-venues). | | `instrument_id` | uint32\_t | The numeric instrument ID. See [Instrument identifiers](https://databento.com/docs/conventions/common-fields-enums-types.md#instrument-identifiers). | | `action` | uint16\_t | The type of status change. See [Status actions](https://databento.com/docs/schemas/status.md#status-action-variants) table below. | | `reason` | uint16\_t | Additional details about the cause of the status change. See [Status reasons](https://databento.com/docs/schemas/status.md#status-reason-variants) table below. | | `trading_event` | uint16\_t | Further information about the status change (if provided). See [Trading events](https://databento.com/docs/schemas/status.md#trading-event-variants) table below. | | `is_trading` | char | The best-efforts state of trading in the instrument, either `Y`, `N`, or `~`. | | `is_quoting` | char | The best-efforts state of quoting in the instrument, either `Y`, `N`, or `~`. | | `is_short_sell_restricted` | char | The best-efforts state of short sell restrictions for the instrument (if applicable), either `Y`, `N`, or `~`. | ### Status action variants | Variant | `action` | Description | | None | `0` | No change. | | Pre-open | `1` | The instrument is in a pre-open period. | | Pre-cross | `2` | The instrument is in a pre-cross period. | | Quoting | `3` | The instrument is quoting but not trading. | | Cross | `4` | The instrument is in a cross/auction. | | Rotation | `5` | The instrument is being opened through a trading rotation. | | New price indication | `6` | A new price indication is available for the instrument. | | Trading | `7` | The instrument is trading. | | Halt | `8` | Trading in the instrument has been halted. | | Pause | `9` | Trading in the instrument has been paused. | | Suspend | `10` | Trading in the instrument has been suspended. | | Pre-close | `11` | The instrument is in a pre-close period. | | Close | `12` | Trading in the instrument has closed. | | Post-close | `13` | The instrument is in a post-close period. | | Short sell restriction (SSR) change | `14` | A change in short-selling restrictions. | | Not available for trading | `15` | The instrument is not available for trading, either trading has closed or been halted. | ### Status reason variants | Variant | `reason` | Description | | None | `0` | No reason given. | | Scheduled | `1` | The change in status occurred as scheduled. | | Surveillance intervention | `2` | The instrument stopped due to a market surveillance intervention. | | Market event | `3` | The status changed due to activity in the market. | | Instrument activation | `4` | The derivative instrument began trading. | | Instrument expiration | `5` | The derivative instrument expired. | | Recovery in process | `6` | Recovery in progress. | | Regulatory | `10` | The status change was caused by a regulatory action. | | Administrative | `11` | The status change was caused by an administrative action. | | Non-compliance | `12` | The status change was caused by the issuer not being in compliance with regulatory requirements. | | Filings not current | `13` | Trading halted because the issuer's filings are not current. | | SEC trading suspension | `14` | Trading halted due to an SEC trading suspension. | | New issue | `15` | The status changed because a new issue is available. | | Issue available | `16` | The status changed because an issue is available. | | Issues reviewed | `17` | The status changed because the issue(s) were reviewed. | | Filing requirements satisfied | `18` | The status changed because the filing requirements were satisfied. | | News pending | `30` | Relevant news is pending. | | News released | `31` | Relevant news was released. | | News and resumption times | `32` | The news has been fully disseminated and times are available for the resumption in quoting and trading. | | News not forthcoming | `33` | The relevant news was not forthcoming. | | Order imbalance | `40` | Halted for order imbalance. | | LULD pause | `50` | The instrument hit limit up or limit down. | | Operational | `60` | An operational issue occurred with the venue. | | Additional information requested | `70` | The status changed until the exchange receives additional information. | | Merger effective | `80` | Trading halted due to merger becoming effective. | | ETF | `90` | Trading is halted in an ETF due to conditions with the component securities. | | Corporate action | `100` | Trading is halted for a corporate action. | | New Security offering | `110` | Trading is halted because the instrument is a new offering. | | Market wide halt level 1 | `120` | Halted due to the market-wide circuit breaker level 1. | | Market wide halt level 2 | `121` | Halted due to the market-wide circuit breaker level 2. | | Market wide halt level 3 | `122` | Halted due to the market-wide circuit breaker level 3. | | Market wide halt carryover | `123` | Halted due to the carryover of a market-wide circuit breaker from the previous trading day. | | Market wide halt resumption | `124` | Resumption due to the end of a market-wide circuit breaker halt. | | Quotation not available | `130` | Halted because quotation is not available. | ### Trading event variants | Variant | `trading_event` | Description | | None | `0` | No additional information given. | | No cancel | `1` | Order entry is allowed. Modification and cancellation are not allowed. | | Change trading session | `2` | A change of trading session occurred. Daily statistics are reset. | | Implied matching on | `3` | Implied matching is available. | | Implied matching off | `4` | Implied matching is not available. | ### Snapshots As trading status can carry across UTC days, the historical API includes a snapshot of the last status record for each active instrument at UTC midnight. --- Source: https://databento.com/docs/schemas/corporate-actions # Corporate actions **Corporate actions** includes coverage from 215+ exchanges worldwide, 310,000+ listed and delisted securities from 100,000+ companies, categorized as 60+ event types. The dataset focuses on crucial events affecting global securities such as dividends, takeovers, mergers & acquisitions and rights issues. > **See also** > > [Reference data enums](https://databento.com/docs/conventions/reference-data-enums.md) for a lookup table of possible enumeration field values. ### Fields - Timestamps are strings in [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format, expressed in UTC - Dates are strings in YYYY-MM-DD [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format, expressed in the local time of the listing `exchange` - Enums are strings, with each value corresponding to a predefined set of options listed in the associated lookup table | Field [//]: # (column: sortable) | Type [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `ts_record` | Timestamp | The timestamp (UTC) the record last changed. | | `event_unique_id` | String | Unique corporate actions record identifier. Can be used to deduplicate records for the same event. | | `event_id` | String | Event identifier unique at the event level. | | `listing_id` | String | Unique listing numerical ID. Concatenation of a sequence number and the `listing_group_id`. The sequence number will usually be 1 unless additional listings exists for the same security in the same market, usually as a result of secondary trading currencies. | | `listing_group_id` | String | Groups all listings for the same security on a specific exchange, often in different trading currencies. | | `security_id` | String | Security level numerical ID. Can be used to link all multiple listings together. | | `issuer_id` | String | Issuer level numerical ID. Can be used to link all securities of a company together. | | `event_action` | Enum | Record action status code. See [ACTION](https://databento.com/docs/conventions/reference-data-enums.md#action) enum. | | `event` | Enum | Event type. The `event_id` where applicable links all payments rows of an event together to show all the payment options. Can be used as a short event name. See [Corporate actions events](https://databento.com/docs/conventions/corporate-actions-events.md). | | `event_subtype` | Enum or null | Event subtype. This is only used for a limited number of events where the event data supplied falls into distinct sub-groupings. See [EVENTSUBTYPE](https://databento.com/docs/conventions/reference-data-enums.md#eventsubtype) enum and [Corporate actions events](https://databento.com/docs/conventions/corporate-actions-events.md). | | `event_date_label` | String | The main calendar date name for the event. | | `event_date` | Date | The primary date associated with the event, often marking when the event is scheduled to occur or take effect. | | `event_created_date` | Date | The date on which the event was officially created or recorded in the system. | | `ex_date` | Date or null | The ex-dividend date (ex-date), marking the first date on which the security trades without the entitlement to the upcoming dividend, distribution, or other corporate action benefit. Where the `payment_type` of a multiple payment event is flagged as a (**B**)oth, this ex-date field will contain the ex-date of the cash component. The `ex_date2` field in the `event_info` section will carry the stock component ex-date. | | `effective_date` | Date or null | The date on which the event becomes effective or is executed, signifying when the changes or actions are officially recognized. | | `record_date` | Date or null | The date on which a company reviews its records to determine the eligible shareholders entitled to receive dividends or participate in the event. The record date is one business day after the ex-date. | | `record_date_id` | Date or null | Record date ID. This ID links all events for the same security that share the same record date. | | `related_event` | Enum or null | Related event type. See [Corporate actions events](https://databento.com/docs/conventions/corporate-actions-events.md). | | `related_event_id` | Date or null | Direct link to another event. | | `global_status` | Enum | Global status code. Indicates the global listing activity status of a security. See [GLOBSTATUS](https://databento.com/docs/conventions/reference-data-enums.md#globstatus) enum. | | `listing_status` | Enum | Listing status code. Indicates the listing activity status at market level. See [LISTSTAT](https://databento.com/docs/conventions/reference-data-enums.md#liststat) enum. | | `listing_source` | Enum | Indicates if the listing level data in the record is (**M**)ain or (**S**)econdary. See [LISTSOURCE](https://databento.com/docs/conventions/reference-data-enums.md#listsource) enum. | | `listing_date` | Date or null | The date when the security is officially listed on the exchange and becomes available for trading. | | `delisting_date` | Date or null | The date when the security is officially removed from the exchange and ceases to be traded. | | `issuer_name` | String | The issuer name. | | `security_type` | Enum | The security type. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `security_description` | String | The security description. | | `primary_exchange` | Enum or null | Exchange code for the primary security. Empty if unknown. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `exchange` | Enum | Exchange code for the listing. Equivalent to the MIC but more stable as MIC might not be available in a timely fashion. Also note that the MIC can change but the `exchange` will remain the same. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `operating_mic` | Enum or null | Market Identifier Code (MIC) as an ISO 10383 string. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `symbol` | String or null | The query input symbol which matched the record. | | `nasdaq_symbol` | String or null | The security's symbol in the Nasdaq convention (e.g. `BF.B`). See [Symbology](https://databento.com/docs/schemas/security-master.md#symbology) on the security master schema page for known limitations on `USNYSE` listings. | | `local_code` | String or null | The listing exchange's native symbol form, as published by the upstream reference-data vendor. Convention varies by listing exchange. Either an alpha string, or a number. | | `isin` | String or null | ISIN global level identifier as an ISO 6166 string. | | `us_code` | String or null | US domestic CUSIP. | | `bbg_comp_id` | String or null | Bloomberg composite global ID. | | `bbg_comp_ticker` | String or null | Bloomberg composite ticker. | | `figi` | String or null | Bloomberg FIGI, that is exchange level ID. | | `figi_ticker` | String or null | Bloomberg exchange level ticker. | | `listing_country` | Enum or null | Listing country as two letter ISO 3166-1 alpha-2 string. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `register_country` | Enum or null | Register country as two letter ISO 3166-1 alpha-2 string. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `trading_currency` | Enum or null | Trading currency as an ISO 4217 string. See [CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen) enum. | | `multi_currency` | Boolean | True if there is currently more than one listing in the market, otherwise false. | | `segment_mic_name` | String or null | Market Segment Name. Used when no Segment MIC is available (Not the ISO 10383 MICSEG lookup name). | | `segment_mic` | Enum or null | Market Identifier Code (MIC) as an ISO 10383 string. See [MICSEG](https://databento.com/docs/conventions/reference-data-enums.md#micseg) enum. | | `mand_volu_flag` | Enum | Indicates mandatory and/or voluntary event participation requirement. With a few event types the flag change in the course of the corporate action lifecycle. See [MANDVOLU](https://databento.com/docs/conventions/reference-data-enums.md#mandvolu) enum. | | `rd_priority` | Number or null | The priority order number sequence where populated is necessary to correctly calculate resultant cash and stock outcomes. | | `lot_size` | Number or null | Lot Size. Indicates the minimum number of shares that can be acquired in one transaction. | | `par_value` | Number or null | Par value amount. | | `par_value_currency` | Enum or null | Par value currency as an ISO 4217 string. See [CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen) enum. | | `payment_date` | Date or null | The date on which the dividend or payment is made to eligible shareholders. Where the `payment_type` of a multiple payment event is flagged as a (**B**)oth, this payment date field will contain the payment date of the cash component. | | `duebills_redemption_date` | Date or null | Due bills redemption date. | | `from_date` | Date or null | The earliest date from which a specific corporate action or event is valid, active, or can be exercised. This date marks the beginning of the relevant event or process, such as when shareholders can start participating in a buyback, conversion, or rights issue. Applicable to a wide range of events. | | `to_date` | Date or null | The final date by which the specific corporate action or event is valid, active, or must be completed. This date marks the conclusion of the relevant event or process, after which the associated rights, entitlements, or obligations may expire. Applicable to a wide range of events. | | `registration_date` | Date or null | Registration date. | | `start_date` | Date or null | The date on which the specific corporate action or event begins or becomes effective. This marks the commencement of the event's active period, such as the start of a buyback program, the beginning of a class action period, or the initiation of an odd-lot offer. Applicable to events like Class Actions (`CLSAC`), Certificate of Exchange (`CTX`), Buyback (`BB`), Odd-Lot Offer (`ODDLT`), and Financial Transaction Tax (`FTT`). | | `end_date` | Date or null | The final date by which the corporate action or event is valid, active, or must be completed. This marks the conclusion of the event's active period, such as the end of a buyback program, the closing of a class action period, or the deadline for participating in an odd-lot offer. Applicable to events like Class Actions (`CLSAC`), Certificate of Exchange (`CTX`), Buyback (`BB`), Odd-Lot Offer (`ODDLT`), and Financial Transaction Tax (`FTT`). | | `open_date` | Date or null | The date when the corporate action or event opens for participation or becomes available to stakeholders. This marks the beginning of the active period during which actions like tender offers, purchase offers, or redemptions can be initiated. Applicable to events like Takeovers (`TKOVR`), Purchase Offers (`PO`), and Redemptions (`REDEM`). | | `close_date` | Date or null | The final date by which participation in the corporate action or event must be completed. This marks the end of the active period for actions like tender offers, purchase offers, or redemptions. Applicable to events like Takeovers (`TKOVR`), Purchase Offers (`PO`), and Redemptions (`REDEM`). | | `start_subscription_date` | Date or null | The date when the subscription period for the event begins. | | `end_subscription_date` | Date or null | The date when the subscription period for the event ends. | | `option_election_date` | Date or null | The deadline by which participants must choose or elect an option related to the event. | | `withdrawal_rights_from_date` | Date or null | The date from which withdrawal rights become effective, allowing participants to retract their election or participation. | | `withdrawal_rights_to_date` | Date or null | The final date by which participants can exercise their withdrawal rights, retracting their election or participation. | | `notification_date` | Date or null | The date on which the event notification is issued or made public. | | `financial_year_end_date` | Date or null | The closing date of the company's financial year. | | `exp_completion_date` | Date or null | The anticipated date when the event or related transaction is expected to be completed. | | `payment_type` | Enum or null | Payment type. See [PAYTYPE](https://databento.com/docs/conventions/reference-data-enums.md#paytype) enum. | | `option_id` | String or null | Option number of the event where applicable - options are ORs. | | `serial_id` | String or null | Serial number of the event where applicable - serials are ANDs. | | `default_option_flag` | Boolean or null | Flag indicating the benefit the shareholder would receive by default in case of several options being offered to them. The default is not always announced. | | `rate_currency` | Enum or null | Payment currency as an ISO 4217 string. See [CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen) enum. | | `ratio_old` | Number or null | Ratio Denominator is the existing holding. | | `ratio_new` | Number or null | Ratio Numerator is the new holding (for `FSPLT`, `DIV` and `BON` events *includes* the existing holding). | | `fraction` | Enum or null | Describes how fractions are handled in settlement calculations. See [FRACTIONS](https://databento.com/docs/conventions/reference-data-enums.md#fractions) enum. | | `outturn_style` | Enum or null | Style of outturn security. See [OUTTURNSTYLE](https://databento.com/docs/conventions/reference-data-enums.md#outturnstyle) enum. | | `outturn_security_type` | Enum or null | Security asset type. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `outturn_security_id` | String or null | Outturn security. | | `outturn_isin` | String or null | Outturn ISIN. | | `outturn_us_code` | String or null | Outturn CUSIP. | | `outturn_local_code` | String or null | Outturn local code. | | `outturn_bbg_comp_id` | String or null | Outturn Bloomberg composite ID. | | `outturn_bbg_comp_ticker` | String or null | Outturn Bloomberg composite ticker. | | `outturn_figi` | String or null | Outturn FIGI - Bloomberg exchange level ID. | | `outturn_figi_ticker` | String or null | Outturn FIGI Ticker - Bloomberg exchange level ticker. | | `min_offer_qty` | Number or null | The quantity range within which the shareholder can offer from their total holding in the event. | | `max_offer_qty` | Number or null | The quantity range within which the shareholder can offer from their total holding in the event. | | `min_qualify_qty` | Number or null | If the quantity held by shareholder is within this range then they qualify for taking part in the event. | | `max_qualify_qty` | Number or null | If the quantity held by shareholder is within this range then they qualify for taking part in the event. | | `min_accept_qty` | Number or null | The total quantity the company will accept from all the shareholders tendering their shares in the event for the event to be binding on the offeror. | | `max_accept_qty` | Number or null | The total quantity the company will accept from all the shareholders tendering their shares in the event for the event to be binding on the offeror. | | `tender_strike_price` | Number or null | If the event is via a tender process it is the cut off price at which all the bids are accepted. | | `tender_price_step` | Number or null | If the event is via a tender then there is a price step in which one can put in the bids. E.g. If the price step specified is 10p then one has to make bids in steps of 10p i.e. GBP 1.10, GBP 1.20, GBP 1.30. | | `option_expiry_time` | String or null | Option expiry time. | | `option_expiry_tz` | String or null | Option expiry time zone. | | `withdrawal_rights_flag` | Boolean or null | Withdrawal rights flag. | | `withdrawal_rights_expiry_time` | String or null | Withdrawal rights expiry time. | | `withdrawal_rights_expiry_tz` | String or null | Withdrawal rights expiry time zone. | | `expiry_time` | String or null | Expiry time. | | `expiry_tz` | String or null | Expiry time zone. | | `date_info` | JSON | Event-specific date information. | | `rate_info` | JSON | Event-specific payment rate information. | | `event_info` | JSON | Additional event-specific information. | | `ts_created` | Timestamp | The timestamp (UTC) the record was added by Databento. | ### Identifiers Identifiers for listing, security and issuer are grouped at the beginning of each record, which enables clearer comparisons of where a single security is associated with many individual listings across different exchanges. There is one `primary_exchange` for a security, and it may then be listed on multiple secondary `exchange`s. ### Dynamic fields The following dynamically-sized fields provide additional supplementary information as key-value pairs: - `date_info`. Date specific information related to the event - `rate_info`. Rate specific information related to the event - `event_info`. Additional supplementary event information > **Info** > The content of these dynamic fields is contingent on the type of `event` and its `event_subtype`, > allowing for flexible and detailed data representation. --- Source: https://databento.com/docs/schemas/adjustment-factors # Adjustment factors **Adjustment factors** covers worldwide corporate actions capital events, spanning multiple currencies for the same event. Each record includes all necessary fields to identify the country, security, event, and adjustment factor. > **See also** > > [Reference data enums](https://databento.com/docs/conventions/reference-data-enums.md) for a lookup table of possible enumeration field values. ### Fields - Timestamps are strings in [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format, expressed in UTC - Dates are strings in YYYY-MM-DD [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format, expressed in the local time of the listing `exchange` - Enums are strings, with each value corresponding to a predefined set of options listed in the associated lookup table | Field [//]: # (column: sortable) | Type [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `security_id` | String | Security level numerical ID. Can be used to link all multiple listings together. | | `event_id` | String | Event identifier unique at the event level. Links to the corporate actions `event_id`. | | `event` | Enum | Event type. See [EVENT](https://databento.com/docs/conventions/reference-data-enums.md#event) enum. | | `issuer_name` | String | The issuer name (sourced from corporate action). | | `security_type` | Enum | The security type. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `primary_exchange` | Enum | Exchange code for the primary security. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `exchange` | Enum or null | Exchange code for the listing. Equivalent to the MIC but more stable as MIC might not be available in a timely fashion. Also note that the MIC can change but the `exchange` will remain the same. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `operating_mic` | Enum | Market Identifier Code (MIC) as an ISO 10383 string. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `symbol` | String or null | The query input symbol which matched the record. | | `nasdaq_symbol` | String or null | The security's symbol in the Nasdaq convention (e.g. `BF.B`). See [Symbology](https://databento.com/docs/schemas/security-master.md#symbology) on the security master schema page for known limitations on `USNYSE` listings. | | `local_code` | String or null | The listing exchange's native symbol form, as published by the upstream reference-data vendor. Convention varies by listing exchange. Either an alpha string, or a number. | | `local_code_resulting` | String or null | Resultant local code when applicable/known. | | `isin` | String or null | ISIN global level identifier as an ISO 6166 string (sourced from corporate action). | | `isin_resulting` | String | Resultant ISIN when applicable/known (sourced from corporate action). | | `us_code` | String or null | US domestic CUSIP (sourced from corporate action). | | `status` | String | Adjustment status. Can be (**A**)pply, (**R**)escind, (**P**)ending. | | `ex_date` | Date | Date from which the event is effective. | | `factor` | Number | Adjustment factor to apply. | | `close` | Number or null | Closing price on the `ex_date`. | | `currency` | Enum or null | Currency for the closing price as an ISO 4217 string. See [CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen) enum. | | `sentiment` | Number | Market sentiment - the market's reaction to the event. Simply the previous close divided by today's open. Only correct if factor calculation requires previous close. | | `reason` | Number | Reason/type of event, used to distinguish between different event types. See [Adjustment factors](https://databento.com/docs/datasets/adjustment-factors.md) dataset guide. | | `gross_dividend` | Number or null | The amount of the dividend before any taxes or fees are deducted. This value represents the total dividend declared by the company. | | `dividend_currency` | String or null | The currency in which the dividend is paid as an ISO 4217 string. See [CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen) enum. | | `frequency` | String or null | The frequency at which the dividend is paid. See [FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq) enum lookup table. | | `option` | Number | The choice or option number associated with the event, often used when shareholders are given multiple options for how they would like to receive the dividend or other corporate action benefit (either cash, or script). See [option field](https://databento.com/docs/datasets/adjustment-factors.md#option-field) description. | | `detail` | String | A human-readable description of the event. | | `ts_created` | Timestamp | The timestamp (UTC) the record was added by Databento. | --- Source: https://databento.com/docs/schemas/security-master # Security master **Security master** provides up-to-date (latest) or point-in-time (PIT) static information on securities across international markets for 860,000+ listed and delisted securities. This provides a means for resolving securities attributes, identification, symbology, and informational challenges efficiently. Some common use cases for the security master include research, backtesting, compliance and report generation, risk management, and keeping portfolios current. > **See also** > > [Reference data enums](https://databento.com/docs/conventions/reference-data-enums.md) for a lookup table of possible enumeration field values. ### Fields - Timestamps are strings in [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format, expressed in UTC - Dates are strings in YYYY-MM-DD [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format, expressed in the local time of the listing `exchange` - Enums are strings, with each value corresponding to a predefined set of options listed in the associated lookup table | Field [//]: # (column: sortable) | Type [//]: # (column: sortable) | Description [//]: # (column: sortable) | | `ts_record` | Timestamp | The timestamp (UTC) the record last changed. | | `ts_effective` | Timestamp | The timestamp (UTC) the record details are effective from. | | `listing_id` | String | Unique listing numerical ID. Concatenation of a sequence number and the `listing_group_id`. The sequence number will usually be 1 unless additional listings exists for the same security in the same market, usually as a result of secondary trading currencies. | | `listing_group_id` | String | Groups all listings for the same security on a specific exchange, often in different trading currencies. | | `security_id` | String | Security level numerical ID. Can be used to link all multiple listings together. | | `issuer_id` | String | Issuer level numerical ID. Can be used to link all securities of a company together. | | `listing_status` | Enum | Listing status code. Indicates the listing activity status at market level. See [LISTSTAT](https://databento.com/docs/conventions/reference-data-enums.md#liststat) enum. | | `listing_source` | Enum | Indicates if the listing level data in the record is (M)ain or (S)econdary. See [LISTSOURCE](https://databento.com/docs/conventions/reference-data-enums.md#listsource) enum. | | `listing_created_date` | Date | Listing creation date. | | `listing_date` | Date or null | Listing date. | | `delisting_date` | Date or null | Delisting date. | | `issuer_name` | String | Issuer name. | | `security_type` | Enum | The security type. See [SECTYPE](https://databento.com/docs/conventions/reference-data-enums.md#sectype) enum. | | `security_description` | String | The security description. | | `primary_exchange` | Enum or null | Exchange code for the primary security. Empty if unknown. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `exchange` | Enum | Exchange code for the listing. Equivalent to the MIC but more stable as MIC might not be available in a timely fashion. Also note that the MIC can change but the `exchange` will remain the same. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `operating_mic` | Enum or null | Market Identifier Code (MIC) as an ISO 10383 string. See [EXCHANGE](https://databento.com/docs/conventions/reference-data-enums.md#exchange) enum. | | `symbol` | String or null | The query input symbol which matched the record. | | `nasdaq_symbol` | String or null | The security's symbol in the Nasdaq convention (e.g. `BF.B`). See [Symbology](https://databento.com/docs/schemas/security-master.md#symbology) for known limitations on `USNYSE` listings. | | `local_code` | String or null | The listing exchange's native symbol form, as published by the upstream reference-data vendor. Convention varies by listing exchange. Either an alpha string, or a number. | | `isin` | String or null | ISIN global level identifier as an ISO 6166 string. | | `us_code` | String or null | US domestic CUSIP. | | `bbg_comp_id` | String or null | Bloomberg composite global ID. | | `bbg_comp_ticker` | String or null | Bloomberg composite ticker. | | `figi` | String or null | Bloomberg FIGI, that is exchange level ID. | | `figi_ticker` | String or null | Bloomberg exchange level ticker. | | `fisn` | String or null | **F**inancial **I**nstrument **S**hort **N**ame. | | `lei` | String or null | **L**egal **E**ntity **I**dentifier. | | `sic` | String or null | **S**tandard **I**ndustrial **C**lassification **C**ode. | | `cik` | String or null | **C**entral **I**ndex **K**ey. | | `gics` | String or null | **G**lobal **I**ndustry **S**tandard **C**lassification. | | `naics` | String or null | **N**orth **A**merican **I**ndustrial **C**lassification **S**ystem. | | `cic` | String or null | **C**omplementary **I**dentification **C**ode. | | `cfi` | String or null | **C**lassification of **F**inancial **I**nstruments as an ISO 10962 string. | | `incorporation_country` | Enum or null | Country of incorporation code of the issuer as two letter ISO 3166-1 alpha-2 string. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `listing_country` | Enum or null | Listing country as two letter ISO 3166-1 alpha-2 string. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `register_country` | Enum or null | Register country as two letter ISO 3166-1 alpha-2 string. See [CNTRY](https://databento.com/docs/conventions/reference-data-enums.md#cntry) enum. | | `trading_currency` | Enum or null | Trading currency as an ISO 4217 string. See [CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen) enum. | | `multi_currency` | Boolean | True if there is currently more than one listing in the market, otherwise false. | | `segment_mic_name` | String or null | Market Segment Name. Not the MICSEG lookup name but used when no segment MIC is available. | | `segment_mic` | Enum or null | Market Identifier Code (MIC) as an ISO 10383 string. See [MICSEG](https://databento.com/docs/conventions/reference-data-enums.md#micseg) enum. | | `structure` | Enum or null | Security structure code. See [STRUCTCD](https://databento.com/docs/conventions/reference-data-enums.md#structcd) enum. | | `lot_size` | Number or null | Lot Size. Indicates the minimum number of shares that can be acquired in one transaction. | | `par_value` | Number or null | Par value amount. | | `par_value_currency` | Enum or null | Par value currency as an ISO 4217 string. See [CUREN](https://databento.com/docs/conventions/reference-data-enums.md#curen) enum. | | `voting` | Enum or null | Voting or non-voting rights. See [VOTING](https://databento.com/docs/conventions/reference-data-enums.md#structcd) enum. | | `vote_per_sec` | Number or null | Number of votes per security. | | `shares_outstanding` | String or null | Shares outstanding. | | `shares_outstanding_date` | String or null | Effective date for `shares_outstanding`. | | `ts_created` | Timestamp | The timestamp (UTC) the record was added to database. | ### Identifiers Identifiers for listing, security and issuer are grouped at the beginning of each record, which enables clearer comparisons of where a single security is associated with many individual listings across different exchanges. There is one `primary_exchange` for a security, and it may then be listed on multiple secondary `exchange`s. ### Symbology `nasdaq_symbol` is normalized to the Nasdaq convention. `local_code` keeps the listing exchange's native symbol; its convention varies by listing exchange. For reliable linkage to time series data, identify the security by `isin`, `us_code` (CUSIP), or `figi`, then submit the time series query with the matching symbol and `stype_in`. > **See also** > > [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the convention reference and per-dataset map. > **Warning** > > For symbols with suffix codes (class shares, warrants, preferreds, etc.) listed on `exchange=USNYSE`, the upstream reference-data vendor delivers the symbol in a no-delimiter form that the normalization step cannot recover. > In these cases, `nasdaq_symbol` is left in the no-delimiter form (e.g. `BRKB` for Berkshire Hathaway Class B, `HEIA` for Heico Class A) and does not match the time series `raw_symbol` for the same security. > Cross-listings of the same security on other US exchanges (e.g. `USCIN`, `USAMEX`) are unaffected; matching by `isin`, `us_code`, or `figi` is the reliable workaround.