Standards and conventions

Common fields, enums and types

Publishers, datasets, and venues

We use a few different terms to describe our data:

• A dataset is a source of data.
• A venue is an exchange, OTC market (e.g., ATS, ECN) or reporting entity.
• A publisher is a specific venue from a specific dataset.

See also

Read our Venues and publishers section for a more detailed explanation.

Publisher identifiers

All of our schemas include a publisher_id field, which is a unique numeric ID assigned by Databento to each publisher. A full list of publishers can be found using the metadata.list_publishers endpoint.

Venue and dataset identifiers

Each publisher is also assigned a string identifier (e.g., OPRA.PILLAR.XCBO), composed of two parts:

• Dataset ID (e.g., OPRA.PILLAR). This is used as the dataset argument in any API or client method. Dataset IDs can be found on the Databento portal on each dataset's details page or via the metadata.list_datasets endpoint.
• Venue (e.g., XCBO). For most markets, this is its ISO 10383 MIC code, which is guaranteed to be four characters long. For entities without a MIC code, this string is arbitrarily assigned by Databento and will also be four characters long.

Instrument identifiers

All of our schemas contain an instrument_id field which is a numeric ID that maps to a given instrument. In most cases, this numeric ID is assigned by the publisher. For publishers that do not assign this value, we create a synthetic mapping for it.

instrument_id is only guaranteed to be unique within a given day. Some publishers provide a different instrument ID on different days for the same underlying instrument. Other publishers may use the same instrument ID for different underlying instruments at different points in time.

Depending on the use case, it may be easier to work with other symbology types such as raw_symbol. Our symbology documentation outlines the various symbology types we support.

Timestamps

All the timestamps in our data are expressed as the number of nanoseconds since the UNIX epoch, i.e. UNIX timestamps. All timestamp fields are prefixed with ts_. Some of our timestamps are encoded as the difference, i.e. delta, relative to another timestamp. Such timestamp fields are suffixed with _delta.

We provide four types of timestamps, through the following fields:

• Event timestamps, ts_event
• Publisher sending timestamp, ts_in_delta
• Databento receive timestamp, ts_recv
• Databento sending timestamp (live only), ts_out

UNDEF_TIMESTAMP (UINT64_MAX, 18446744073709551615) is used to denote a null or undefined timestamp.

The event and publisher sending timestamps are provided by the publisher (or market), and we provide their original values without any adjustment.

ts_event

Most users will only need the event timestamp, i.e. ts_event. For market data, this represents the time that the event is received by the matching engine (tag 60 in FIX encoding).

The exact location where this timestamp is taken varies with matching engine architecture of each market. Some markets will handle different subsets of instruments on independent order gateways, while other markets will load balance the same subset of instruments across independent order gateways. Some markets take the event timestamp at the time it is received on the independent order gateways, while others may take this timestamp at the time it reaches a FIFO matching queue. In the former case, the clocks on independent order gateways are often not properly synchronized to the same clock source. Since we do not adjust the publisher's timestamps, any non-monotonicity in the original data will remain.

ts_in_delta

The publisher sending timestamp represents the time when the data message associated with an event is sent (tag 52 in FIX encoding). We encode this information in ts_in_delta, which expresses the number of nanoseconds between the Databento receive timestamp (ts_recv) and the publisher sending timestamp. To get the sending timestamp itself, simply subtract ts_in_delta from ts_recv. Since the publisher and Databento are not necessarily synchronized to the same clock source, ts_in_delta may be negative.

ts_in_delta is a 32-bit signed integer. The minimum will clamp to INT32_MIN and the maximum will clamp to INT32_MAX, even if the true value exceeds these limits.

Some markets do not provide both match event timestamps and sending timestamps. Often, they will provide only one of the two. In such cases, we take it that the event timestamp and sending timestamp assume the same value. As such, ts_event will be the provided timestamp and ts_in_delta will be equal to the difference between ts_recv and ts_event.

ts_recv

Unless otherwise specified, Databento receive timestamps, i.e. ts_recv, are synchronized against UTC with sub-microsecond accuracy. Moreover, these receive timestamps are always guaranteed to be monotonic for any given symbol.

These receive timestamps rely on hardware timestamping on the network adapter and are synchronized against a GPS clock source using PTP. The clock is corrected by slewing the time, which prevents discrete jumps backwards in time. In other words, our local receive timestamps are guaranteed to be monotonic for any given symbol.

ts_recv is also adjusted for leap seconds. The local receive timestamp is not immediately adjusted intraday when a leap second is introduced. Instead, the leap second update is applied at the end of the market session.

ts_out

For live data, we optionally include a timestamp of our data before it leaves our data gateways. This information is encoded as ts_out. Both ts_out and ts_recv are synced to the same GPS clock source. Subtracting ts_recv gives the number of nanoseconds spent in our system.

Index timestamp

All schemas have a primary timestamp that should be used for sorting records as well as indexing into any symbology data structure. This index timestamp will be ts_recv if it exists in the schema, otherwise it will be ts_event.

When requesting historical data, the data will be filtered based on the index timestamp.

When requesting data in CSV and JSON encodings, the first field will be set to this index timestamp. Additionally, for schemas that contain ts_recv, the second field will be set to ts_event.

Encodings

We support DBN, CSV, and JSON encodings for our data. DBN is an extremely fast message encoding and storage format for normalized market data. All official Databento client libraries use DBN under the hood, both as a data interchange format and for in-memory representation of data. DBN is also the default encoding for all Databento APIs, including live data streaming, historical data streaming, and batch flat files.

Our batch download system also supports CSV and JSON encodings.

Time zone

By default, all of our data is set in UTC time zone. Likewise, our site displays all dates and times in UTC by default.

Dates and times

We use the ISO 8601 date and time format to express dates and times used as parameters to our APIs. All dates and times used as parameters are in UTC by default.

The "reduced precision" concept in the ISO 8601 standard allows for dates and times to be represented with varying levels of detail. Any number of values may be dropped from any of the date and time representations, but in the order from the least to the most significant. For example, "2024-05" corresponds to "2024-05-01T00:00:00".

Any parameter that takes an ISO 8601 timestamp can instead be given a timestamp in nanoseconds since the UNIX epoch, as described in the above section.

All of our timestamp parameters are start-inclusive and end-exclusive.

Forward filling end parameters

For our APIs that take an optional end parameter as an ISO 8601 string, we will implement the following behavior when the end parameter is not provided:

We will forward fill any date or time components of the associated start parameter that are omitted. This "rounds up" the start timestamp for use as the end timestamp, and is done for more concise usage.

Examples of this behavior are shown below.

Info

We will only forward fill timestamps with less than one-second resolution.

For example, a query for the entire month of March 2024 can be specified with start="2024-03" without an end.

rtype

An rtype or record type is an unsigned 8-bit discriminant in the header of every DBN record that indicates the type of record structure. Each schema has one rtype and by extension one record structure associated with it.

Info

Some rtypes are not associated with a schema and are only present in live data.

Prices

Prices are expressed as signed integers in fixed-precision format, whereby every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001. For example, a price of 5411750000000 corresponds to 5411.75 (decimal format).

When requesting data via batch download in CSV and JSON encodings, you can optionally choose for prices to be returned in decimal format. If you are requesting data using the online portal, you can select Decimal prices in the Advanced customization section. Otherwise, you can specify the pretty_px parameter in batch.submit_job using the client libraries.

Additionally, our client libraries support functionality to view prices in decimal format.

In certain scenarios—such as calendar spreads in futures—prices can be negative.

UNDEF_PRICE is used to denote a null or undefined price. It will be equal to 9223372036854775807 (INT64_MAX) when using the fixed-precision integer format. When expressed in decimal format, it will be equal to null in JSON, or "" (an empty string) in CSV.

Side

The side field contains information about the side of an order event. It's meaning will vary depending on the action field.

• When action is Trade:

  • A - The trade aggressor was a seller
  • B - The trade aggressor was a buyer
  • N - No side specified

• When action is Fill:

  • A - A resting sell order was filled
  • B - A resting buy order was filled
  • N - No side specified

• When action is Add, Modify, and Cancel:

  • A - A resting sell order updated the book
  • B - A resting buy order updated the book
  • N - No side specified

• When action is cleaR book, side will always be N

side can be N in the following cases:

• The source does not disseminate a side for trades.
• Trades happening during opening and closing auctions
• Trades against non-displayed orders
• Trades involving implied orders
• Off-exchange trades

The Venues and datasets section provides more information regarding the specific cases for each dataset where no side will be specified.

Action

The action field contains information about the type of order event contained in the message.

Flags

The flags field is a bit field that contains information about the message. Multiple flags can be set on a single message.

The meaning of each bit is as follows:

Top-of-book datasets

Some datasets are built on feeds from vendors that only provide top-of-book information (best bid and offer). Top-of-book messages are normalized into a pair of MBO records with the Add action and the F_TOB flag (0x40, 64) set. Typically for these datasets, there is no information available about the passive side of trades, so there are no Fill records and the side of the Trade record is always set to None.

The removal of a price level is normalized as an Add action with a size of 0 and price of UNDEF_PRICE (INT64_MAX, 9223372036854775807) or NaN in Python. This indicates there's currently no quotes for that side.

Other schemas, such as MBP-1, Trades, and OHLCV remain the same for top-of-book datasets.

Market-by-price datasets

Some datasets are built on feeds from vendors that only provide market-by-price information (with limited depth).

Messages adding/modifying/deleting a price level are normalized into an MBO record with the Add/Modify/Cancel action, with the size field containing the full quantity at that level and the F_MBP flag (0x10, 16) set. A price level can be identified from the combination of the side and the price. The order_id field should be ignored for those messages.

If the upstream feed has a maximum depth, an additional record with Cancel action will be sent whenever a price level falls outside the maximum depth - even if there are still orders at that price level.

Typically for these datasets, there is no information available about the passive side of trades, so there are no Fill records.

MBP-10 will only include depth up to the depth provided by the publisher. The remaining levels will always be empty.

Other schemas, such as Trades and OHLCV are otherwise unaffected.

Normalization

Normalization refers to the process of exporting data in their various source formats to a single, unified format. Such a unified format is often called a normalization format (or normalized schema). The primary reason for normalizing market data is to abstract away differences between source formats, making the data easier to work with.

The normalization process is one of the most likely places where inaccuracies or data errors are introduced. This article describes these issues, the trade-offs for addressing them, and the reasons behind the design of Databento's normalization schema.

Examples of normalized data

• For example, Nasdaq's proprietary TotalView data feed has a protocol with its own message format and provides market-by-order data, while IEX's proprietary TOPS data feed has a completely different protocol with another message format and provides top-of-book data. These are examples of raw data formats.
• When you consume market data from a data redistributor's feed, the redistributor will have its own protocol and message format, distinct from the venues'. This is an example of normalized data.
• The most sophisticated trading firms will generally collect data directly from their sources and normalize them to a proprietary format.

Common issues found in normalized market data

There are many ways in which normalization can introduce data errors, lossiness or performance issues.

Our normalization schema is designed to mitigate most of these issues.

Why use normalized data?

Though it may seem counterintuitive, some degree of lossiness introduced during normalization can be preferable.

A normalized schema that has too many data fields, as a result of trying to preserve information from too many different source schemas, is hard to use.

Here are some ways in which lossiness can be useful:

• Discarding unnecessary data fields can reduce storage and bandwidth requirements, and improve application performance. For example, many strategies execute at time scales where extra timestamps are unnecessary.
• Most status or reference data events are irrelevant for any given business use case. For example, many users only trade during the regular market session and their applications do not need to be aware of special matching conditions that are more typically found outside of regular hours or during pre-market.
• Floating point prices can be easier to use and the modeling error introduced by them could be negligible compared to other, more likely sources of error for the given use case.
• Order book snapshots can be unnecessary on liquid products whose orders turnover very quickly, as pre-existing orders in the snapshot will eventually be filled or canceled - a process which is commonly referred to as natural refresh.

It should be noted that normalized data is not necessarily going to be smaller or have a simpler specification than the source data. If your use case only requires a single dataset, there may be complexity using normalized data whose schema was designed to accommodate differences between multiple datasets.

We normalize to our proprietary Databento Binary Encoding DBN.

Symbology

Financial datasets usually contain symbols or product identifiers. The mapping of symbols to their corresponding product names can be extracted from our definition schema as well as the metadata packaged with our data.

Databento supports four symbology types, also referred to as stypes. They are: raw_symbol, instrument_id, parent, and continuous.

We include methods for mapping between symbology types and resolving symbols under the symbology family of methods.

We do not retroactively reassign symbols in our historical data. Symbols found in our historical data are exactly as they appeared in the live data at the original event time. For example, if a stock symbol was changed due to a corporate action, we preserve the original symbol for data before the event and the new symbol for data after the event. This approach guarantees that the historical data looks identical to the live data at the original time, and encourages our users to write their integration in a manner that handles historical and live data in the same way.

A symbol can be reused and point to two different instruments on two different dates.

Supported symbology combinations

When requesting data, such as with the timeseries.get_range or batch.submit_job endpoints, an input and output symbology type can be specified. Not all symbology types are supported for output and some symbology types are not available in certain datasets. The following table contains all currently supported stype_in and stype_out combinations per dataset.

Raw symbol

Raw symbols are the original string symbols used by the publisher in the source data. This can be useful for environments with direct market connectivity. Examples of raw symbols include AAPL, ESH3, etc.

This symbology is used by setting the stype_in=raw_symbol parameter in the API.

Instrument ID

Instrument IDs are the unique numeric ID assigned to each instrument by the publisher. Most venues use such numeric IDs under the hood. Numeric IDs have the benefit of taking less space than most string symbols. However, numeric IDs can be difficult to work with, especially as some publishers remap them daily.

This symbology is used by setting the stype_in=instrument_id parameter in the API.

Parent

Parent symbology is a smart symbology feature that allows you to easily refer to groups of related symbols using a single root symbol. The root symbols are sourced from the asset field of the definition schema. All futures for a root symbol can be referenced using the parent symbol [ROOT].FUT, for options: [ROOT].OPT. For example, ES.FUT refers to all E-mini S&P 500 futures and futures spreads and ES.OPT refers to all quarterly E-mini S&P 500 options and option spreads.

The type of instrument will be specified in the instrument_class field. When requesting data using futures parent symbology, this field will indicate whether the instrument is a future or futures spread. When requesting data using options parent symbology, it will indicate whether the instrument is a call or put. A full list of variants can be found in the instrument class documentation.

This symbology is used by setting the stype_in=parent parameter in the API.

Continuous

Info

Our continuous contract symbology is a notation that maps to an actual, tradable instrument on any given date. The continuous contract prices returned are the original, unadjusted prices. We don't create a synthetic time series by back-adjusting the prices to remove jumps during rollovers.

Continuous contract symbology is a smart symbology feature that allow a single symbol to refer to different instruments over time. For example, continuous contract symbology allows you to query a single symbol that changes or rolls forward before expiration.

For futures outrights, we use the format [ROOT].[ROLL_RULE].[RANK] to refer to continuous contracts that change over time according to a roll rule and rank. Like with parent symbology, the root symbol corresponds with the asset field of the definition schema.

RANK is a zero-indexed integer, and ROLL_RULE is either c, n, or v from the table below.

This symbology is used by setting the stype_in=continuous parameter in the API. It is not currently possible to select continuous contracts through our web portal.

All symbols

It is possible to request all symbols within a dataset without providing them explicitly. This is done by specifying ALL_SYMBOLS with stype_in=raw_symbol or stype_in=parent in the API.

When requesting all symbols using timeseries.get_range symbology data is not provided. This means that for the CSV and JSON encodings the parameter map_symbols=True is not allowed. For the DBN encoding, the metadata header will not contain symbology mappings.

When requesting all symbols using batch.submit_job, the symbology.json support file will not contain symbology mappings.

Symbology.resolve endpoint

Databento offers symbology resolution for free in our symbology.resolve endpoint and in our client libraries. This endpoint can be used to request mappings from one symbology type to another and contains all the data necessary to perform these conversions.

symbology.json

For batch downloads, symbology information for the job is contained in the symbology.json support file. Below is a sample symbology.json file. This is a batch support file that is included in every batch download request. It's contents are directly obtained from the symbology.resolve endpoint:

{
   "result": {
      "ES.c.0": [
         { "d0": "2023-01-01", "d1": "2023-03-19", "s": "206299"},
         { "d0": "2023-03-19", "d1": "2023-06-01", "s": "95414"}
      ]
   },
   "symbols": ["ES.c.0"],
   "stype_in": "continuous",
   "stype_out": "instrument_id",
   "start_date": "2023-01-01",
   "end_date": "2023-06-01",
   "partial": [],
   "not_found": [],
   "message": "OK",
   "status": 0
}

Examining this sample we can see a requested mapping of the stype_in ("continuous") symbol ("ES.c.0") to stype_out ("instrument_id") over the date range start_date ("2023-01-01") to the end_date ("2023-06-01").

We can check the message ("OK") and status (0) fields to confirm that our request was successful over the entire date interval. Additionally, the not_found and partial fields are empty.

Most importantly, the result field contains our symbology mappings keyed by each input symbol. Each symbol entry in the result mapping will contain a list of entries. These entries contain a start date in the d0 field, and an end date in the d1 field for the mapping. The s field contains the output symbol.

SymbolMappingMsg

Databento's live data publishes symbology information using the SymbolMappingMsg. This message will always contain the input symbol and the resolved output symbol. The record header of the SymbolMappingMsg will always contain the instrument_id. See our DBN encoding article for more information on our binary format.

Databento Binary Encoding

Databento Binary Encoding (DBN) is an extremely fast message encoding and storage format for normalized market data. The DBN specification includes a simple, self-describing metadata header and a fixed set of struct definitions, which enforce a standardized way to normalize market data.

All official Databento client libraries use DBN under the hood, both as a data interchange format and for in-memory representation of data. DBN is also the default encoding for all Databento APIs, including live data streaming, historical data streaming, and batch flat files.

Getting started with DBN

The easiest way to get started with DBN is through any of the official Databento client libraries, which support reading and writing DBN files. See:

• The DBNStore.from_file and DBNStore.to_file methods for Python
• The DbnFileStore::Replay method for C++
• The AsyncDbnStore::from_file method, and databento::dbn::encode module for Rust

Other resources are also available:

• The dbn Rust crate is the reference implementation of DBN and provides a library for decoding and encoding DBN, and also for converting from DBN to CSV and JSON.
• dbn-cli is a command line tool that can be used to read DBN files; transcode DBN to CSV or JSON; print the output, or write it to disk. You can install it with cargo install dbn-cli.

Why should you use DBN?

The key advantages of using DBN are:

• End-to-end. DBN can be used to store and transport normalized data across all components of a typical trading system. It can fulfill the requirements of a file format for efficient storage; a message encoding for fast real-time streaming, and an in-memory representation of market data for a low latency system—all at once. This simplifies your trading system and eliminates the use of multiple serialization formats. It also ensures that market data is immutable, lossless, and consistent as it passes between components.
• Use the same code for historical and live. Our client libraries exploit the end-to-end aspect of DBN and allow you to use the same code in historical and live; you can write an event-driven trading platform that runs the exact same code in backtest and production trading.
• Zero-copy. DBN data is structured the same way whether in-memory, on the wire, or on disk. Thus, the data gets read and written directly as-is, without costly encoding or decoding steps that move the data into CPU and back out.
• Symbology metadata. The DBN protocol includes a lightweight header that provides metadata for interpreting and using the market data payload, such as symbology mappings, so a DBN file is self-sufficient for many use cases.
• Highly compressible. DBN strictly uses fixed lengths and offsets for all fields. This layout enables typical compression algorithms, such as zstd and lz4, to achieve high compression ratios.
• Optimized for modern CPUs. The predictable layout of DBN records also allows for highly-optimized, sequential access patterns that take advantage of instruction pipelining and prefetching on modern CPUs. The struct definitions are also deliberately designed so that most records fit into a single cache line.
• Extremely fast. DBN achieves extremely fast speeds on reads and writes. Most use cases of DBN are compression-bound or I/O-bound while only using a single CPU core. DBN has been used in environments with 6.1 microseconds median internal latency; we've also seen user-reported benchmarks of full order book replay over 19.1 million messages per second using our C++ client library on a Google Cloud VM.
• Normalization format. Using DBN also automatically means you're adopting its normalization format. While there are many ways to normalize data, our team arrived at these best practices after many years of combined experience at top-tier trading firms and integrating dozens of trading venues. For example, DBN allows you to replay full order book data at I/O bound, backtest with passive orders in precise sequence, and losslessly achieve much of what's possible with raw packet captures with several factors of improvement in speed and storage requirements.

Layout

A valid DBN stream or file has two parts, beginning with metadata, and immediately followed by records.

The following diagram shows the field layout of the DBN encoding:

Version 1

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             magic string = "DBN"              |  version = 1  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            length                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                            dataset                            +
|                                                               |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             schema            |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                       start (UNIX nanos)                      |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                        end (UNIX nanos)                       |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                      limit (max records)                      |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                            reserved                           |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |   stype_in    |   stype_out   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    ts_out     |                                               |
+-+-+-+-+-+-+-+-+                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                 reserved (47 bytes of padding)                |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    schema_definition_length                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  schema_definition (variable)                 |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         symbols_count                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       symbols (variable)                      |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         partial_count                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       partial (variable)                      |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        not_found_count                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      not_found (variable)                     |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         mappings_count                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      mappings (variable)                      |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-end metadata; begin body--+-+-+-+-+-+-+-+-+-+
|                            records                            |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Versions 2 and above

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             magic string = "DBN"              |    version    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            length                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                            dataset                            +
|                                                               |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             schema            |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                       start (UNIX nanos)                      |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                        end (UNIX nanos)                       |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                      limit (max records)                      |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |   stype_in    |   stype_out   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    ts_out     |        symbol_cstr_len        |               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               |
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                 reserved (53 bytes of padding)                |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    schema_definition_length                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  schema_definition (variable)                 |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         symbols_count                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       symbols (variable)                      |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         partial_count                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       partial (variable)                      |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        not_found_count                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      not_found (variable)                     |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         mappings_count                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      mappings (variable)                      |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-end metadata; begin body--+-+-+-+-+-+-+-+-+-+
|                            records                            |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Metadata

Metadata is included at the beginning of every DBN stream or file. Basic information is found at the start of the metadata, followed by optional symbology mappings.

The metadata contains all of the parameters needed to construct a request for the exact same data via Databento's historical API. Likewise, if you fetched DBN-encoded data from Databento's historical API, the metadata header will contain the parameters of your original request.

The following table describes the metadata fields, in the order of appearance. All fields are little-endian.

where SymbolMapping has the following structure:

and where MappingInterval has the following structure:

Records

The metadata is immediately followed by DBN records. A valid DBN stream or file contains zero or more records.

All records begin with the same 16-byte RecordHeader with the following structure:

See the Schemas and data formats section for a full list of fields for the record associated with each schema.

Versioning

We use the version field in the metadata header to signal changes to the structure of record types and metadata.

Version 2

The following was changed:

• Metadata:
  • Sets version to 2
  • Adds symbol_cstr_len field
  • Rearranges padding
  • The fixed-length strings for symbology are now defined to have symbol_cstr_len characters (currently 71), whereas in version 1 they always had 22
• InstrumentDefMsg (definition schema):
  • raw_symbol now has symbol_cstr_len characters (71)
  • Rearranges padding
• SymbolMappingMsg (live symbology):
  • stype_in_symbol and stype_out_symbol now have symbol_cstr_len characters (71)
  • Adds stype_in and stype_out fields
  • Removes padding
• ErrorMsg (gateway errors in live)
  • Adds space to err for longer error messages
  • Adds code and is_last fields
• SystemMsg (non-error gateway messages in live)
  • Add space to msg for longer messages
  • Adds code field

Version 3

This set of changes adds support for strategy legs to the definition schema and an expanded quantity field in the statistics schemas.

• Added 8-byte alignment padding to the end of metadata
• Expanded quantity to 64 bits in StatMsg (statistics schema)
• InstrumentDefMsg (definition schema):
  • A definition record will be created for each strategy leg
  • Adds the following leg fields:
    • leg_count
    • leg_index
    • leg_instrument_id
    • leg_raw_symbol
    • leg_side
    • leg_underlying_id
    • leg_instrument_class
    • leg_ratio_qty_numerator
    • leg_ratio_qty_denominator
    • leg_ratio_price_numerator
    • leg_ratio_price_denominator
    • leg_price
    • leg_delta
  • Expands asset to 11 bytes
  • Expands raw_instrument_id to 64 bits to support publishers that use larger IDs
  • Removal of statistics-schema related fields trading_reference_price, trading_reference_date, and settl_price_type
  • Removal of the status-schema related field md_security_trading_status

Info

CSV and JSON are affected by the new fields.

Currently, version 2 is used for the IFEU.IMPACT and NDEX.IMPACT datasets. The DBN crate and client libraries will continue to support decoding version 1 data.

Upgrading to versions

DBN version 1 files can be upgraded to version 2 with the dbn CLI tool by passing the --upgrade or -u flag.

dbn version1.dbn --output version2.dbn --upgrade

Comparison with other encodings and formats

DBN is designed specifically for normalized market data. It adopts a fixed set of struct definitions, also called message schemas, for this purpose. It's important to note that DBN is not a general-purpose serialization format like Simple Binary Encoding (SBE) or Google Protocol Buffers (protobufs), which provide a flexible schema definition language. Unlike these formats, DBN doesn't offer tools for generating decoders or encoders from any user-specified schema.

When comparing DBN to most encodings or serialization formats, a critical difference is that DBN is a zero-copy encoding. Moreover, what makes DBN most unique is that it's simultaneously intended for three common use cases in a trading system: file format, real-time messaging format, and in-memory representation. This is a very specific convergence of use cases that manifests frequently in financial trading systems.

Other encodings or formats typically used in situations where DBN would be a suitable replacement include:

• SBE
• Apache Parquet
• Apache Arrow
• Feather

Comparing to these, DBN is intended to be good at all three abovementioned use cases of a trading system—so you don't have to mix multiple serialization formats in one system—while the others tend to excel only in one or two use cases.

A single format for all use cases carries a more important benefit for trading than just the performance upside that comes with minimizing copies; it ensures that market data is immutable when it passes through your trading system. The following diagram helps you visualize the difference between a potential trading system that uses DBN compared to a typical trading system that doesn't.

Typical trading environment

Trading environment using DBN

Immutable market data makes it easy to align live trading with post-production logs and historical data; it makes it easy to use the same code for live trading, backtesting, and exploratory research; it also makes it easy to write GUIs that need accurately synchronize order events with market data events, especially market data events that triggered those order events. In short, using the same encoding or format everywhere ensures that state is synchronized throughout distributed parts of your trading system.

For these reasons, most mature trading firms eventually end up implementing their own proprietary encoding that resembles DBN.

The following table summarizes other key comparisons:

Frequently asked questions

Isn't this basically a bunch of raw structs? What's so special about this?

Yes, pretty much! And it's not exactly novel—in our experience, most top-tier trading firms will have something similar already implemented, along with proprietary tooling to support it. The significance of DBN is that we're open sourcing the whole toolset, with many best practices for normalization and performance optimization, so that you don't have to reinvent the wheel.

Another purpose of DBN is that it provides a standardized data interchange format that can be used for high-throughput, low latency streaming between a data provider like us and you. At the time of the initial release of DBN, we're not aware of any data provider that adopts a binary flat file or messaging format with similar zero-copy semantics.

Even if you don't want to use DBN exactly, it's a lightweight specification so it's easy to mimic some of its practices or fork our reference implementation for your own use case.

When should you not use DBN?

• When you depend on many tools and frameworks in the Apache ecosystem. Many data processing tools have native support for Apache Parquet and Arrow. However in our experience, mature trading environments generally use fewer general-purpose computation frameworks. In these cases, DBN is still an excellent as an interchange format for receiving and storing data from Databento, and we still support converting DBN to pandas dataframes and hence Arrow format.
• If you don't use Databento, only ever plan on trading on one trading venue, have already written parsers for the raw feeds, and have direct extranet connectivity, then there's a strong argument for just using the original wire protocol like ITCH and even rolling your own, thinner normalization format.
• If you have an academic or toy project and only plan on working with historical data. Many such projects employ relatively small amounts of data and don't require live data. In these circumstances, it makes sense to just store the data in your own, thinner binary format or a binary format with some structure like Parquet or HDF5.
• If you have to support many teams with one platform, with different trading styles and business functions, of which many of them only require low frequency data. In this situation, the performance benefits of fixed schemas become much less important within your organization, and the flexibility becomes more important. It's also quite likely in this situation that you have to constantly update your normalization format for new exploratory workflows. In those cases, DBN is still an excellent as an interchange format for receiving and storing data from Databento, but your firm will likely benefit from converting DBN data into flexible formats downstream.

Encoding, serialization format, protocol—what's the difference?

There are slight differences in these terms but DBN is all three at once.

One way to look at DBN is that it's an OSI layer 6 presentation protocol, much like SBE—except that DBN is much stricter about message schemas, whereas SBE is flexible.

Data written in accordance to such a protocol can be persisted to disk, so it can also serve as a storage format; it can also be written on the wire as a message encoding or wire format. SBE excels as a message encoding but is less often used as a storage format, whereas DBN accommodates both equally well.

Why is the reference implementation written in Rust? Can I use it?

The majority of Databento's infrastructure is written in Rust, C, and Python, and the reference implementation of DBN is a rewrite of the original C implementation and was originally written with internal use in mind.

Rust's C ABI makes it interoperable across multiple languages with thin bindings, and its memory management model makes it safe and performant. It's simple to integrate the Rust DBN library into your Python application, as seen in our Python client library.

Why is DBN sequential and not column-oriented? Aren't modern column-oriented layouts more optimized for querying?

The sequential layout of DBN makes it more performant for real-time messaging use cases.

Column-oriented formats do have the theoretical potential for more optimization in non-real time use cases, but this depends on the actual implementation. Our reference DBN implementation is heavily optimized and still on par with column-oriented formats like Apache Arrow on common use cases for historical market data.

Am I locked-in to a proprietary binary format here?

No, the DBN reference implementation is open-sourced under the permissive Apache 2.0 License. We also provide transcoders to convert your DBN data into CSV and JSON.

How is DBN being used currently?

We store upwards of 4 PB and over 30 trillion records of normalized historical data in DBN internally at Databento. Every single message that passes through our infrastructure gets encoded in DBN—over billions of messages per day, at single-port peak messaging rates over 60 Gbps, spanning multiple asset classes and over 1.8 million instruments on any given day. It is used for all of our data schemas, including full order book, tick-by-tick trades, top of the book, OHLCV aggregates, venue statistics, instrument definitions, and more.

Most of users, including some of the world's largest hedge funds and market making firms, are already using DBN through our client libraries, putting it through multiple production use cases that involve real-time streaming and historical flat files.

Zstandard (zstd)

Zstandard is a fast, lossless compression algorithm that offers high compression ratios, making it suitable for storing and transmitting large amounts of data.

This compression algorithm is recommended by Databento for all historical streaming and batch downloads, as it minimizes the amount of data transmitted over the network and stored on file systems. You can select Zstandard compression by using the zstd option through the client libraries or the HTTP API.

Once you have obtained Zstandard compressed data from Databento, you have a number of options for decompressing:

• Databento client libraries. Decompression from Zstandard is handled internally. Refer to the API reference for usage details
• dbn-cli. The Databento CLI tool for working with DBN data also includes support for decompressing .zst files
• 7-Zip. To decompress using 7-Zip for Windows, you'll need to install the 7-Zip Zstandard edition. Then right-click on the .zst file, navigate to the 7-Zip menu, and select 'Extract here' or 'Extract to <folder>'
• zstd CLI. Used directly from the command line (described below), or via your own bash scripts

Installing dbn-cli

To install the dbn-cli (dbn) library, ensure you have Cargo (the Rust package manager) installed. Then, run the following command:

cargo install dbn-cli

For more details, visit dbn-cli on crates.io.

Installing Zstandard

The Zstandard (zstd) library can be installed on most operating systems, as detailed below:

macOS

brew install zstd

Linux

Debian/Ubuntu:

sudo apt-get update
sudo apt-get install zstd

CentOS/RHEL:

sudo dnf install zstd

Arch:

sudo pacman -S zstd

Windows

Using Chocolatey:

choco install zstd

Alternatively, you can also download the Zstandard binaries from the official releases page on GitHub and add the directory to your PATH. You can also install from source by following the instructions in the Zstandard GitHub repository.

Verify your installation by checking the version of Zstandard:

zstd --version

Decompressing

Once Zstandard is installed on your machine, you can utilize its command line interface (CLI) for various operations, including decompression and recompression. Below, we'll explore commands specifically for decompressing Zstandard-compressed files in different scenarios.

To view all available options in the Zstandard CLI, you can run the help command:

zstd -h

Decompress to a file

To decompress a file that has been compressed with Zstandard, use the -d (or --decompress) option followed by the filename. For example, if you have a file named data.zst, you can decompress it with the following command:

zstd -d data.zst

This will create a decompressed file named data in the same directory.

Tip

You can decompress a Zstandard-compressed Databento batched data file by running the following command:

zstd -d glbx-mdp3-20231201.trades.csv.zst

This will result in the decompressed file, glbx-mdp3-20231201.trades.csv, being created in the current directory.

If you want to decompress the file to a specific output file, you can use the -o option:

zstd -d data.zst -o decompressed_data.txt

This command will decompress data.zst into a file named decompressed_data.txt.

By default, zstd will not overwrite existing files. If you need to decompress and overwrite any existing files, you can use the -f (force) option:

zstd -d -f data.zst

This will decompress data.zst and overwrite any existing file with the same name as the output.

Decompressing multiple files

You can also decompress multiple files at once by specifying multiple filenames:

zstd -d file1.zst file2.zst file3.zst

Using wildcards for batch decompression

In a directory with multiple .zst files, you can decompress all of them using a wildcard (*). This is particularly useful for batch processing:

zstd -d *.zst

This command will decompress all files in the current directory with the .zst extension.

Tip

Running this command in a batch download directory will decompress all .zst files at once.

Decompressing to standard output

If you want to decompress a file and output the contents directly to the terminal (standard output), use the --stdout option:

zstd -d --stdout data.zst

This could be useful for piping the decompressed output to other programs in bash scripts.

MBO snapshots

An MBO snapshot represents the order book at a specific point in time, including all outstanding buy and sell orders. The snapshot is streamed as a sequence of MBO records to insert new orders in the book (a sequence of Add Actions). The snapshot records preserve the priority order (per instrument) at each price level, enabling accurate reconstruction of the order book.

All snapshot records are marked with the F_SNAPSHOT and F_BAD_TS_RECV flags (ts_recv is set to the snapshot generation timestamp). An instrument's snapshot starts with a cleaR action, followed by zero or more Add Actions.

The snippet below shows the result of a snapshot request for 2 instruments.

MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718539204593519451 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 0 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718539204593519451 }, order_id: 6413364814613, price: 5560.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 752 }
...
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718582343502504063 }, order_id: 6413384952694, price: 5655.000000000, size: 2, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175066 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 4916, ts_event: 1718582343502957759 }, order_id: 6413384952695, price: 5691.000000000, size: 2, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175067 }

MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718539204593519451 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 0 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718539204593519451 }, order_id: 6413256341927, price: 4650.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 519 }
...
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718582366242085265 }, order_id: 6413384879630, price: 5612.500000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175633 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 5002, ts_event: 1718582366260973309 }, order_id: 6413384879629, price: 5632.500000000, size: 1, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 175760 }

The snapshot for an empty order book has a single MBO record with cleaR action, marked with the flags F_SNAPSHOT and F_LAST, as in the snippet below.

MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 14160, ts_event: 1718117495809255541 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'R', side: 'N', ts_recv: 18446744073709551615, ts_in_delta: 0, sequence: 0 }

MBO snapshots are available from both the historical and live APIs.

Historical API

We offer MBO snapshot through the Historical API for venues that follow a weekly session structure (CME Globex MDP 3.0), and for venues whose daily trading sessions cross 00:00:00 UTC (ICE Europe Commodities iMpact). The order book snapshot is generated at 00:00:00 UTC each weekday (Monday-Friday).

Snapshot records are streamed through the Historical API when the requested interval includes midnight UTC for a given weekday. The snippet below shows an example of a timeseries request in Python whose result includes an MBO snapshot.

client.timeseries.get_range(
    dataset="GLBX.MDP3",
    symbols="ES.c.2",
    stype_in="continuous",
    schema="mbo",
    start="2024-06-16T23:58:50",
    end="2024-06-17T00:00:10",
)

MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582336782043745 }, order_id: 6413384952478, price: 5564.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582336782569129, ts_in_delta: 12306, sequence: 174718 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582336782045281 }, order_id: 6413384879627, price: 5560.500000000, size: 1, flags: LAST (130), channel_id: 0, action: 'M', side: 'B', ts_recv: 1718582336782613211, ts_in_delta: 12838, sequence: 174721 }
...
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582388400474957 }, order_id: 6413384953470, price: 5562.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582388400587693, ts_in_delta: 14247, sequence: 176276 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718539204593519451 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 0 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718539204593519451 }, order_id: 6413383913050, price: 5500.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 936 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718539204593519451 }, order_id: 6413373623913, price: 5475.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 936 }
...
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582388400474957 }, order_id: 6413384953470, price: 5562.750000000, size: 1, flags: LAST | SNAPSHOT | BAD_TS_RECV (168), channel_id: 0, action: 'A', side: 'B', ts_recv: 1718582400000000000, ts_in_delta: 0, sequence: 176276 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582400066482471 }, order_id: 6413384953619, price: 5564.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'A', ts_recv: 1718582400066580869, ts_in_delta: 14145, sequence: 176515 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582401163834877 }, order_id: 6413384953619, price: 5564.750000000, size: 1, flags: LAST (130), channel_id: 0, action: 'C', side: 'A', ts_recv: 1718582401163946845, ts_in_delta: 13326, sequence: 176774 }
...
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 183748, ts_event: 1718582409155703017 }, order_id: 6413384953440, price: 5413.250000000, size: 1, flags: LAST (130), channel_id: 0, action: 'M', side: 'B', ts_recv: 1718582409155796258, ts_in_delta: 12560, sequence: 178707 }

Live API

Users can request an MBO snapshot through the live API to obtain the recent order book state without replaying the whole trading session. The following sequence of messages is streamed from a live session after a snapshot subscription.

1. Symbol mapping messages
2. Snapshot records (clear book and outstanding orders at each price level)
3. Real-time records

Warning

The order book of a given instrument is not guaranteed to be in a complete and valid state after the last snapshot record because it might not correspond to the final event record (indicated with the F_LAST flag). If the last snapshot record does not have the F_LAST flag set, the order book will not be valid until the next MBO record with the F_LAST flag set.

The example below shows the live stream for a snapshot subscription for a single instrument, followed by a real-time record.

SymbolMappingMsg { hd: RecordHeader { length: 44, rtype: SymbolMapping, publisher_id: 0, instrument_id: 118, ts_event: 1721732190956747490 }, stype_in: 255, stype_in_symbol: "ES.c.0", stype_out: 255, stype_out_symbol: "ESU4", start_ts: 18446744073709551615, end_ts: 18446744073709551615 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152358684229 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 0 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514530616, price: 5547.500000000, size: 8, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1022 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514530330, price: 5547.250000000, size: 7, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1022 }
...
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413506441952, price: 5628.000000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'A', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1101 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413500305436, price: 5628.250000000, size: 6, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'A', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 1058 }
...
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152290217707 }, order_id: 6413522310536, price: 5613.250000000, size: 3, flags: SNAPSHOT | BAD_TS_RECV (42), channel_id: 0, action: 'A', side: 'A', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 13076493 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152290632121 }, order_id: 6413522309988, price: 5512.500000000, size: 6, flags: LAST | SNAPSHOT | BAD_TS_RECV (170), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152358684229, ts_in_delta: 0, sequence: 13076496 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152365904111 }, order_id: 6413522310535, price: 5613.250000000, size: 1, flags: LAST (130), channel_id: 0, action: 'C', side: 'A', ts_recv: 1721732152366001513, ts_in_delta: 14406, sequence: 13076497 }
MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721732152365916169 }, order_id: 6413522310537, price: 5613.000000000, size: 1, flags: LAST (130), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721732152366014389, ts_in_delta: 12865, sequence: 13076498 }
...

Timestamp fields on snapshot messages

The timestamp fields for snapshot messages are described below.

• ts_event: unchanged. The cleaR book MBO record from the Live API snapshot contains the snapshot generation timestamp instead
• ts_in_delta: always 0
• ts_recv: set to the snapshot generation timestamp (indicated with the F_BAD_TS_RECV flag)
• ts_out: unchanged

Public client support for MBO snapshot

Live and historical snapshot features are available on all our official client libraries (Python, C++, Rust), as well as through our Raw API.

An example of a live snapshot subscription using our client libraries can be found in this article.

Reference data enums

Exchanges, event types, other fields included in Databento's corporate actions data and reference data API. Over 60 event types, including splits, dividends, adjustment factors, and listings.

The following tables specify the descriptions for all reference data enums found in responses.

See also

Corporate actions dataset guide for further details.

ACTION

CNTRY

CUREN

DRTYPE

EVENT

EVENTSUBTYPE

EXCHANGE