Basics

Overview

Databento's live API offers both real-time subscriptions and intraday replay within the last 24 hours to various market data schemas and symbols. Multiple subscriptions may be combined into a single session, such as trades, subsampled data (second, minute, hour, daily aggregates), and definitions (expirations, settlement, etc.). For an easy transition from backtesting to live trading, the live API supports the same schemas, datasets, and symbology as the historical API and returns the same record structures. Our live clients use our binary DBN encoding for performant zero-copy market data decoding.

Authentication

Databento uses API keys to authenticate requests. You can view and manage your keys on the API keys page of your portal.

Each API key is a 32-character string. By default, our library uses the environment variable DATABENTO_API_KEY as your API key. However, if you pass an API key to the Live constructor through the key parameter, then this value will be used instead.

Our Live API uses a challenge-response authentication mechanism to ensure that your API key is never sent over the network.

Related: Securing your API keys.

Example usage

import databento as db

# Create a live client
client = db.Live(key="$YOUR_API_KEY")

# Authentication happens on the first subscribe
client.subscribe(
    dataset="GLBX.MDP3",
    schema="trades",
    stype_in="parent",
    symbols="ES.FUT",
)

Sessions

Each instance of the Live client manages a single Raw API session. Each session is associated with one dataset.

A session will begin streaming when Live.start is called. A session can be stopped gracefully with Live.stop or forcefully with Live.terminate.

A session can also be stopped by specifying a timeout with Live.block_for_close for synchronous applications and Live.wait_for_close for asynchronous applications.

Schemas and conventions

A schema is a data record format represented as a collection of different data fields. Our datasets support multiple schemas, such as order book, trades, bar aggregates, and so on. You can get a dictionary describing the fields of each schema from our List of market data schemas.

You can get a list of all supported schemas for any given dataset using the Historical client's list_schemas method. The same information can also be found on the dataset details pages on the user portal.

The following table provides details about the data types and conventions used for various fields that you will commonly encounter in the data.

Name	Field	Description
Dataset	`dataset`	A unique string name assigned to each dataset by Databento. Full list of datasets can be found from the metadata.
Publisher ID	`publisher_id`	A unique 16-bit unsigned integer assigned to each publisher by Databento. Full list of publisher IDs can be found from the metadata.
Instrument ID	`instrument_id`	A unique 32-bit unsigned integer assigned to each instrument by the venue. Information about instrument IDs for any given dataset can be found in the symbology.
Order ID	`order_id`	A unique 64-bit unsigned integer assigned to each order by the venue.
Timestamp (event)	`ts_event`	The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch.
Timestamp (receive)	`ts_recv`	The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch.
Timestamp delta (in)	`ts_in_delta`	The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. See timestamping guide.
Timestamp out	`ts_out`	The Databento gateway-sending timestamp expressed as the number of nanoseconds since the UNIX epoch. See timestamping guide.
Price	`price`	The price expressed as signed integer where every 1 unit corresponds to 1e-9, i.e. 1/1,000,000,000 or 0.000000001.
Book side	`side`	The side that initiates the event. Can be Ask for a sell order (or sell aggressor in a trade), Bid for a buy order (or buy aggressor in a trade), or None where no side is specified by the original source.
Size	`size`	The order quantity.
Flag	`flag`	A bit field indicating event end, message characteristics, and data quality.
Action	`action`	The event type or order book operation. Can be Add, Cancel, Modify, cleaR book, Trade, Fill, or None.
Sequence number	`sequence`	The original message sequence number from the venue.

Datasets

Databento provides time series datasets for a variety of markets, sourced from different publishers. Our available datasets can be browsed through the search feature on our site.

Each dataset is assigned a unique string identifier (dataset ID) in the form PUBLISHER.DATASET, such as GLBX.MDP3. For publishers that are also markets, we use standard four-character ISO 10383 Market Identifier Codes (MIC). Otherwise, Databento arbitrarily assigns a four-character identifier for the publisher.

These dataset IDs are also found on the Data catalog and Download request features of the Databento user portal.

When a publisher provides multiple data products with different levels of granularity, Databento subscribes to the most-granular product. We then provide this dataset with alternate schemas to make it easy to work with the level of detail most appropriate for your application.

More information about different types of venues and publishers is available in our FAQs.

Symbology

Databento's live API supports several ways to select an instrument in a dataset. An instrument is specified using a symbol and a symbology type, also referred to as an stype. The supported symbology types are:

Raw symbology (raw_symbol) original string symbols used by the publisher in the source data.

Instrument ID symbology (instrument_id) unique numeric ID assigned to each instrument by the publisher.

Parent symbology (parent) groups instruments related to the market for the same underlying.

Continuous contract symbology (continuous) proprietary symbology that specifies instruments based on certain systematic rules.

Info

In the live API, existing subscriptions to continuous contracts will not be remapped to different instruments. However, submitting a new identical subscription may result in a new mapping.

When subscribing to live data, an input symbology type can be specified. By default, our client libraries will use raw symbology for the input type. Not all symbology types are supported for every dataset.

If one or more symbols in a subscription fail to resolve, the gateway emits an ErrorMsg record with code=SYMBOL_RESOLUTION_FAILED for each unresolved symbol and continues streaming data for any symbols that did resolve. If a symbol later resolves (for example, a new listing from the publisher), the gateway publishes a SymbolMappingMsg and begins streaming data for that symbol.

For live data, symbology mappings are provided through SymbolMappingMsg records. These records are sent after the session has started and can be used to map the instrument_id from a data record's header to a text symbol.

For more about symbology at Databento, see our Standards and conventions.

Dates and times

Our Python client library has several functions with timestamp arguments. These arguments will have type pandas.Timestamp | datetime.date | str | int and support a variety of formats.

It's recommended to use pandas.Timestamp, which fully supports timezones and nanosecond-precision. If a datetime.date is used, the time is set to midnight UTC. If an int is provided, the value is interpreted as UNIX nanoseconds.

The client library also handles several string-based timestamp formats based on ISO 8601.

yyyy-mm-dd, e.g. "2022-02-28" (midnight UTC)

yyyy-mm-ddTHH:MM, e.g. "2022-02-28T23:50"

yyyy-mm-ddTHH:MM:SS, e.g. "2022-02-28T23:50:59"

yyyy-mm-ddTHH:MM:SS.NNNNNNNNN, e.g. "2022-02-28T23:50:59.123456789"

Timezone specification is also supported.

yyyy-mm-ddTHH:MMZ

yyyy-mm-ddTHH:MM±hh

yyyy-mm-ddTHH:MM±hhmm

yyyy-mm-ddTHH:MM±hh:mm

Bare dates

Some parameters require a bare date, without a time. These arguments have type datetime.date | str and must either be a datetime.date object, or a string in yyyy-mm-dd format, e.g. "2022-02-28".

Intraday replay

Our live API offers intraday replay within the last 24 hours. On weekends and other periods when data isn't being published, the gateway may report less than 24 hours of replay available. Users can connect to the live service and request data from a particular start time. Data will be filtered on ts_event for all schemas except CBBO and BBO, which will be filtered on ts_recv.

A different start time can be specified for each subscription. There can be multiple subscriptions for the same schema, with each subscription having a different start time. When the session starts, records will be immediately replayed for each schema. A replay completed SystemMsg will be sent for each replayed schema once it catches up to real-time data. Once a session has started, newly added subscriptions are not eligible for intraday replay.

As a special case for the GLBX.MDP3 dataset, we also provide replay of the entire weekly session in the MBO and definition schemas outside of the 24-hour window to aid with recovery, as these schemas are highly stateful.

Pass start=0 to easily request the full replay history available for each schema.

Our Python client library supports several convenient date formats, such as pd.Timestamp, Python datetime, Python date, ISO 8601 strings, or UNIX timestamps in nanoseconds. Refer to the Dates and times article for more information on how our client library handles timestamps.

Example usage

import databento as db

# Create a live client
client = db.Live(key="$YOUR_API_KEY")

# Subscribe with a specified start time for intraday replay
client.subscribe(
    dataset="GLBX.MDP3",
    schema="trades",
    symbols="ES.FUT",
    stype_in="parent",
    start="2023-04-17T09:00:00",
)

Snapshot

Users can request a snapshot for a live subscription to obtain the recent order book state without replaying the whole trading session. This is only supported for the MBO schema.

More details can be found in this article.

Example usage

import databento as db

# Create a live client
client = db.Live(key="$YOUR_API_KEY")

# Subscribe to live snapshot
client.subscribe(
    dataset="GLBX.MDP3",
    schema="mbo",
    symbols="ES.FUT",
    stype_in="parent",
    snapshot=True,
)

System messages

Our live API uses a system record (SystemMsg) to indicate non-error information to clients.

One use is heartbeating, to ensure the TCP connection remains open and to help clients detect a connection issue. A heartbeat will only be sent if no other data record was sent to the client during the heartbeat interval. The interval between heartbeat messages can be configured with the heartbeat_interval_s parameter to Live. The is_heartbeat() method will return True if the record is a heartbeat.

The gateway sends an end-of-interval message (END_OF_INTERVAL) once it has published every record for one interval of an OHLCV, BBO, or CBBO schema. Its ts_event is the exclusive end of that interval. For OHLCV schemas, that is the bar's ts_event plus the bar interval, so an ohlcv-1m end-of-interval message with a ts_event of 14:31:00 follows the bars stamped 14:30:00. For BBO and CBBO schemas, it equals the ts_recv of the records in the interval. The gateway sends one end-of-interval message per schema and interval.

Field	Type	Description
`msg`	str	The message from the gateway.
`code`	int	Describes the type of system message. See table below.

System code variants

Corresponds with the SystemCode enum.

Variant	`code`	Description
HEARTBEAT	`0`	A message sent in the absence of other records to indicate the connection remains open.
SUBSCRIPTION_ACK	`1`	An acknowledgement of a subscription request.
SLOW_READER_WARNING	`2`	The gateway has detected this session is falling behind real-time data.
REPLAY_COMPLETED	`3`	Indicates a replay subscription has caught up with real-time data. This message will be sent per schema.
END_OF_INTERVAL	`4`	Signals that all records for an interval-based schema have been published for the interval ending at `ts_event`.

Errors

Our live API uses an error record (ErrorMsg) to indicate failures to clients. Error records are processed like any other record, and as such will be passed on to any callbacks, streams, and iterators for the Live client.

Field	Type	Description
`err`	str	The error message.
`code`	int	Describes the type of error message. See table below.
`is_last`	bool	`True` if this is the last in a series of error records.

Error code variants

Corresponds with the ErrorCode enum.

Variant	`code`	Fatal	Description
AUTH_FAILED	`1`	✓	The authentication step failed.
API_KEY_DEACTIVATED	`2`	✓	The user account or API key were deactivated.
CONNECTION_LIMIT_EXCEEDED	`3`	✓	The user has exceeded their open connection limit.
SYMBOL_RESOLUTION_FAILED	`4`		A symbol in a subscription failed to resolve. Emitted per unresolved symbol.
INVALID_SUBSCRIPTION	`5`	✓	There was an issue with a subscription request (other than symbol resolution).
INTERNAL_ERROR	`6`	✓	An error occurred in the gateway.
SKIPPED_RECORDS_AFTER_SLOW_READING	`7`		A slow client was detected and records were skipped by the gateway to allow catching up.
REPLAY_DATA_AGED_OUT	`8`	✓	Replay data is no longer retained and the subscription's schema is incompatible with skipping records.

Fatal errors cause the gateway to close the session after sending the error record. Non-fatal errors are informational and the session continues.

Logging

Our Python client library is fully compatible with the built-in logging module.

When using the live client to build an application, it is recommended to enable logging in the databento module. This logging is disabled by default.

Connection limits

With our live API, there is a limit of 10 simultaneous connections (sessions) per (dataset) per team for Standard plans. Plus and Unlimited plans will be limited to 50 simultaneous connections per dataset per team. Creating additional API keys will not affect the maximum number of connections per team.

In addition, a single gateway will allow at most five incoming connections per second from the same IP address. If an IP address goes over this limit, incoming connections will be immediately closed by the gateway — existing connections will not be affected. If this happens, clients should wait one second before retrying.

Subscription rate limits

Subscription requests are throttled to prevent abuse and accidental performance impact on other users. Subscriptions above the limit of 10 per second will be artificially delayed until the session is back under the rate limit, but not rejected; the gateway will send a subscription acknowledgement when it has finished processing a subscription request.

Error detection

When maintaining a connected Live client, clients should monitor their connection for errors.

There are three main ways in which a session can enter an error state:

Hung connection: The client is not receiving any data from the gateway

Disconnect without error: The client is explicitly disconnected by the gateway, without receiving an error message

Disconnect with error: The client is explicitly disconnected by the gateway. Immediately prior to being disconnected, the client will receive an error record

Hung connection

To detect hung connections, clients are instructed to make use of system heartbeats. Clients can configure a heartbeat interval when creating the Live client by setting the heartbeat_interval_s parameter. If the heartbeat interval is not set by the client, it will default to 30 seconds.

Once a session is started, if no data is sent by the gateway for the entirety of a heartbeat interval, the gateway will send a system message to the client to indicate a heartbeat. If the gateway is regularly sending other messages to the client (for example, MboMsg), heartbeats will not be sent.

Once a session is started, if a client does not receive any messages from the gateway for the duration of one heartbeat interval plus ten seconds, the session can be considered hung. Clients are instructed to disconnect from the gateway and reconnect upon detecting hung connections.

Clients with unstable internet connections may need larger intervals than ten seconds to ensure the connection is truly hung, as opposed to merely delayed.

Disconnect without error

From the point of view of the Live client, a disconnect is detected when the underlying TCP session is closed. When using Live.block_for_close or Live.wait_for_close, an exception will be raised if the TCP session was closed without an error. Upon being disconnected, clients are instructed to wait one second and initiate a new connection. Waiting too short an interval to reconnect may trigger the gateway's rate limiter.

Disconnect with error

From the point of view of the Live client, a disconnect with error is detected when the underlying TCP session is closed after an ErrorMsg or a API error response is received. Any ErrorMsg records received by the client can be consumed by the application, such as when iterating or using a callback with Live.add_callback. When using Live.block_for_close or Live.wait_for_close, any ErrorMsg records received will be raised in an exception.

Clients disconnected with an error are instructed to not reconnect automatically. In the vast majority of cases, reconnecting and resubscribing with the same parameters will lead to the same errors being received again. The error sent to the client will indicate the issue to be fixed prior to resubscribing.

Versioning

Our historical and live APIs and their client libraries adopt MAJOR.MINOR.PATCH format for version numbers. These version numbers conform to semantic versioning. We are using major version 0 for initial development, where our API is not considered stable.

Once we release major version 1, our public API will be stable. This means that you will be able to upgrade minor or patch versions to pick up new functionality, without breaking your integration.

Starting with major versions after 1, we will provide support for previous versions for one year after the date of the subsequent major release. For example, if version 2.0.0 is released on January 1, 2024, then all versions 1.x.y of the API and client libraries will be deprecated. However, they will remain supported until January 1, 2025.

We may introduce backwards-compatible changes between minor versions in the form of:

New data encodings

Additional fields to existing data schemas

Additional batch download customizations

Our Release notes will contain information about both breaking and backwards-compatible changes in each release.

Our API and official client libraries are kept in sync with same-day releases for major versions. For instance, 1.x.y of the C++ client library will use the same functionality found in any 1.x.y version of the Python client.

Related: Release notes.

Compression

Databento provides options for compressing files from our API.

ZSTD

The ZSTD compression option uses the Zstandard format. This adds a slight performance cost to encoding and decoding.

Read more about working with Zstandard-compressed files.

NONE

The NONE compression option disables compression entirely, resulting in larger data transfer. By default, live data is uncompressed.

Recovering after a disconnection

When reconnecting to the gateway, clients should resubscribe to all desired symbols. In order to avoid missing data after a reconnection, there are three main approaches to recovery:

Natural refresh

Intraday replay

Snapshot (MBO only)

The best approach to recovery will depend on the client's use case and specific needs.

Natural refresh

To recover via natural refresh, clients can resubscribe to all desired symbols without the start or snapshot parameters. This means no data will be replayed, and the client will immediately receive the newest messages upon subscribing.

This recovery approach is the fastest (since there's no data replay), and is recommended for stateless schemas such as MBP-10, in cases where the client only requires the current state of each instrument.

Intraday replay

To recover via intraday replay, clients should store the last ts_event and the number of records received with that last timestamp, per schema and instrument. The ts_event and record count should be continuously updated when processing incoming records.

When reconnecting, clients should set the start parameter of the resubscription to the lowest stored ts_event across all instruments for that schema. The gateway will then send all records starting from that timestamp (including records with the exact same ts_event).

The resubscription may lead to duplicated data being sent to the client. Clients who require that each message is delivered exactly once are instructed to:

Discard all records with a lower ts_event than the stored one for the corresponding instrument

Discard the first N records with the same ts_event as the stored one for the corresponding instrument, where N is the number of records already seen with that ts_event prior to the disconnection. This is important in case there are multiple events with the same ts_event and the client is disconnected halfway through processing those events

This recovery approach is recommended when clients require the uninterrupted history of records for the desired schema (for example, when using the Trades schema to construct a ticker tape). However, this approach can take a longer time to synchronize with the live stream, depending on how long the client was disconnected.

For the CBBO and BBO schemas where filtering is based on ts_recv, clients should store the last ts_recv per instrument. When reconnecting, clients should set the start parameter of the resubscription to the lowest stored ts_recv across all instruments. The gateway will then send all records starting from that timestamp (including records with the same ts_recv). Since CBBO and BBO are stateless schemas, you should always refer to the most recent record per instrument.

Snapshot (MBO only)

When resubscribing to the MBO schema, clients can request a snapshot to receive the current state of the book after a disconnection. This eliminates the need to replay the missed messages and leads to faster synchronization with the live stream. This recovery approach is generally recommended over intraday replay when using the MBO schema.

Maintenance schedule

We restart our live gateways at the following times:

CME Globex. Saturday 02:15 CT

All ICE venues. Sunday 09:45 UTC

All other datasets. Sunday 10:30 UTC

All clients will be disconnected during this time. It is recommended to wait 15 minutes after the scheduled restart time before reconnecting.

Additionally, we may restart our gateways mid-week. While we generally post these mid-week restarts on our status page, we may perform these restarts without notice due to an urgent fix. You should configure your client to handle reconnecting automatically.

While some exchanges are closed on Saturday, our systems are still connected to the exchange feeds. The exchange may send test data, and our gateways will disseminate this data to all connected clients. If you are not interested in receiving this test data, we recommend you disconnect after the Friday close and reconnect on Sunday after the scheduled restart.

Info

Any test data sent through the Live API will not be seen in our historical data.

Slow reader behavior

Occasionally, a client session's data processing may lag behind real-time data. This can happen due to multiple reasons, such as a spike in messaging volume from the venue, network fluctuations between the client and the server, or an inability of the client application to process messages fast enough to keep up with the venue's message flow.

The live API allows clients to configure different behaviors when this happens via the slow_reader_behavior parameter in the Live constructor. This parameter can be set to the values skip or warn, and defaults to skip when not set. This parameter cannot be changed after the session is authenticated.

Skip

If the slow reader behavior is set to skip, whenever the gateway detects a client falling behind real-time data, an ErrorMsg with error code SKIPPED_RECORDS_AFTER_SLOW_READING will be sent to the client and the session will skip records until caught up with real-time data.

This behavior only applies to subscriptions for the following schemas: MBP-1, MBP-10, BBO-1s, BBO-1m, CMBP-1, CBBO-1s, and CBBO-1m. Subscriptions to other schemas will always use the warn behavior.

Warn

If the slow reader behavior is set to warn, whenever the gateway detects a client falling behind real-time data, a SystemMsg with system code SLOW_READER_WARNING will be sent to the client. The session will not skip any records and will attempt to replay every single record in the subscription.

In some cases, setting warn as the slow reader behavior can lead to increasing lag between the session and real-time data.

Data aged out of retention

A replay can fall behind far enough that the data it was reading is removed, i.e. it now exceeds the supported replay window. Sessions whose effective behavior is skip resume from the earliest retained records. Sessions using warn, either by request or because the schema is incompatible with skipping, receive a fatal ErrorMsg with error code REPLAY_DATA_AGED_OUT naming the earliest available start time.