Source: https://databento.com/docs/examples # Examples ", "[Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md)", "[Custom OHLCV bars from trades](https://databento.com/docs/examples/basics-historical/custom-ohlcv.md)" ], "LIVE DATA": [ "[Stream live data to a file](https://databento.com/docs/examples/basics-live/live-stream-to-file.md)", "[Handle multiple record types](https://databento.com/docs/examples/basics-live/live-dispatch.md)", "[Estimate latency](https://databento.com/docs/examples/basics-live/live-latency.md)" ] } }, { "title": "Advanced applications", "subtitle": "Develop systematic strategies and algorithmic trading applications.", "columns": { "HISTORICAL DATA": [ "[Limit order book construction](https://databento.com/docs/examples/order-book/limit-order-book.md)", "[Execution slippage and markouts](https://databento.com/docs/examples/algo-trading/execution-slippage.md)", "[Machine learning prediction models](https://databento.com/docs/examples/algo-trading/machine-learning.md)" ], "LIVE DATA": [ "[A high-frequency liquidity-taking strategy](https://databento.com/docs/examples/algo-trading/high-frequency.md)", "[Symbology mapping for live data](https://databento.com/docs/examples/symbology/live-symbol-mapping.md)", "[Calculate TICK and TRIN indicators](https://databento.com/docs/examples/basics-live/live-tick-trin.md)" ] } } ] } }) ", "[Create a candlestick plot](https://databento.com/docs/examples/basics-historical/candlestick-plot.md)", "[Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md)", "[Top pre-market movers](https://databento.com/docs/examples/equities/equities-premarket-movers.md)", "[Matching engine latencies](https://databento.com/docs/examples/algo-trading/latency.md)", "[Estimate implied volatility](https://databento.com/docs/examples/options/estimating-implied-volatility.md)" ] } ], [ { "title": "C++", "columns": [ "[Best bid, best offer, and midprice](https://databento.com/docs/examples/basics-historical/midprice.md)", "[Programmatic batch downloads](https://databento.com/docs/examples/basics-historical/programmatic-batch-download.md)" ] }, { "title": "Rust", "columns": [ "[Types of order book events](https://databento.com/docs/examples/order-book/order-actions.md)", "[End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md)" ] } ] ] } }) **Can't find an example for your use case?** Our team can help. **[Contact support](https://databento.com/support)** or **[join our Slack community](https://to.dbn.to/slack)** to chat with developers. ## Equities | Title | Subtitle | Image | Supported languages | | [Equities: Introduction](https://databento.com/docs/examples/equities/equities-introduction.md) | Get started with equities data by fetching top-of-book quotes and last sale. | ![Equities introduction image](https://databento.com/docs/equities-introduction.svg.md) | Python, Equities, Historical | | [Top pre-market movers](https://databento.com/docs/examples/equities/equities-premarket-movers.md) | Get a ranking of the biggest gainers in the Nasdaq pre-market session. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Rust, Equities, Historical | | [Find average spread for a symbol](https://databento.com/docs/examples/equities/spread-boxplots.md) | Find the average spread for a symbol across the different equity venues. | ![Spread boxplots](https://databento.com/docs/spread-boxplots-thumbnail.svg.md) | Python, Equities, Historical | | [Calculate synthetic NBBO from prop feeds](https://databento.com/docs/examples/equities/consolidated-bbo.md) | Consolidate MBP-1 from prop feeds to get an improved microprice vs. SIP NBBO. | ![Image placeholder](https://databento.com/docs/consolidated-bbo-thumbnail.svg.md) | Python, Historical | | [Get daily closing prices for equities](https://databento.com/docs/examples/equities/closing-prices.md) | Retrieve the official consolidated closing prices for US equities. | ![Image placeholder](https://databento.com/docs/closing-prices-thumbnail.svg.md) | Python, Historical | | [Analyze auction imbalance dynamics](https://databento.com/docs/examples/equities/auction-imbalance.md) | Compare the closing auction process for different US equity exchanges. | ![Image placeholder](https://databento.com/docs/auction-imbalance-thumbnail.svg.md) | Python, Historical | # Equities: Introduction > **Info** > > This article introduces additional concepts to get started with equities data on > Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll show how to: - Find an equities dataset - Fetch top-of-book quotes and last sale - Stream live data OHLCV data - Explain special conventions for equities vs. other asset classes on Databento ## Finding an equities dataset Our US Equity data provides full coverage of all RegNMS symbols, which includes but is not limited to: - Stocks (e.g. AAPL) - ETFs (e.g. SPY) - Warrants (e.g. GME+) While a symbol has a primary listing on one exchange, it may trade on all exchanges. We have coverage of the individual prop feeds from the various equity exchanges. Each prop feed will have its own dataset. To use equities data on Databento, first navigate to our [equities catalog page]($PLATFORM_HOSTNAME/catalog/us-equities). ![Databento Equities landing](https://databento.com/docs/equities-landing.png.md) Next, click on the dataset of your choice. ![Dataset Name](https://databento.com/docs/equities-dataset-id.png.md) For this example, we'll use the [Nasdaq TotalView-ITCH]($PLATFORM_HOSTNAME/portal/catalog/us-equities#XNAS.ITCH) dataset, whose dataset name is `XNAS.ITCH`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## Getting MBP-1 data The easiest way to fetch top-of-book quotes and last sale is by requesting the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md), which gives all trades and all book updates that affect the top-of-book; some data providers refer to this as "L1" data. > **Info** > > Note that MBP-1 should not be mistaken with the TBBO schema. > MBP-1 is a superset of the TBBO schema, which is essentially MBP-1 > sampled in trade space. See our [FAQ page](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md) for more information on the differences between these schemas. ```python import databento as db import matplotlib.pyplot as plt client = db.Historical("$YOUR_API_KEY") dataset = "XNAS.ITCH" symbol = "AAPL" start = "2024-10-28T12:30:00" end = "2024-10-28T12:45:00" data = client.timeseries.get_range( dataset=dataset, schema="mbp-1", symbols=symbol, start=start, end=end, ) df = data.to_df() df = df.rename(columns={"bid_px_00": "bid", "ask_px_00": "ask"}) ax = df[["bid", "ask"]].plot(drawstyle="steps-post") df[df["action"] == "T"]["price"].plot( ax=ax, style=".", markersize=10, label="trade", color="C2", zorder=2, ylabel="Price", xlabel="Time", title=symbol, ) plt.legend() plt.show() ``` ![AAPL price plot](https://databento.com/docs/equities-price-plot-0.svg.md) ## Streaming live OHLCV data The [Databento US Equities Mini](https://databento.com/docs/datasets/equs-mini.md) (`EQUS.MINI`) dataset is our most cost-effective solution for real-time, top-of-book US equities data. It combines multiple trading venues into one composite dataset that provides accurate best bid and offer (BBO) quotes, liquidity, and trade volume data--all without exchange license fees. In this example, we will use the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md) and print out records every second. A full list of available schemas can be found on the [dataset documentation page](https://databento.com/docs/datasets/equs-mini.md). > **Info** > > This example requires a live data subscription to Databento US Equities. For more information, please visit our [equities catalog page]($PLATFORM_HOSTNAME/portal/catalog/us-equities). ```python import databento as db # Enable some basic logging db.enable_logging("INFO") # Create a live client and connect live_client = db.Live("$YOUR_API_KEY") # Subscribe to the OHLCV-1s schema for AMZN live_client.subscribe( dataset="EQUS.MINI", schema="ohlcv-1s", symbols="AMZN", ) # We add a print callback to view each record live_client.add_callback(print) # Start the live client to begin data streaming live_client.start() # Run the stream for 15 seconds before closing live_client.block_for_close(timeout=15) ``` > **Info** > > If you do not see any output, it could be because the markets are closed. See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) for using [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=853, ts_event=1769047202984118060, pretty_ts_event='2026-01-22T02:00:02.984118060Z', stype_in=None, stype_in_symbol='AMZN', stype_out=None, stype_out_symbol='AMZN', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768913037000000000, pretty_ts_event='2026-01-20T12:43:57.000000000Z', open=233480000000, pretty_open=233.48, high=233480000000, pretty_high=233.48, low=233480000000, pretty_low=233.48, close=233480000000, pretty_close=233.48, volume=15) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914152000000000, pretty_ts_event='2026-01-20T13:02:32.000000000Z', open=233370000000, pretty_open=233.37, high=233390000000, pretty_high=233.39, low=233370000000, pretty_low=233.37, close=233390000000, pretty_close=233.39, volume=103) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914160000000000, pretty_ts_event='2026-01-20T13:02:40.000000000Z', open=233490000000, pretty_open=233.49, high=233490000000, pretty_high=233.49, low=233490000000, pretty_low=233.49, close=233490000000, pretty_close=233.49, volume=65) ... OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914714000000000, pretty_ts_event='2026-01-20T13:11:54.000000000Z', open=233500000000, pretty_open=233.5, high=233500000000, pretty_high=233.5, low=233490000000, pretty_low=233.49, close=233490000000, pretty_close=233.49, volume=100) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768914724000000000, pretty_ts_event='2026-01-20T13:12:04.000000000Z', open=233570000000, pretty_open=233.57, high=233570000000, pretty_high=233.57, low=233570000000, pretty_low=233.57, close=233570000000, pretty_close=233.57, volume=50) OhlcvMsg(rtype=, publisher_id=EQUS.MINI.EQUS (95), instrument_id=853, ts_event=1768915049000000000, pretty_ts_event='2026-01-20T13:17:29.000000000Z', open=233320000000, pretty_open=233.32, high=233320000000, pretty_high=233.32, low=233320000000, pretty_low=233.32, close=233320000000, pretty_close=233.32, volume=25) ``` ## Special conventions for equities on Databento - **Odd lots.** All order quantities are expressed in units of 1 share. Databento's equities data generally includes odd lots as we source our data directly from prop feeds. If you've switched to Databento from a data provider that only distributes CTA/UTP SIP data, you may have never encountered order quantities for less than 100 shares as the SIPs only provide round lots. - **Explicit trade aggressor side.** Equities prop feeds usually provide the aggressor side of each trade explicitly. Hence, the `side` field in our data is exact; it is not inferred using a trade classification algorithm. If the aggressor side is not known, we mark it with a side of `N`. If you've switched to Databento from a data provider that only distributes CTA/UTP SIP data, you may not have dealt with explicit trade aggressor side before but should try to make use of this field as it provides valuable premia over the SIPs. - **Ephemeral and Databento-assigned permanent instrument IDs.** Several equities venues do not have a system of numeric instrument IDs that correspond to each ticker symbol; these venues rely solely on the ticker symbols to identify each instrument. On certain venues like Nasdaq, the venue does assign instrument IDs to each instrument that can be used alongside the ticker symbols, but such venue-assigned instrument IDs are ephemeral and rotate daily. This is unlike venues like CME Globex, where the venue-assigned instrument IDs are typically permanent until the instrument expires. Where the venue assigns an instrument ID, we pass this directly through--so the instrument IDs of a venue like Nasdaq may be difficult to work with. Where the venue doesn't assign an instrument ID, we'll generate our own synthetic instrument ID for consistency across all venues. - **Atomic executions and passive-side fills.** On most equities venues, the fill event and order deletion is published at the same time as a single, atomic execution event. In such cases, Databento represents each execution event with action `F`, implying the passive-side fill, and provides a synthetic trade event with action `T` to represent the aggressor side of the trade. Note that this is merely for convenience, so you can reuse the same book management logic for all delta one products like equities and futures--in such cases that the aggressor size is not explicit since we only know the sizes of the passive fills corresponding to the trade, and hence the actual aggressor size needs to be inferred. - **Trade conditions.** We do not provide trade condition codes as those are not provided on the prop feeds. - **Instrument definitions.** Equities instrument definitions tend to be much sparser than those of derivatives like options and futures, so you should expect to see many null values when you fetch instrument definitions. ## Symbology In US equities, the same stock may be traded on different venues, but have slightly different ticker symbols on each venue. Examples include `BRK.B` and `AAC+` in the Nasdaq convention vs. `BRK B` and `AAC WS` in the CMS convention. In addition to the `raw_symbol` symbology, which uses the publisher's convention, we also offer the `cms` and `nasdaq` symbology types to allow explicitly using one convention. For Databento consolidated equities datasets and Nasdaq TotalView-ITCH, `raw_symbol` and `nasdaq` symbology types are equivalent. See [CMS vs. Nasdaq symbology](https://databento.com/docs/conventions/symbology.md#raw-symbol) for the per-dataset convention map. ## Numeric publisher IDs Consolidated equities feeds include data from several publishers. The `publisher_id` field included with each record indicates the source venue of the record. These IDs can be converted to publisher names through the `Publisher` enum. > **See also** > > [End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md) and > [Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md). # Top pre-market movers ## Overview This example demonstrates how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve OHLCV-1m bar aggregate data to analyze the largest movers in the Nasdaq pre-market. Equities which are trading in the early pre-market may be reacting to recent events or news announcements, relevant to a single stock, an industry, or an entire sector. This notable pre-market activity usually manifests in large price changes and volume, which significantly exceed the normal range, and may be indicative of subsequent moves after the session open. For our purposes we'll define the pre-market as 4:00 AM to 9:30 AM in the 'US/Eastern' time zone, which accounts for changes for daylight savings. You can find more information on Nasdaq trading hours [here](https://www.nasdaq.com/stock-market-trading-hours-for-nasdaq). ## Nasdaq dataset For this example, we'll use the [Nasdaq TotalView-ITCH]($PLATFORM_HOSTNAME/datasets/XNAS.ITCH) dataset, whose dataset ID is `XNAS.ITCH`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## OHLCV-1m schema The data we need will come from the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low, and closing prices, as well as the aggregated volume of trades occurring within each time interval. Since we are interested in a specific market session, we'll use an interval of one minute, which is specified by the suffix `-1m`. ## Example ```python import databento as db import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, request OHLCV-1m bars for the time range we're interested in data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="ohlcv-1m", symbols="ALL_SYMBOLS", start=pd.Timestamp("2023-06-06T04:00", tz="US/Eastern"), end=pd.Timestamp("2023-06-06T09:30", tz="US/Eastern"), ) # We need to obtain the symbology mappings for 'ALL_SYMBOLS' symbology = data.request_symbology(client) # This allows us to populate the 'symbol' column when we convert to a DataFrame data.insert_symbology_json(symbology) # Convert to dataframe with timestamps in the US/Eastern timezone df = data.to_df(tz="US/Eastern") # Combine the first open, last close and volume sum into a DataFrame df_moves = df.groupby("symbol").agg({"open": "first", "close": "last", "volume": "sum"}) # Calculate the change from start to end of pre-market df_moves["change"] = (df_moves["close"] - df_moves["open"]) / df_moves["open"] df_moves = df_moves.sort_values(by="change", ascending=False) # Display the top 10 largest movers print(df_moves.head(10)) ``` ## Result ```plaintext open close volume change symbol CRGOW 0.1564 0.3200 1977 1.046036 ZVZZT 13.0200 25.0000 369 0.920123 MLECW 0.0974 0.1798 3394 0.845996 HOTH 2.0300 3.5400 1600550 0.743842 BOAC+ 0.0458 0.0782 700 0.707424 RCRT 0.1918 0.2953 575273 0.539625 ATMCR 0.1489 0.2237 400 0.502351 SBFMW 0.5938 0.8915 305 0.501347 RCRTW 0.0229 0.0302 300 0.318777 TCBPW 0.0436 0.0573 300 0.314220 ``` > **See also** > > [End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md) and > [Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md). # Top pre-market movers ## Overview This example demonstrates how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve OHLCV-1m bar aggregate data to analyze the largest movers in the Nasdaq pre-market. Equities which are trading in the early pre-market may be reacting to recent events or news announcements, relevant to a single stock, an industry, or an entire sector. This notable pre-market activity usually manifests in large price changes and volume, which significantly exceed the normal range, and may be indicative of subsequent moves after the session open. For our purposes we'll define the pre-market as 4:00 AM to 9:30 AM in the US Eastern time zone, which accounts for daylight savings. You can find more information on Nasdaq trading hours [here](https://www.nasdaq.com/stock-market-trading-hours-for-nasdaq). ## Nasdaq dataset For this example, we'll use the [Nasdaq TotalView-ITCH]($PLATFORM_HOSTNAME/datasets/XNAS.ITCH) dataset, whose dataset ID is `XNAS.ITCH`. We'll use the `Dataset` enum to pass this as the `dataset` parameter. ## OHLCV-1m schema The data we need will come from the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low, and closing prices, as well as the aggregated volume of trades occurring within each time interval. Since we are interested in a specific market session, we'll use an interval of one minute, which is specified by the suffix `1M`. ## Example ```rust use std::collections::HashMap; use databento::{ HistoricalClient, Symbols, dbn::{Dataset, OhlcvMsg, Schema, pretty::Px}, historical::{symbology::ResolveParams, timeseries::GetRangeParams}, }; use time::macros::datetime; use time_tz::{PrimitiveDateTimeExt, timezones}; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let tz = timezones::db::us::EASTERN; // First, request the OHLCV-1m bars for the time range we're interested in let params = GetRangeParams::builder() .dataset(Dataset::XnasItch) .date_time_range( datetime!(2023 - 06 - 06 04:00).assume_timezone(tz).unwrap() ..datetime!(2023 - 06 - 06 09:30).assume_timezone(tz).unwrap(), ) .symbols(Symbols::All) .schema(Schema::Ohlcv1M) .build(); let mut decoder = client.timeseries().get_range(¶ms).await?; // We need to obtain symbology mappings for all symbols requests separately let symbol_map = client .symbology() .resolve(&ResolveParams::from(params)) .await? .symbol_map()?; // Aggregate the results into a single bar for the pre-open period let mut aggregated_bars = HashMap::new(); while let Some(bar) = decoder.decode_record::().await? { aggregated_bars .entry(bar.hd.instrument_id) .and_modify(|agg_bar: &mut OhlcvMsg| { agg_bar.volume += bar.volume; agg_bar.close = bar.close; }) .or_insert(bar.clone()); } // Calculate the percent change let mut bars_with_change = aggregated_bars .into_values() .map(|bar| { let change = (bar.close_f64() - bar.open_f64()) / bar.open_f64(); let symbol = symbol_map[&bar].clone(); (symbol, bar, change) }) .collect::>(); // Find and print the top 10 largest movers bars_with_change.sort_by(|(_, _, a), (_, _, b)| a.partial_cmp(b).unwrap().reverse()); println!( "{:<6}{:>9}{:>9}{:>9}{:>10}", "symbol", "open", "close", "volume", "change" ); for (symbol, bar, change) in bars_with_change.iter().take(10) { println!( "{symbol:<6}{:>9.4}{:>9.4}{:>9}{change:>10.6}", Px(bar.open), Px(bar.close), bar.volume ); } Ok(()) } ``` ## Result ```plaintext symbol open close volume change CRGOW 0.1564 0.3200 1977 1.046036 ZVZZT 13.0200 25.0000 369 0.920123 MLECW 0.0974 0.1798 3394 0.845996 HOTH 2.0300 3.5400 1600550 0.743842 BOAC+ 0.0458 0.0782 700 0.707424 RCRT 0.1918 0.2953 575273 0.539625 ATMCR 0.1489 0.2237 400 0.502351 SBFMW 0.5938 0.8915 305 0.501347 RCRTW 0.0229 0.0302 300 0.318777 TCBPW 0.0436 0.0573 300 0.314220 ``` > **See also** > > [End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md) and > [Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md). # Find average spread for a symbol There are 16 US equity exchanges that provide trading for RegNMS symbols. Knowing the liquidity characteristics for each venue is necessary in understanding the microstructure of the equity markets. Cboe publishes the [US Equities Market Volume Summary report](https://www.cboe.com/us/equities/market_statistics/) that indicates the relative market volume share for each venue. Databento provides coverage for all equity exchanges except LTSE, which provides <0.01% of total market volume. ## Overview In this example, we will use the [historical client](https://databento.com/docs/api-historical/client.md) to get the average spread for SPY across the different equity venues before and after an increase in volatility. We'll use the [BBO-1s schema](https://databento.com/docs/schemas/bbo.md) which provides the bid and ask price subsampled in 1-second intervals to calculate the spread in basis-points (BPS). We'll compare these pre-volatility and post-volatility figures for the different venues. ## Example ```python import datetime as dt import databento as db import matplotlib.pyplot as plt import pandas as pd import seaborn as sns # Set parameters symbol = "SPY" prevol_date = dt.date(2025, 4, 1) # Tuesday before volatility spike postvol_date = dt.date(2025, 4, 8) # Tuesday after volatility spike # Regular Trading Hours (RTH) for US equities rth_start = dt.time(9, 30) rth_end = dt.time(16, 0) equity_datasets = [ "XNAS.ITCH", # Nasdaq "XBOS.ITCH", # Nasdaq Texas "XPSX.ITCH", # Nasdaq PSX "XNYS.PILLAR", # NYSE "ARCX.PILLAR", # NYSE Arca "XASE.PILLAR", # NYSE American "XCHI.PILLAR", # NYSE Texas "XCIS.TRADESBBO", # NYSE National "MEMX.MEMOIR", # Members Exchange "EPRL.DOM", # MIAX Pearl "IEXG.TOPS", # IEX "BATS.PITCH", # Cboe BZX "BATY.PITCH", # Cboe BYX "EDGA.PITCH", # Cboe EDGA "EDGX.PITCH", # Cboe EDGX # Databento does not provide coverage for LTSE (MIC: LTSE) ] client = db.Historical(key="$YOUR_API_KEY") venue_map = {x["dataset"]: x["venue"] for x in client.metadata.list_publishers()} def calculate_spread( dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """ Calculate spread size in basis points (bps) """ df = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="bbo-1s", # BBO-1s is required for this example start=date, ).to_df(tz="America/New_York") df = df.resample("1s").ffill().loc[rth_start:rth_end] # Calculate spread in basis points df["midpoint"] = (df["ask_px_00"] + df["bid_px_00"]) / 2 df["spread_abs"] = (df["ask_px_00"] - df["bid_px_00"]).clip(lower=0) df["spread_bps"] = df["spread_abs"] / df["midpoint"] * 1e4 df["Venue"] = venue_map[dataset] df["Date"] = df.index.date return df[["Venue", "Date", "spread_bps"]] df_list: list[pd.DataFrame] = [] for date in (prevol_date, postvol_date): for dataset in equity_datasets: df_list.append(calculate_spread(dataset, symbol, date)) df = pd.concat(df_list, ignore_index=True) plt.figure(figsize=(16, 8)) ax = sns.boxplot( data=df, x="Venue", y="spread_bps", hue="Date", order=df[df["Date"] == postvol_date] .groupby("Venue")["spread_bps"] .median() .sort_values() .index, gap=0.1, width=0.8, showfliers=False, ) # Set colors for each venue n_venues = len(equity_datasets) venue_colors = sns.color_palette("Set1", n_colors=n_venues) for i, patch in enumerate(ax.patches): patch.set_facecolor(venue_colors[i % n_venues]) # Color the post-volatility boxplots a lighter shade if i >= n_venues: patch.set_facecolor((*patch.get_facecolor()[:3], 0.4)) # Adjust legend prevol_patch, postvol_patch = ax.get_legend().get_patches() prevol_patch.set_facecolor(venue_colors[0]) postvol_patch.set_facecolor((venue_colors[0], 0.4)) plt.legend(handles=[prevol_patch, postvol_patch], labels=[f"{prevol_date}", f"{postvol_date}"]) plt.xticks(rotation=45) plt.ylabel("Spread size (bps)") plt.title(f"{symbol} spreads before and after April 2025 tariff crash") plt.tight_layout() plt.show() ``` The venues with the highest market share, such as NYSE Arca (`ARCX`) and Nasdaq (`XNAS`), have spreads that remain relatively tight even after the increase in volatility. Venues with a low market share see the largest increase in spreads post-volatility. ![Spread boxplot](https://databento.com/docs/spread-boxplots-0.svg.md) # Calculate synthetic NBBO from prop feeds ## Overview In this example, we will calculate a synthetic NBBO by taking the best bid and offer seen across the different exchanges. Unlike the official SIP NBBO which ignores odd lots and is calculated by CTA and UTP in their datacenters, this synthetic NBBO is derived from the direct prop feeds. Databento provides top-of-book coverage for all equity exchanges (except LTSE, which only provides <0.01% of total market volume). Databento captures all equity data in the NY4 datacenter with PTP timestamps. See our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md) for more information. ## Design We'll use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) and download data for the different equity exchanges. Next, we'll process the data sequentially based on [ts_recv](https://databento.com/docs/conventions/common-fields-enums-types.md#ts-recv), which is the timestamp when Databento received the data from the exchange. We'll calculate the synthetic NBBO by taking the best bid and offer across these exchanges. After that, we'll plot this over a 100-millisecond window to take a closer look at how the synthetic NBBO reacts when price moves. ## Example ```python import os from collections import defaultdict from dataclasses import dataclass, field from heapq import merge import databento as db import matplotlib.dates as mdates import matplotlib.pyplot as plt import pandas as pd @dataclass(slots=True) class PriceLevel: price: float size: int = 0 count: int = 0 update_time: int = 0 def __str__(self) -> str: return f"{self.size:4} @ {self.price:6.2f} | {self.count:2} order(s)" @staticmethod def _bid_sort(r) -> tuple[float, int, int]: return r.price, r.size, -r.update_time @staticmethod def _ask_sort(r) -> tuple[float, int, int]: return -r.price, r.size, -r.update_time @dataclass(slots=True) class MbpBook: offer: PriceLevel | None = field(default=None) bid: PriceLevel | None = field(default=None) def bbo(self) -> tuple[PriceLevel | None, PriceLevel | None]: return self.bid, self.offer def apply(self, mbp1: db.MBP1Msg) -> None: level = mbp1.levels[0] ts_recv = mbp1.ts_recv if level.bid_px == db.UNDEF_PRICE: self.bid = None else: self.bid = PriceLevel(level.pretty_bid_px, level.bid_sz, level.bid_ct, ts_recv) if level.ask_px == db.UNDEF_PRICE: self.offer = None else: self.offer = PriceLevel(level.pretty_ask_px, level.ask_sz, level.ask_ct, ts_recv) @dataclass(slots=True) class MbpMarket: books: defaultdict[int, defaultdict[int, MbpBook]] = field( default_factory=lambda: defaultdict(lambda: defaultdict(MbpBook)), ) def get_book(self, instrument_id: int, publisher_id: int) -> MbpBook: return self.books[instrument_id][publisher_id] def bbo( self, instrument_id: int, publisher_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: return self.books[instrument_id][publisher_id].bbo() def aggregated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: """Calculate the aggregated BBO across all venues""" agg_bbo: list[PriceLevel | None] = [None] all_bbos = list(zip(*(book.bbo() for book in self.books[instrument_id].values()))) for idx, reducer in ((0, max), (1, min)): all_best = [b for b in all_bbos[idx] if b] if all_best: best_price = reducer(b.price for b in all_best) best = [b for b in all_best if b.price == best_price] agg_bbo[idx] = PriceLevel( price=best_price, size=sum(b.size for b in best), count=sum(b.count for b in best), ) return tuple(agg_bbo) def consolidated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: all_bids, all_offers = zip(*(book.bbo() for book in self.books[instrument_id].values())) best_bid = max((b for b in all_bids if b), key=PriceLevel._bid_sort, default=None) best_offer = max((o for o in all_offers if o), key=PriceLevel._ask_sort, default=None) return best_bid, best_offer def apply(self, msg: db.MBP1Msg) -> None: book = self.books[msg.instrument_id][msg.publisher_id] book.apply(msg) if __name__ == "__main__": equity_datasets = [ "XNAS.ITCH", # Nasdaq "XBOS.ITCH", # Nasdaq Texas "XPSX.ITCH", # Nasdaq PSX "XNYS.PILLAR", # NYSE "ARCX.PILLAR", # NYSE Arca "XASE.PILLAR", # NYSE American "XCHI.PILLAR", # NYSE Texas "XCIS.TRADESBBO", # NYSE National "MEMX.MEMOIR", # Members Exchange "EPRL.DOM", # MIAX Pearl "IEXG.TOPS", # IEX "BATS.PITCH", # Cboe BZX "BATY.PITCH", # Cboe BYX "EDGA.PITCH", # Cboe EDGA "EDGX.PITCH", # Cboe EDGX ] symbol = "NVDA" start = pd.Timestamp(2025, 6, 17, 9, 30, tz="US/Eastern") end = pd.Timestamp(2025, 6, 17, 10, 0, tz="US/Eastern") schema = "mbp-1" client = db.Historical(key="$YOUR_API_KEY") # Get data for all datasets dataset_data_dict: dict[str, db.DBNStore] = {} for dataset in equity_datasets: dataset_name = dataset.replace(".", "-").lower() data_path = ( f"{dataset_name}-{symbol}-{start.date().isoformat().replace('-', '')}.{schema}.dbn.zst" ) if os.path.exists(data_path): data = db.DBNStore.from_file(data_path) else: data = client.timeseries.get_range( dataset=dataset, start=start, end=end, symbols=symbol, schema=schema, path=data_path, ) dataset_data_dict[dataset] = data # Merge all datasets into one stream sorted by ts_recv data = merge(*dataset_data_dict.values(), key=lambda x: x.ts_recv) # Iterate over the records and calculate the consolidated BBO cbbo_list: list[tuple[pd.Timestamp, float, float]] = [] market = MbpMarket() for record in data: market.apply(record) best_bid, best_offer = market.consolidated_bbo(record.instrument_id) cbbo_list.append( ( record.pretty_ts_recv, best_bid.price if best_bid is not None else float("Nan"), best_offer.price if best_offer is not None else float("Nan"), ), ) # Create DataFrame df = pd.DataFrame(cbbo_list, columns=["Timestamp", "Bid", "Offer"]) df = df.set_index("Timestamp") df["is_crossed"] = df["Bid"] >= df["Offer"] # Now we'll plot a small slice of time when the book is crossed start_time = pd.Timestamp(2025, 6, 17, 9, 56, 31, 650000, tz="US/Eastern") end_time = pd.Timestamp(2025, 6, 17, 9, 56, 31, 750000, tz="US/Eastern") df = df.loc[start_time:end_time] fig, ax = plt.subplots(figsize=(11, 6)) # Shade periods where book is not crossed green plt.fill_between( df.index, df["Bid"], df["Offer"], where=~df["is_crossed"], alpha=0.2, linewidth=0, color="green", step="post", label="Normal market", ) # Shade periods where book is crossed red plt.fill_between( df.index, df["Offer"], df["Bid"], where=df["is_crossed"], alpha=0.2, linewidth=0, color="red", step="post", label="Crossed market", ) # Plot BBO lines for col, color in [("Offer", "C1"), ("Bid", "C0")]: plt.hlines( y=df[col][:-1], xmin=df.index[:-1], xmax=df.index[1:], colors=color, label=col, ) plt.gca().xaxis.set_major_formatter(mdates.DateFormatter("%H:%M:%S.%f", tz="US/Eastern")) plt.ylabel("Price (USD)") plt.xlabel("Timestamp (ET)") plt.title(f"{symbol} synthetic NBBO") plt.legend() plt.tight_layout() plt.show() ``` ## Results Notice that the synthetic NBBO may occasionally appear crossed, unlike the regulatory NBBO published by the SIPs. This is expected behavior due to two factors: first, proprietary feeds include odd lot quotations, which are excluded from SIP NBBO calculations; second, the feeds originate from different data centers, resulting in receive-time deltas. These characteristics can temporarily produce crossed markets, but they also enable the construction of a more predictive microprice. ![Consolidated BBO](https://databento.com/docs/consolidated-bbo-0.svg.md) # Get daily closing prices for equities Official consolidated end-of-day summary data for all US equities is published by the Nasdaq NLS+ feed. This data is normalized into our [Databento US Equities Summary dataset](https://databento.com/docs/datasets/equs-summary.md). ## Overview We'll use the [historical client](https://databento.com/docs/api-historical/client.md) to request daily data for multiple equity symbols. For this example, we'll look at a few popular companies like AMZN, NFLX, and TSLA. We'll calculate the cumulative percent change for these symbols over time, then plot the data to compare their relative performance. ## Example ```python import databento as db import matplotlib.pyplot as plt # Set parameters dataset = "EQUS.SUMMARY" symbols = ["AMZN", "NFLX", "TSLA"] start = "2024-10-01" end = "2025-10-01" client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1d data for the selected symbols df = client.timeseries.get_range( dataset=dataset, symbols=symbols, schema="ohlcv-1d", start=start, end=end, ).to_df() # Calculate cumulative percent change over the request range df["cum_pct_change"] = df.groupby("symbol")["close"].transform( lambda x: (x - x.iloc[0]) / x.iloc[0] * 100, ) print(df) # Plot the relative performance of these symbols df.groupby("symbol")["cum_pct_change"].plot( figsize=(12, 6), legend=True, xlabel="Date", ylabel="Cumulative % Change", title="Relative Performance\nOctober 2024 - October 2025", ) plt.axhline(y=0, color="grey", linestyle="-", zorder=1) plt.grid(True) plt.tight_layout() plt.show() ``` ## Results ```plaintext rtype publisher_id instrument_id open high low close volume symbol cum_pct_change ts_event 2024-10-01 00:00:00+00:00 35 90 853 184.90 186.1900 183.4519 185.13 36044906 AMZN 0.000000 2024-10-01 00:00:00+00:00 35 90 11275 713.64 717.7600 698.5900 706.13 2813482 NFLX 0.000000 2024-10-01 00:00:00+00:00 35 90 16244 262.67 263.9800 248.5300 258.02 87397613 TSLA 0.000000 2024-10-02 00:00:00+00:00 35 90 16244 247.55 251.1585 241.5000 249.02 93983930 TSLA -3.488102 2024-10-02 00:00:00+00:00 35 90 853 184.44 186.6000 184.0400 184.76 23704056 AMZN -0.199860 ... ... ... ... ... ... ... ... ... ... ... 2025-09-29 00:00:00+00:00 35 90 16244 444.35 450.9800 439.5000 443.21 79491510 TSLA 71.773506 2025-09-29 00:00:00+00:00 35 90 853 220.08 222.6000 219.3000 222.17 44259177 AMZN 20.007562 2025-09-30 00:00:00+00:00 35 90 853 222.03 222.2400 217.8900 219.57 48396369 AMZN 18.603144 2025-09-30 00:00:00+00:00 35 90 11275 1206.41 1208.5000 1178.0000 1198.92 3830304 NFLX 69.787433 2025-09-30 00:00:00+00:00 35 90 16244 441.52 445.0000 433.1200 444.72 74357960 TSLA 72.358732 [750 rows x 10 columns] ``` > **See also** > > The prices seen in the EQUS.SUMMARY dataset are unadjusted for any splits or dividends. > This data can be enriched with our [corporate actions and adjustment factors datasets]($PLATFORM_HOSTNAME/docs/portal/live-data). ![Daily prices](https://databento.com/docs/closing-prices-0.svg.md) # Analyze auction imbalance dynamics The closing auction for US Equities is one of the most important events of the trading day. With nearly 10% of average daily volume transacting on the close, the dynamics around price discovery are critical, as these closing prices are used in various use cases, such as benchmark pricing for index funds. While different exchanges share many common behaviors during the closing auction, there are some microstructure differences, such as accepted order types, that can affect the price discovery mechanism. ## Overview We'll use the [historical client](https://databento.com/docs/api-historical/client.md) to request data for the closing auction. The [imbalance schema](https://databento.com/docs/schemas/imbalance.md) provides auction imbalance data such as reference price, paired quantity, and auction status. We'll also use the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md) to look at the price and volume transacting on the limit order book. We'll compare the closing auction process for SPY on NYSE Arca and QQQ on Nasdaq. Due to differences in the data disseminated by the exchange, we'll use different fields for Nasdaq and NYSE. For both venues, the `ref_price` field represents the price at which imbalance shares are being calculated. This price is generally within the current BBO on the limit order book. For Nasdaq, we'll look at the `cont_book_clr_price` field, which represents the price at which auction and continuous orders would transact. For NYSE Arca, we'll look at the `ind_match_price` field instead. This field represents the price at which the highest number of shares would trade, subject to auction collars. ## Example ```python import datetime as dt from zoneinfo import ZoneInfo import databento as db import pandas as pd from matplotlib import dates as mdates from matplotlib import pyplot as plt def get_imbalance_df( client: db.Historical, dataset: str, symbol: str, start: dt.datetime, end: dt.datetime, ) -> pd.DataFrame: """Get imbalance schema data""" imbalance_df = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="imbalance", start=start, end=end, ).to_df(tz=start.tzinfo) return imbalance_df def get_ohlcv_df( client: db.Historical, dataset: str, symbol: str, start: dt.datetime, end: dt.datetime, ) -> pd.DataFrame: """Get OHLCV schema data""" ohlcv_df = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="ohlcv-1s", start=start, end=end, ).to_df(tz=start.tzinfo) return ohlcv_df def plot_closing_auction( venue: str, symbol: str, imbalance_df: pd.DataFrame, ohlcv_df: pd.DataFrame, start_time: dt.time, end_time: dt.time, ) -> None: imbalance_df = imbalance_df.between_time(start_time, end_time) ohlcv_df = ohlcv_df.between_time(start_time, end_time) ohlcv_df["cumulative_volume"] = ohlcv_df["volume"].cumsum() # Create the subplots _, (ax1, ax2) = plt.subplots( 2, 1, figsize=(14, 9), sharex=True, gridspec_kw={"height_ratios": [3, 1]}, ) # First subplot will display prices ax1.plot( imbalance_df.index, imbalance_df["ref_price"], label="Reference Price", drawstyle="steps-pre", ) ax1.plot( ohlcv_df.index, ohlcv_df["close"], label="Last Price", drawstyle="steps-post", ) ax1.plot( imbalance_df.index, imbalance_df["uncrossing_price"], label="Theoretical Uncrossing Price", drawstyle="steps-post", ) ax1.set_ylabel("Price") ax1.grid(True) ax1.legend(loc="best") ax1.set_title(f"{venue} Closing Auction - {symbol}") # Second subplot will display traded volume and auction paired quantity ax2.plot( ohlcv_df.index, ohlcv_df["cumulative_volume"], label="Cumulative Volume", drawstyle="steps-post", ) ax2.plot( imbalance_df.index, imbalance_df["paired_qty"], label="Paired Quantity", drawstyle="steps-post", ) ax2.set_xlabel("Time (ET)") ax2.set_ylabel("Volume") ax2.grid(True) ax2.legend(loc="best") ax2.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M:%S", tz=start_time.tzinfo)) plt.xticks(rotation=45) plt.show() # Set parameters tz = ZoneInfo("America/New_York") start = dt.datetime(2025, 11, 5, 15, 58, 0, tzinfo=tz) end = dt.datetime(2025, 11, 5, 16, 0, 1, tzinfo=tz) client = db.Historical("$YOUR_API_KEY") # First, we'll look at SPY data on NYSE Arca (ARCX.PILLAR) arcx_symbol = "SPY" arcx_dataset = "ARCX.PILLAR" arcx_ohlcv_df = get_ohlcv_df(client, arcx_dataset, arcx_symbol, start, end) arcx_imbalance_df = get_imbalance_df(client, arcx_dataset, arcx_symbol, start, end) arcx_imbalance_df = arcx_imbalance_df.rename(columns={"ind_match_price": "uncrossing_price"}) arcx_imbalance_df = arcx_imbalance_df[["ref_price", "uncrossing_price", "paired_qty", "symbol"]] print(arcx_imbalance_df) plot_closing_auction( "ARCX", arcx_symbol, arcx_imbalance_df, arcx_ohlcv_df, dt.time(15, 58, 0, tzinfo=tz), dt.time(16, 0, 0, tzinfo=tz), ) # Next, we'll look at QQQ data on Nasdaq (XNAS.ITCH) xnas_symbol = "QQQ" xnas_dataset = "XNAS.ITCH" xnas_ohlcv_df = get_ohlcv_df(client, xnas_dataset, xnas_symbol, start, end) xnas_imbalance_df = get_imbalance_df(client, xnas_dataset, xnas_symbol, start, end) xnas_imbalance_df = xnas_imbalance_df.rename(columns={"cont_book_clr_price": "uncrossing_price"}) xnas_imbalance_df = xnas_imbalance_df[["ref_price", "uncrossing_price", "paired_qty", "symbol"]] print(xnas_imbalance_df) plot_closing_auction( "XNAS", xnas_symbol, xnas_imbalance_df, xnas_ohlcv_df, dt.time(15, 58, 0, tzinfo=tz), dt.time(16, 0, 0, tzinfo=tz), ) ``` ## Results First, we'll take a look at the closing auction for SPY on NYSE Arca. > **See also** > > See the [NYSE Arca auction documentation](https://www.nyse.com/markets/nyse-arca/trading-info#auctions) for more information. ```plaintext ref_price uncrossing_price paired_qty symbol ts_recv 2025-11-05 15:58:00.061423353-05:00 677.715 684.49 951885 SPY 2025-11-05 15:58:01.055768239-05:00 677.760 676.22 1193528 SPY 2025-11-05 15:58:02.054678442-05:00 677.760 676.50 1194940 SPY 2025-11-05 15:58:03.056152198-05:00 677.760 676.50 1195106 SPY 2025-11-05 15:58:04.054268311-05:00 677.720 676.18 1200714 SPY ... ... ... ... ... 2025-11-05 15:59:55.052843213-05:00 677.670 677.81 1401805 SPY 2025-11-05 15:59:56.050469422-05:00 677.600 677.76 1402565 SPY 2025-11-05 15:59:57.051653917-05:00 677.590 677.72 1402565 SPY 2025-11-05 15:59:58.052726733-05:00 677.530 677.68 1403065 SPY 2025-11-05 15:59:59.050924091-05:00 677.470 677.60 1403218 SPY ``` ![NYSE Arca Auction imbalance](https://databento.com/docs/auction-imbalance-0.svg.md) Next, we'll look at the closing auction for QQQ on Nasdaq. You'll notice that in both examples, these theoretical uncrossing prices will converge towards the last traded price as we approach the close. > **See also** > > See the [Nasdaq Closing Cross documentation](https://www.nasdaqtrader.com/content/productsservices/Trading/ClosingCrossfaq.pdf) for more information. ```plaintext ref_price uncrossing_price paired_qty symbol ts_recv 2025-11-05 15:58:00.060035531-05:00 623.49 621.42 677169 QQQ 2025-11-05 15:58:01.019052477-05:00 623.52 622.10 677696 QQQ 2025-11-05 15:58:02.055023604-05:00 623.52 622.06 678796 QQQ 2025-11-05 15:58:03.010276615-05:00 623.54 622.20 678836 QQQ 2025-11-05 15:58:04.040411490-05:00 623.47 622.00 699336 QQQ ... ... ... ... ... 2025-11-05 15:59:55.008542818-05:00 623.43 623.43 827157 QQQ 2025-11-05 15:59:56.099171134-05:00 623.37 623.37 826774 QQQ 2025-11-05 15:59:57.059011900-05:00 623.36 623.36 826774 QQQ 2025-11-05 15:59:58.025345576-05:00 623.27 623.27 826393 QQQ 2025-11-05 15:59:59.032786745-05:00 623.31 623.31 826393 QQQ ``` ![Nasdaq Auction imbalance](https://databento.com/docs/auction-imbalance-1.svg.md) ## Futures | Title | Subtitle | Image | Supported languages | | [Futures: Introduction](https://databento.com/docs/examples/futures/futures-introduction.md) | Get started with futures and symbology. Volumes, tick size, expirations, and more. | ![Futures introduction image](https://databento.com/docs/futures-introduction.svg.md) | Python, Futures, Historical | | [Volume, open interest, and settlement prices](https://databento.com/docs/examples/futures/retrieving-oi-and-settlement-prices.md) | Retrieve volume, open interest, and settlement prices from the statistics schema. | ![Futures statistics](https://databento.com/docs/retrieving-oi-and-settlement-prices-thumbnail.svg.md) | Python, Futures, Historical | | [Futures trading hours](https://databento.com/docs/examples/futures/trading-hours.md) | Get trading hours for a futures product. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Futures, Historical | | [Get options chain for a futures product](https://databento.com/docs/examples/futures/futures-product-options.md) | Get daily, weekly, and monthly options for a futures product. | ![Image placeholder](https://databento.com/docs/futures-product-options-thumbnail.svg.md) | Python, Futures, Historical | | [Determine publishing times for statistics](https://databento.com/docs/examples/futures/statistics-schedule.md) | Discover when different types of statistics are published for an instrument. | ![Image placeholder](https://databento.com/docs/statistics-schedule-thumbnail.svg.md) | Python, Futures, Historical | # Futures: Introduction > **Info** > > This article introduces additional concepts to get started with futures data on > Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll show how to: - Find a futures dataset - Find the 10 futures contracts with the highest volume - Use instrument definitions to get the tick size, expiration, and matching algorithm of an instrument - Stream live BBO data - Use parent symbology to fetch all contracts expirations - Use continuous contract symbology to get the lead month contract We'll also highlight any special conventions of futures datasets on Databento that differ from those of other asset classes. ## Finding a futures dataset To use futures data on Databento, first identify the dataset that you want from our [data catalog]($PLATFORM_HOSTNAME) and go to its detail page. Here, you can find its dataset ID at the top left of the page. ![Dataset ID](https://databento.com/docs/futures-dataset-id.png.md) For this example, we'll use the [CME Globex MDP 3.0]($PLATFORM_HOSTNAME/datasets/GLBX.MDP3) dataset, whose dataset ID is `GLBX.MDP3`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## Finding futures contracts with highest volume A quick way to find the most actively-traded futures contracts, across all expirations, is to fetch the daily volumes from the OHLCV-1d schema. (You can also get a similar result using the `statistics` schema, which provides the official daily settlement prices and trade volumes.) ```python import databento as db # Create historical client client = db.Historical("$YOUR_API_KEY") def rank_by_volume(top=10): # Request OHLCV-1d data data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols="ALL_SYMBOLS", schema="ohlcv-1d", start="2023-08-15" ) # Convert to DataFrame and filter for top 10 instruments by volume df = data.to_df() return df.sort_values(by="volume", ascending=False)["instrument_id"].to_list()[:top] top_instruments = rank_by_volume() print(top_instruments) ``` This returns the following list of numeric instrument IDs, from the `instrument_id` field. ```json [338574, 3445, 404144, 9235, 2922, 2130, 72156, 399495, 225833, 1562] ``` ## Using instrument definitions to get tick size, expiration, and matching algorithm In the case of CME Globex MDP 3.0, these instrument IDs are sourced from tag 48-SecurityID of the original [Security Definition messages](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Security+Definition+-+Futures). Instrument IDs are necessary for many order routing and post-trade scenarios with the exchange, but can be hard to work with, so we will print their raw symbols instead. Let's extract useful properties of these instruments, like their tick sizes, expiration dates, and matching algorithms. ```python def get_symbol_properties(instrument_id_list): # Request definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", stype_in="instrument_id", symbols=instrument_id_list, schema="definition", start="2023-08-15", ) # Convert to DataFrame df = data.to_df() return df[["instrument_id", "raw_symbol", "min_price_increment", "match_algorithm", "expiration"]] print(get_symbol_properties(top_instruments)) ``` ```json instrument_id raw_symbol min_price_increment match_algorithm expiration ts_recv 2023-08-15 00:00:00+00:00 225833 ZBU3 0.031250 F 2023-09-20 17:01:00+00:00 2023-08-15 00:00:00+00:00 1562 SR3Z3 0.005000 A 2024-03-19 21:00:00+00:00 2023-08-15 00:00:00+00:00 2922 MESU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 399495 TNU3 0.015625 F 2023-09-20 17:01:00+00:00 2023-08-15 00:00:00+00:00 338574 ZNU3 0.015625 F 2023-09-20 17:01:00+00:00 2023-08-15 00:00:00+00:00 3445 ESU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 72156 ZTU3 0.003906 K 2023-09-29 17:01:00+00:00 2023-08-15 00:00:00+00:00 9235 MNQU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 2130 NQU3 0.250000 F 2023-09-15 13:30:00+00:00 2023-08-15 00:00:00+00:00 404144 ZFU3 0.007812 F 2023-09-29 17:01:00+00:00 ``` Observe that the `matching_algorithm` values are in their raw values passed through from the exchange. The full list of supported instrument definition fields and info about the output can be found on the exchange's [specifications page](https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Market+Data+Security+Definition+-+Futures). ## Streaming live BBO data While highly liquid futures contracts generally maintain narrow bid-ask spreads, certain market conditions can lead to this spread widening. Stream our [BBO-1s schema](https://databento.com/docs/schemas/bbo.md) to monitor the current best bid and best offer, subsampled at 1-second intervals. This example uses the BBO-1s schema for top-of-book information, but other similar schemas exist that may better suit your use case. Read more about these schemas on our [MBP-1 vs. TBBO vs. BBO schemas page](https://databento.com/docs/faqs/difference-between-mbp-and-tbbo.md). > **Info** > > This example requires a live license to `GLBX.MDP3`. Visit our [live data portal]($PLATFORM_HOSTNAME/portal/live-data#manage) > to sign up. ```python import databento as db # Enable basic logging db.enable_logging("INFO") # Create a live client live_client = db.Live("$YOUR_API_KEY") # Subscribe to the BBO-1s schema for the continuous NQ contract live_client.subscribe( dataset="GLBX.MDP3", schema="bbo-1s", symbols="NQ.v.0", stype_in="continuous", ) # Add a print callback live_client.add_callback(print) # Start the live client to begin streaming live_client.start() # Run the stream for 15 seconds before closing live_client.block_for_close(timeout=15) ``` > **Info** > > If you do not see any output, it could be because the markets are closed. > See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) to utilize [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=42002475, ts_event=1769047300419171930, pretty_ts_event='2026-01-22T02:01:40.419171930Z', stype_in=None, stype_in_symbol='NQ.v.0', stype_out=None, stype_out_symbol='NQH6', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') BboMsg(ts_recv=1768960877000000000, pretty_ts_recv='2026-01-21T02:01:17.000000000Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42002475, ts_event=1768960876926278195, pretty_ts_event='2026-01-21T02:01:16.926278195Z', side='B', price=25201000000000, pretty_price=25201, size=1, flags=LAST (128), sequence=151272098, bid_px_00=25200750000000, pretty_bid_px_00=25200.75, ask_px_00=25201500000000, pretty_ask_px_00=25201.5, bid_sz_00=2, ask_sz_00=1, bid_ct_00=2, ask_ct_00=1) SystemMsg(rtype=, publisher_id=0, instrument_id=0, ts_event=1768960877000000000, pretty_ts_event='2026-01-21T02:01:17.000000000Z', msg='End of interval for bbo-1s', code=) ... BboMsg(ts_recv=1768960880000000000, pretty_ts_recv='2026-01-21T02:01:20.000000000Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42002475, ts_event=1768960879783497307, pretty_ts_event='2026-01-21T02:01:19.783497307Z', side='B', price=25203250000000, pretty_price=25203.25, size=1, flags=LAST (128), sequence=151273402, bid_px_00=25202750000000, pretty_bid_px_00=25202.75, ask_px_00=25203750000000, pretty_ask_px_00=25203.75, bid_sz_00=1, ask_sz_00=2, bid_ct_00=1, ask_ct_00=2) SystemMsg(rtype=, publisher_id=0, instrument_id=0, ts_event=1768960880000000000, pretty_ts_event='2026-01-21T02:01:20.000000000Z', msg='End of interval for bbo-1s', code=) BboMsg(ts_recv=1768960881000000000, pretty_ts_recv='2026-01-21T02:01:21.000000000Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42002475, ts_event=1768960879783497307, pretty_ts_event='2026-01-21T02:01:19.783497307Z', side='B', price=25203250000000, pretty_price=25203.25, size=1, flags=LAST (128), sequence=151273771, bid_px_00=25202250000000, pretty_bid_px_00=25202.25, ask_px_00=25203000000000, pretty_ask_px_00=25203, bid_sz_00=2, ask_sz_00=1, bid_ct_00=2, ask_ct_00=1) ``` ## Parent symbology On futures trading venues, it can be tedious to fetch every child instrument and contract expiration (like ESU3, ESZ3, ESU3-ESZ3) for a given parent instrument (like ES). You can use our [parent symbology type](https://databento.com/docs/conventions/symbology.md#parent) to do this, by passing in `stype_in="parent"`. ```python def get_child_instruments(parents=["ZB.FUT", "SR3.FUT"]): # Request definition data for parent symbols data = client.timeseries.get_range( dataset="GLBX.MDP3", symbols=parents, stype_in="parent", schema="definition", start="2023-08-15", ) # Convert to DataFrame df = data.to_df() return df[["instrument_id", "raw_symbol"]] print(get_child_instruments().head()) ``` ```json instrument_id raw_symbol ts_recv 2023-08-15 00:00:00+00:00 34810 SR3:AB 01Y M8 2023-08-15 00:00:00+00:00 45040 SR3M6-SR3Z7 2023-08-15 00:00:00+00:00 51186 SR3Z4-SR3M6 2023-08-15 00:00:00+00:00 22508 SR3:DF H7Z7U8M9 2023-08-15 00:00:00+00:00 350151 SR3:SB PK M4-M5 ``` Alternatively, you can replicate this logic by requesting definition data for all symbols and filtering on the `asset` field. The `asset` field is populated with the root of the parent symbol. Databento's web portal only exposes parent products, and not child instruments. When you set up a batch download through the web portal, note that you're making a parent symbology request and you'll receive interleaved data from multiple instruments. If you want to set up a batch download of individual child instruments, you must use our API instead. ## Continuous contract symbology Likewise, it's tedious to track the lead month contract of a futures product over a long period of time, due to rollovers. You can use our [continuous contract symbology type](https://databento.com/docs/conventions/symbology.md#continuous) to get a single symbol that is pegged to the lead month contract, by passing in `stype_in="continuous"`. For example, let's plot the two lead month ES contracts `ES.n.0` and `ES.n.1`. ```python import databento as db import matplotlib.pyplot as plt client = db.Historical("$YOUR_API_KEY") dataset = "GLBX.MDP3" symbols = ["ES.n.0", "ES.n.1"] start = "2024" data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="ohlcv-1d", stype_in="continuous", symbols=symbols, start=start, ) df = data.to_df() df.groupby("symbol")["close"].plot( xlabel="Date", ylabel="Price", ) plt.legend() plt.show() ``` ![ES plot](https://databento.com/docs/futures-continuous-0.svg.md) > **Tip** > > If you'd like to see the two calendar front month contracts instead, you may use `ES.c.0` and > `ES.c.1`. However, in most cases these will resolve to the same symbols as `ES.n.0` and `ES.n.1` > respectively because open interest tends to decay with increasing time to expiration. Many futures products reflect seasonality in commodities or term structure in fixed income, so the nearest calendar month may not be the lead month. In such cases, you need to specify how you want to resolve the lead month, also known as a **roll rule**. For example, you could use `ES.v.0` to resolve the lead month by volume instead of open interest. ## Special conventions for futures on Databento - **Weekly trading session.** Unlike many venues, CME Globex has a weekly trading session. This affects how you process MBO data. We provide a synthetic snapshot of the book at the start of each UTC day to make it easier to start from any day of the week. - **User-defined instruments and spreads.** CME Globex has a large number of [user-defined instruments](https://www.cmegroup.com/confluence/display/EPICSANDBOX/User+Defined+Spread+-+UDS). While many vendors do not expose these and their raw symbols may be foreign to a user who's seeing these for the first time, Databento includes all of them as many are highly liquid and active. - **Asynchronous trade publication.** CME Globex prints fills and order deletions associated with the fills asynchronously, with the fills published before the deletions. This is unlike most venues, which treat the individual fill and corresponding order deletion as a single atomic event. You may choose to preemptively update your book based on trades or fills, or wait until the corresponding deletes, which we represent with action `C`. - **Implied book.** CME Globex has implied matching. If a trade is partially filled by contra liquidity on the implied book, we show the full quantity of the trade but only the fill quantities on the direct book. A full implied trade will have trade side `N`. - **Inverted spreads.** CME Globex's matching engine has various price limits and circuit breakers. The spread may be inverted during a trading halt. - **No rollover back-adjustments.** Our continuous contract symbology is a notation that maps to an actual, tradable instrument on any given date. The continuous contract prices provided are the original, unadjusted prices. We don't create a synthetic time series by back-adjusting the prices to remove jumps during rollovers. > **See also** > > [Options on futures introduction](https://databento.com/docs/examples/options/options-on-futures-introduction.md) and [continuous contracts](https://databento.com/docs/conventions/symbology.md#continuous). # Volume, open interest, and settlement prices The [statistics schema](https://databento.com/docs/schemas/statistics.md) provides official summary statistics of each instrument that's published by the venue. Each statistics record contains the value or values associated with only a single statistics type, indicated by the `stat_type` field. This makes the schema flexible and forwards-compatible, but can make it harder to work with. ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve daily statistics for a [parent symbol](https://databento.com/docs/conventions/symbology.md#parent). We'll download statistics data for a parent symbol and use the `StatType` enum to filter for cleared volume, open interest, and settlement price statistics. Next, we'll filter for statistics on outright futures only, using data found in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). Then, we'll group statistics by symbol and date to create a table for each date and symbol. In the end, we'll generate a plot which shows open interest and cleared volume over time. ## ts_ref Many statistics are published daily or refer to a specific trading session. When provided by the publisher, the trading session associated with the statistic is indicated with the `ts_ref` field. Depending on the dataset, statistics can be published at various times, so you may need to look at the next UTC date for a given statistic. For statistics that are published multiple times, you should always refer to the most recently published value. > **See also** > > See our [statistics schedule example](https://databento.com/docs/examples/futures/statistics-schedule.md) which shows how you can determine when the various statistics types are published. ## Example ```python import databento as db import matplotlib.pyplot as plt # Set parameters dataset = "GLBX.MDP3" product = "ZC" start = "2025-05-04" end = "2025-05-19" # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Request definition data and convert to DataFrame def_df = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=f"{product}.FUT", stype_in="parent", start=start, end=end, ).to_df() # Filter for outright futures only and keep the last definition def_df = def_df[def_df["instrument_class"] == db.InstrumentClass.FUTURE] def_df = def_df[["instrument_id", "raw_symbol", "expiration"]] def_df = def_df.drop_duplicates(subset=["instrument_id"], keep="last") # Request statistics data and convert to DataFrame stats_df = client.timeseries.get_range( dataset=dataset, schema="statistics", symbols=f"{product}.FUT", stype_in="parent", start=start, end=end, ).to_df() # Filter for daily statistics on outright futures stats_df = stats_df.merge(def_df, on="instrument_id") stats_df = stats_df.rename(columns={"raw_symbol": "Symbol"}) # Only keep the date component of `ts_ref` stats_df["Trade date"] = stats_df["ts_ref"].dt.date # Create columns for each statistic stats_df["Settlement price"] = stats_df[stats_df["stat_type"] == db.StatType.SETTLEMENT_PRICE][ "price" ] stats_df["Cleared volume"] = stats_df[stats_df["stat_type"] == db.StatType.CLEARED_VOLUME][ "quantity" ] stats_df["Open interest"] = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST]["quantity"] # Groupby and keep the last published record for each statistic stats_df = ( stats_df.groupby(["Trade date", "Symbol"]).agg("last").sort_values(["Trade date", "expiration"]) ) stats_df["Cleared volume"] = stats_df["Cleared volume"].fillna(0).astype(int) stats_df["Open interest"] = stats_df["Open interest"].fillna(0).astype(int) stats_df = stats_df[["Settlement price", "Cleared volume", "Open interest"]] print(stats_df) # Now, we'll plot aggregated volume and open interest statistics for this product date_df = stats_df[["Cleared volume", "Open interest"]].groupby("Trade date").agg("sum") ax_vol = date_df["Cleared volume"].plot.bar(ylabel="Cleared volume") ax_oi = ( date_df["Open interest"] .reset_index() .plot.line( ax=ax_vol, color="C1", marker="o", ylabel="Open interest", secondary_y=True, mark_right=False, legend=False, ) ) plt.gcf().legend(bbox_to_anchor=(0, 1), loc="upper left", bbox_transform=ax_vol.transAxes) plt.setp([ax_vol.get_xticklabels(), ax_oi.get_xticklabels()], rotation=45) plt.title(f"{product} cleared volume and open interest") plt.tight_layout() plt.show() ``` ## Result ```plaintext Settlement price Cleared volume Open interest Trade date Symbol 2025-05-02 ZCK5 461.25 2624 1966 ZCN5 469.00 163656 665836 ZCU5 440.00 60030 314718 ZCZ5 450.25 93701 432738 ZCH6 464.50 10615 50264 ... ... ... ... 2025-05-16 ZCN7 475.25 12 139 ZCU7 453.75 5 12 ZCZ7 455.50 17 630 ZCN8 474.50 0 7 ZCZ8 458.00 0 18 [174 rows x 3 columns] ``` The table is similar to the one found on the [CME website](https://www.cmegroup.com/markets/agriculture/grains/corn.volume.html) that shows these statistics for every active contract. The plot shows the aggregated volume and open interest for all active contracts for a product. ![Statistics plot](https://databento.com/docs/retrieving-oi-and-settlement-prices-0.svg.md) # Futures trading hours Tracking exchange trading hours is an essential task for all market participants. Venues can modify their regular trading hours over time, so having an up-to-date record of these schedules can aid in organizing historical data. Since trading hours are typically modified during holidays, it's important to be aware of these altered schedules. ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical.md) to get trading hours for ES futures. We'll look at two holidays to show how trading hours are modified. We'll use the [status schema](https://databento.com/docs/schemas/status.md) for this example. This schema provides updates about the trading session. These updates can be scheduled changes, or they can be unscheduled, such as volatility halts or other market integrity controls. We'll filter on the `reason` field to look for scheduled changes. We'll also look at the `is_trading` column, since not all changes in market state result in a change to trading functionality. ## Example ```python import datetime as dt import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2024-12-22" end = "2025-01-05" # Get status data status_data = client.timeseries.get_range( dataset=dataset, schema="status", symbols=f"{product}.v.0", stype_in="continuous", start=start, end=end, ) # Convert to DataFrame status_df = status_data.to_df() # Filter out midnight snapshots status_df = status_df[status_df.index.time != dt.time.min] # Keep scheduled changes resulting in a change in trading state status_df = status_df[status_df["reason"] == db.StatusReason.SCHEDULED] status_df = status_df[status_df["trading_event"] == db.TradingEvent.NONE] status_df = status_df[status_df["is_trading"] != status_df["is_trading"].shift()] # Clean up for readability status_df = status_df.set_index("ts_event") status_df = status_df[["symbol", "is_trading"]] print(status_df) ``` CME Globex can have different trading hours for different products. For ES futures, there is an early close on Christmas Eve (2024-12-24), as well as a full close on Christmas (2024-12-25) and New Year's Day (2025-01-01). ```plaintext symbol is_trading ts_event 2024-12-22 23:00:00+00:00 ES.v.0 Y 2024-12-23 22:00:00+00:00 ES.v.0 N 2024-12-23 23:00:00+00:00 ES.v.0 Y 2024-12-24 18:15:00+00:00 ES.v.0 N 2024-12-25 23:00:00+00:00 ES.v.0 Y 2024-12-26 22:00:00+00:00 ES.v.0 N 2024-12-26 23:00:00+00:00 ES.v.0 Y 2024-12-27 22:00:00+00:00 ES.v.0 N 2024-12-29 23:00:00+00:00 ES.v.0 Y 2024-12-30 22:00:00+00:00 ES.v.0 N 2024-12-30 23:00:00+00:00 ES.v.0 Y 2024-12-31 22:00:00+00:00 ES.v.0 N 2025-01-01 23:00:00+00:00 ES.v.0 Y 2025-01-02 22:00:00+00:00 ES.v.0 N 2025-01-02 23:00:00+00:00 ES.v.0 Y 2025-01-03 22:00:00+00:00 ES.v.0 N ``` # Get options chain for a futures product Identifying all options for a futures product can be difficult because exchange naming conventions may split the options chain across different [root symbols](https://databento.com/docs/conventions/symbology.md#parent). On CME, this can occur in two ways: - Futures and options may use entirely different roots, e.g., `CL.FUT` and `LO.OPT` for crude oil. - Certain expirations may appear under a different root symbol than their underlying. `E1AQ5 C5125` is a weekly option on `ESU5` but has the root `E1A`, so it is included in `E1A.OPT` instead of `ES.OPT`. > **See also** > > CME's Weekly Options on Futures page provides more examples of the second scenario above, where weekly contracts are denoted by different root symbols than their underlying futures. ## Overview This example demonstrates how to use the [historical API](https://databento.com/docs/api-historical/client.md) to find all options linked to a futures product, even when they're listed under different root symbols. We'll request data using the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) for all symbols, where the `asset` field represents the product's root symbol and the `instrument_class` field identifies its outright futures. We'll then retrieve every option whose `underlying` matches any of those futures for the complete options chain. We'll also use the `expiration` field to demonstrate how to filter these options by their Days to Expiration (DTE). ## Example ```python import databento as db import matplotlib.pyplot as plt import pandas as pd from matplotlib.patches import Patch def get_futures( product: str, def_df: pd.DataFrame, ) -> pd.DataFrame: """ Get definitions for all outright futures for a product. """ outright_futures = def_df[ (def_df["instrument_class"] == db.InstrumentClass.FUTURE) & (def_df["asset"] == product) ] return outright_futures def get_options( product: str, def_df: pd.DataFrame, expiration_filter: pd.Interval | None = None, ) -> pd.DataFrame: """ Get definitions for all options for a product. Optionally filter by expiration. """ # Get the symbols for outright futures for the product outright_futures = get_futures(product, def_df)["raw_symbol"].to_list() # Get all options that have any of the outright futures as an underlying options_df = def_df[ (def_df["underlying"].isin(outright_futures)) & (def_df["instrument_class"].isin(["C", "P"])) ].copy() # Create a `days_to_expiration` field from `ts_recv` and `expiration` options_df["days_to_expiration"] = ( options_df["expiration"].dt.normalize() - options_df.index.normalize() ).dt.days if expiration_filter is None: return options_df # Filter for specified expiration range filtered_options = options_df[ options_df["days_to_expiration"].between( expiration_filter.left, expiration_filter.right, inclusive="both", ) ] return filtered_options def plot_volume_by_dte( product: str, date: str, def_df: pd.DataFrame, ohlcv_df: pd.DataFrame, ) -> None: dte_ranges = [ (0, 0, "0"), (1, 2, "1-2"), (3, 4, "3-4"), (5, 9, "5-9"), (10, 1000, "10+"), ] # Create pie chart fig, ax = plt.subplots(figsize=(10, 6)) dte_handles = [] dte_volumes = [] for idx, (start, end, label) in enumerate(dte_ranges): defs = get_options(product, def_df, pd.Interval(start, end, closed="both")) vol = ohlcv_df[ohlcv_df["instrument_id"].isin(defs["instrument_id"])]["volume"].sum() patch = Patch(facecolor=f"C{idx}", label=f"{label} DTE") dte_handles.append(patch) dte_volumes.append(vol) ax.pie(dte_volumes, autopct="%1.1f%%", startangle=90, pctdistance=1.2) ax.set_title(f"{product} options volume on {date} by Days To Expiration (DTE)") fig.legend(handles=dte_handles, loc="upper left", bbox_to_anchor=(0.1, 0.9)) plt.show() # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2025-08-04" # Create a historical client client = db.Historical("$YOUR_API_KEY") # Download ohlcv-1d data for ALL_SYMBOLS for one day ohlcv_df = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols="ALL_SYMBOLS", start=start, ).to_df() # Download definition data for ALL_SYMBOLS for one day def_df = client.timeseries.get_range( dataset=dataset, schema="definition", symbols="ALL_SYMBOLS", start=start, ).to_df() # Create a range of DTE to filter for # You can filter for 1-3 DTE with pd.Interval(1, 3, closed="both") zero_dte_options = get_options(product, def_df, pd.Interval(0, 0, closed="both")) print(zero_dte_options[["raw_symbol", "expiration", "strike_price", "days_to_expiration"]]) # Now, plot of total volume by different DTE ranges plot_volume_by_dte(product, start, def_df, ohlcv_df) ``` ## Result ```plaintext raw_symbol expiration strike_price days_to_expiration ts_recv 2025-08-04 00:00:00+00:00 E1AQ5 C5125 2025-08-04 20:00:00+00:00 5125.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P6600 2025-08-04 20:00:00+00:00 6600.0 0 2025-08-04 00:00:00+00:00 E1AQ5 C4100 2025-08-04 20:00:00+00:00 4100.0 0 2025-08-04 00:00:00+00:00 E1AQ5 C7200 2025-08-04 20:00:00+00:00 7200.0 0 2025-08-04 00:00:00+00:00 E1AQ5 C4700 2025-08-04 20:00:00+00:00 4700.0 0 ... ... ... ... ... 2025-08-04 00:00:00+00:00 E1AQ5 P4400 2025-08-04 20:00:00+00:00 4400.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P7600 2025-08-04 20:00:00+00:00 7600.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P5735 2025-08-04 20:00:00+00:00 5735.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P5715 2025-08-04 20:00:00+00:00 5715.0 0 2025-08-04 00:00:00+00:00 E1AQ5 P5075 2025-08-04 20:00:00+00:00 5075.0 0 [626 rows x 4 columns] ``` You could use the `raw_symbol` values from this filtered list and pass them into another data request. This is demonstrated below which uses the [OHLCV-1d](https://databento.com/docs/schemas/ohlcv.md) schema to calculate the total volume for all options based on Days to Expiration (DTE). ![DTE volume plot](https://databento.com/docs/futures-product-options-0.svg.md) # Determine publishing times for statistics Official exchange-published statistics for an instrument can be published at various times. There are statistics that are published continuously throughout the day, while other statistics are only published a few times per day. Additionally, this publishing behavior may be different between venues. ## Overview We'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve all statistics for an instrument to determine when statistics are published. We'll request data from the [statistics schema](https://databento.com/docs/schemas/statistics.md), which contains all official statistics published by the venue. We'll also request data from the [status schema](https://databento.com/docs/schemas/status.md). This status schema data will show how changes in market state can affect publishing times for various statistics. We'll plot this data to show when each statistic is published. ## Example ```python import datetime as dt import databento as db import matplotlib.pyplot as plt import pandas as pd from matplotlib.dates import DateFormatter, MinuteLocator # Create mapping of statistic type to human-readable name stat_type_dict = {x.value: x.name.replace("_", " ").capitalize() for x in db.StatType.variants()} def plot_statistics_distribution( symbol: str, stats_data: pd.DataFrame, status_data: pd.DataFrame, ): """Plot publishing time distribution of each statistic type""" stats_data = ( stats_data.groupby("stat_type") .resample("15min", include_groups=False) .size() .unstack(level=0) .fillna(0) ).transpose() stats_data = stats_data.rename(index=stat_type_dict) _, ax = plt.subplots(figsize=(16, 8)) times = pd.to_datetime(stats_data.columns) time_width = pd.Timedelta(minutes=15) # Plot horizontal bars for each statistic type for idx, stat_type in enumerate(stats_data.index): mask = stats_data.loc[stat_type] > 0 if mask.any(): ax.barh( idx, width=time_width * 1.1, left=times[mask] - time_width / 2, label=stat_type, ) # Plot vertical lines for changes in market state for idx, row in status_data.iterrows(): ax.axvline( x=idx, color="dimgray", linestyle=":", linewidth=1.5, ) ax.text( x=idx, y=ax.get_ylim()[1], s=str(row["action"]), ha="left", va="top", rotation=-90, color="dimgray", ) # Format axes ax.set( yticks=range(len(stats_data)), yticklabels=stats_data.index, xlabel="Time (UTC)", ylabel="Stat Type", title=f"{symbol} Statistics Message Distribution", ) ax.xaxis.set_major_formatter(DateFormatter("%H:%M")) ax.xaxis.set_major_locator(MinuteLocator([0, 60])) ax.set_axisbelow(True) ax.grid() plt.setp(ax.xaxis.get_majorticklabels(), rotation=45, ha="right") plt.tight_layout() plt.show() # Set parameters dataset = "IFUS.IMPACT" product = "CC" start = "2025-09-17" client = db.Historical("$YOUR_API_KEY") # Request statistics data for the continuous contract stats_df = client.timeseries.get_range( dataset=dataset, symbols=f"{product}.v.0", stype_in="continuous", schema="statistics", start=start, ).to_df() # Request status data for the continuous contract status_df = client.timeseries.get_range( dataset=dataset, symbols=f"{product}.v.0", stype_in="continuous", schema="status", start=start, ).to_df() # Filter out midnight snapshot status_df = status_df[status_df.index.time != dt.time.min] # Only keep certain market states # For a full list of variants, see https://databento.com/docs/schemas-and-data-formats/status#status-action-variants status_map = {1: "Pre-Open", 7: "Open", 12: "Close"} status_df = status_df[status_df["action"].isin(status_map)] status_df = status_df[status_df["trading_event"] != 1] status_df["action"] = status_df["action"].map(status_map) # Print out the number of occurrences for each statistic type print(stats_df["stat_type"].map(stat_type_dict).value_counts()) # Plot distribution of statistic publishing time plot_statistics_distribution( symbol=product, stats_data=stats_df, status_data=status_df, ) ``` ## Result Taking a look at this example for Cocoa futures on ICE Futures US, you will notice how certain statistics, such as cleared volume and VWAP, are published continuously throughout the day. Other statistics, such as open interest and settlement price, are published only a few times per day. ```plaintext stat_type Trading session high price 4482 Trading session low price 4482 Vwap 4482 Cleared volume 4482 Indicative opening price 3060 Opening price 3 Open interest 3 Settlement price 3 Name: count, dtype: int64 ``` ![Statistics publishing times](https://databento.com/docs/statistics-schedule-0.svg.md) ## Indices | Title | Subtitle | Image | Supported languages | | [Indices: Introduction](https://databento.com/docs/examples/indices/indices-introduction.md) | Get started with index data from the Databento Core Indices dataset. | ![Index interval distribution](https://databento.com/docs/indices-introduction-thumbnail.svg.md) | Python, Indices, Historical | | [Implied SPX index from ES futures curve](https://databento.com/docs/examples/indices/implied-spx-index.md) | Estimate an S&P 500 spot proxy from the front two ES contracts. | ![ES-implied SPX proxy](https://databento.com/docs/implied-spx-index-thumbnail.svg.md) | Python, Indices, Futures, Historical | | [Filter options by moneyness](https://databento.com/docs/examples/indices/option-moneyness.md) | Calculate moneyness across the SPXW chain and set a volatility-adjusted strike range. | ![ATM Strike Activity](https://databento.com/docs/option-moneyness-thumb-thumbnail.svg.md) | Python, Indices, Options, Historical | | [Filter SPXW strikes by VIX-implied expected move](https://databento.com/docs/examples/indices/spx-and-the-vix.md) | Interpolate the VIX to set a volatility-adjusted strike range for SPXW expirations. | ![Strikes filtered by volatility](https://databento.com/docs/spx-and-the-vix-thumb-thumbnail.svg.md) | Python, Indices, Options, Historical | # Indices: Introduction > **Info** > > This article introduces additional concepts to get started with indices data on > Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll introduce: - The Databento Core Indices dataset - How to discover index update frequencies - How to plot index prices - Special conventions for indices vs. other asset classes offered by Databento ## Using the indices dataset Our [Databento Core Indices feed]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE) includes, but is not limited to: - Flagship benchmarks (e.g. SPX and VIX) - Strategy indices (e.g. PUTR and BXR) - ETF fair values (e.g. TQQQIV and SSOIV) - Crypto values (e.g. BTC1RP and ETH100RP) - US benchmarks (e.g. RUT and RUI) - US market and style indices (e.g. MSTAR and MWMF) - Global equity indices (e.g. MXEA and MXWLD) To use index data on Databento, navigate to our [indices catalog page]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE) and find its dataset ID at the top left of the page. ![Find dataset id](https://databento.com/docs/core-indices-dataset-id.png.md) For this example, we'll use the [Databento Core Indices]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE) (`DBIX.CORE`) dataset. This ID will be passed into the `dataset` parameter of any API or client method. ## Update frequency Unlike market data for order-driven instruments, which updates in response to individual trades or order book updates, index values are calculated and published according to an index-specific schedule. The publication frequency varies by index, ranging from intraday updates every few seconds to values published only once per day. Since these records are not sent out at perfect intervals, there is some variance in the send times. We can estimate the observed update cadence for any index using Databento's [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md), which provides every single update to the index price at the moment it was released. For instance, to find the update frequency of the Morningstar US Target Market Exposure Index (MUFT): ```python import databento as db client = db.Historical(key="$YOUR_API_KEY") muft = client.timeseries.get_range( dataset="DBIX.CORE", symbols=["MUFT"], start="2026-02-18", schema="mbp-1" ).to_df() intervals = muft["ts_event"].diff().dt.total_seconds() print(intervals.describe()) ``` ```plaintext count 1638.000000 mean 14.999921 std 0.305004 min 11.611000 25% 14.961000 50% 15.000000 75% 15.042750 max 18.405000 Name: ts_event, dtype: float64 ``` We can see that MUFT updates approximately every 15 seconds, with small variations in the observed interval between updates. We can use this method for any index in the DBIX dataset. ## Plotting index bid, ask, and price values The official price of an index is calculated from last sale data. However, for some indices, a bid and ask value may also be calculated which is similar to the index value but is calculated from bid and ask prices instead of last sale prices. Databento normalizes these fields to `bid_px_00`, `ask_px_00`, and `price` in the [bbo-1s schema](https://databento.com/docs/schemas/bbo.md). We can plot these values for SPX: > **Info** > > Although these fields may resemble market prices, the bid and ask values are calculated index values > rather than executable orders, and there is no underlying order book depth. ```python import databento as db import matplotlib.pyplot as plt client = db.Historical(key="$YOUR_API_KEY") spx = client.timeseries.get_range( dataset="DBIX.CORE", symbols=["SPX"], start="2026-02-18", schema="bbo-1s", limit=400 ).to_df() prices = spx[["ask_px_00", "price", "bid_px_00"]] prices = prices.rename(columns={ "ask_px_00": "ask", "bid_px_00": "bid"}) prices.plot() plt.show() ``` ![Index price plot](https://databento.com/docs/indices-introduction-0.svg.md) ## Special conventions for indices on Databento - **Source timestamp precision.** Although Databento timestamps are represented in nanoseconds, some venues within DBIX may not provide that same granularity natively. Databento adds no additional source precision beyond that which is provided by the underlying venues. - **Instrument definitions.** DBIX instruments are always normalized with `instrument_class` set to `INDEX`, and `match_algorithm` set to `UNDEFINED`, since there is no matching engine associated with an index feed. - **Indicative index values.** Some DBIX indices under the the `publisher_id` `DBIX.CORE.CGI` may publish indicative values, which are estimated or preliminary values marked as not being final. For these records, Databento applies the `F_PUBLISHER_SPECIFIC` flag, allowing them to be distinguished from normal index values. - See more on our [venue details page](https://databento.com/docs/datasets/dbix-cgif.md). # Implied SPX index from ES futures curve ## Overview In this example, we'll explore how to estimate the implied spot price for the S&P 500 Index using best bid and offer (BBO) data from the ES curve with Databento's [Historical client](https://databento.com/docs/api-historical/client.md). We will then validate the implied price against the true underlying index value from our [Databento Core Indices dataset]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE). Because ES futures trade nearly continuously and update more frequently than the published SPX cash index, they can be used to construct a real-time proxy for the underlying index level. ## Futures overview and cost of carry A [futures contract](https://databento.com/docs/examples/futures/futures-introduction.md) is a standardized legal agreement to buy or sell an underlying asset at a specified price on a future date. Some futures, including ES futures, are cash-settled rather than settled through delivery of the underlying asset. The price of a futures contract at a given expiration is determined by the current spot price, adjusted for factors associated with carrying exposure to the underlying asset until expiration. This adjustment is commonly referred to as the "net cost of carry." For equity index futures, the net cost of carry primarily reflects the financing cost of purchasing the underlying portfolio and the dividend income forgone by holding the futures contract instead. Under the standard model, the futures price can be expressed using the risk-free interest rate and the continuous dividend yield as follows: \[ F_i = S \cdot e^{(R - D)\, T_i} \] Where: - \(F_i\) is the *i*-th futures contract price - \(S\) is the current spot price of the underlying asset - \(R\) is the risk-free interest rate - \(D\) is the continuous dividend yield - \(T_i\) is the time to expiration of the *i*-th futures contract ## Solving for the spot price Interest rates and expected dividends may be difficult to source. However, because ES futures trade across multiple expirations, we can use a mathematical trick to infer their combined effect from the futures curve. If we assume that the continuously compounded net carry rate is approximately constant across the first two maturities, we can estimate an implied net carry rate from their observed prices and use it to recover the underlying spot price. > **Info** > > It is important to note that this assumption simplifies the true nature of the net cost of carry. > In practice, dividends are paid on discrete dates and may be concentrated unevenly across quarters. > Similarly, interest rates also vary by term and are often modeled as piece-wise functions with knots on FOMC days. > > We are also assuming that the implied net carry rate remains constant across maturities. For the first and second contracts: \[ F_1 = S \cdot e^{(R - D)\, T_1} \] \[ F_2 = S \cdot e^{(R - D)\, T_2} \] Dividing the second equation by the first eliminates the unknown spot price: \[ \frac{F_2}{F_1} = e^{(R - D)(T_2 - T_1)} \] Taking the natural logarithm of both sides yields: \[ \ln\!\left(\frac{F_2}{F_1}\right) = (R - D)(T_2 - T_1) \] We can therefore estimate the net carry rate as: \[ R - D = \frac{\ln\!\left(\dfrac{F_2}{F_1}\right)}{T_2 - T_1} \] Now that we have found an estimate for the net carry rate, we can substitute it into the original futures-pricing equation and solve for spot: \[ S = \frac{F_1}{e^{(R - D)\, T_1}} \] Substituting the estimated value of \(R - D\), we obtain an ES-implied proxy for the SPX spot price: \[ S = \frac{F_1}{\exp\!\left(\dfrac{\ln\!\left(\dfrac{F_2}{F_1}\right)}{T_2 - T_1} \cdot T_1\right)} \] With this equation, implementing the calculation using the API becomes straightforward: ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Set time range start = "2026-01-05T00:00:00" end = "2026-01-05T21:00:00" # Set dataset and symbols dataset = "GLBX.MDP3" symbols = ["ES.v.0", "ES.v.1"] # Get 1-second BBO for the front two continuous ES contracts (ES.v.0, ES.v.1) es = client.timeseries.get_range( dataset=dataset, symbols=symbols, start=start, end=end, schema="bbo-1s", stype_in="continuous", stype_out="instrument_id", ) es_df = es.to_df() # Get definitions to map each instrument to its expiration date definitions = client.timeseries.get_range( dataset=dataset, symbols=symbols, start=start, end=end, schema="definition", stype_in="continuous", ) definitions_df = definitions.to_df() # Build a map from instrument_id to expiration expiration_map = definitions_df.groupby("instrument_id")["expiration"].first().to_dict() # Attach each row's expiration to the BBO DataFrame es_df["expiration"] = es_df["instrument_id"].map(expiration_map) # Time to expiration in years (T_i), and the mid price as our price proxy es_df["time_to_expire"] = (es_df["expiration"] - es_df["ts_event"]).dt.total_seconds() / (365 * 24 * 60 * 60) es_df["mid"] = (es_df["bid_px_00"] + es_df["ask_px_00"]) / 2 # Split into the first (F_1) and second (F_2) contracts es0 = es_df[es_df["symbol"] == "ES.v.0"][["mid", "time_to_expire"]] es1 = es_df[es_df["symbol"] == "ES.v.1"][["mid", "time_to_expire"]] # Estimate the net carry rate (R - D) = ln(F_2/F_1) / (T_2 - T_1) r_d = (np.log(es1["mid"] / es0["mid"])) / (es1["time_to_expire"] - es0["time_to_expire"]) # Back out the implied spot: S = F_1 / e^[(R - D) * T_1] spx = es0["mid"] / np.exp(r_d * es0["time_to_expire"]) # Combine the prices into a combined DataFrame proxy_df = pd.DataFrame({"es0": es0["mid"], "es1": es1["mid"], "spx": spx}) print(proxy_df.head(5)) ``` ```plaintext es0 es1 spx ts_recv 2026-01-05 00:00:00+00:00 6907.125 6958.375 6864.956430 2026-01-05 00:00:01+00:00 6907.375 6958.375 6865.410758 2026-01-05 00:00:02+00:00 6907.375 6958.625 6865.206449 2026-01-05 00:00:03+00:00 6907.500 6958.625 6865.433613 2026-01-05 00:00:04+00:00 6907.250 6958.375 6865.183636 ``` In just a few lines of code, we have created an SPX index proxy that updates more quickly than the true index value. However, we still need to validate its accuracy. ## Validation We can use Databento's Core Indices dataset to request the actual underlying index value, check the accuracy of our estimate, and plot the two time series against one another: ```python # Create a request using the DBIX.CORE dataset. index = client.timeseries.get_range( dataset="DBIX.CORE", symbols="SPX", start=start, end=end, schema="bbo-1m", ) index_df = index.to_df() index_df["idx"] = (index_df["bid_px_00"] + index_df["ask_px_00"]) / 2 # Combine data, keeping only rows from the DBIX DataFrame prices_df = pd.merge(left=index_df[["idx"]], right=proxy_df, how="left", left_index=True, right_index=True) prices_df = prices_df[prices_df["spx"].notnull()] # Plot the implied SPX proxy against the true index price plt.figure(figsize=(16, 8)) prices_df["spx"].plot(label="SPX Proxy") prices_df["idx"].plot(label="SPX Index") plt.title("ES-implied SPX spot proxy vs. true SPX spot") plt.xlabel("Date") plt.ylabel("Price") plt.legend() plt.show() ``` ![SPX implied proxy](https://databento.com/docs/implied-spx-index-0.svg.md) ### Error To measure the error between our SPX index proxy and the true SPX index, we can use the Root Mean Squared Error (RMSE), which quantifies the differences (residuals) between predicted and actual values. It squares each difference to penalize larger errors and eliminate negative errors, averages the squared differences, and takes the square root to return the error to the original units. The exact equation is: \[ \text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} \left(\hat{S}_i - S_i\right)^2} \] Where: - \(\hat{S}_i\) is the ES-implied proxy at observation *i* - \(S_i\) is the true index value at observation *i* - \(n\) is the number of observations We can achieve this quite easily in Python with no additional packages: ```python # Calculate RMSE rmse = np.sqrt(((prices_df["spx"] - prices_df["idx"])**2).sum() / len(prices_df)) print("Root Mean Squared Error:", round(rmse, 4)) ``` ```python Root Mean Squared Error: 0.6386 ``` So our estimate is only off by an RMSE of ~0.64 index points. This is an excellent result, but to ensure this metric is reasonable, we still need to inspect the distribution of errors. ### Residuals For a better idea of how the errors are distributed, we can create a residual plot. This plot displays each error as a point on a graph, allowing us to visually inspect whether our estimate is systematically biased in one direction. We can find a consistent bias by measuring the average size of our residuals: \[ \text{Bias} = \frac{1}{n} \sum_{i=1}^{n} \left(\hat{S}_i - S_i\right) \] We can calculate this value in Python and plot it alongside our residuals plot: ```python # Calculate mean error bias = (prices_df["spx"] - prices_df["idx"]).mean() # Plot residuals vs. true index price. plt.scatter(x=prices_df["idx"], y=prices_df["spx"] - prices_df["idx"] ) plt.axhline(bias, color="C1", linestyle="--", label=f"Bias = {bias:.3f}") plt.axhline(0, color="C2", linestyle=":", label="Zero") plt.xlabel("SPX Index") plt.ylabel("Residuals (SPX Proxy - SPX Index)") plt.title("SPX proxy residuals vs. true SPX index") plt.legend() plt.show() print("Bias:", round(bias, 4)) ``` ![SPX residual plot](https://databento.com/docs/implied-spx-index-1.svg.md) ```python Bias: 0.517 ``` This plot shows that our estimate was consistently about half an index point too high, indicating that our simplified cost-of-carry model may introduce structural bias. This bias may vary across days due to the non-linear nature of the cost of carry. However, given that error is measured as follows: \[ \text{RMSE}^2 = \text{Bias}^2 + \sigma^2 \] We can calculate the standard deviation of our estimate around the bias. \[ \sigma = \sqrt{\text{RMSE}^2 - \text{Bias}^2} \] Then, we can calculate this value in Python: ```python variance = rmse**2 - bias**2 print("Standard Deviation:", round(variance**0.5)) ``` ```python Standard Deviation: 0.375 ``` Our estimate has a standard deviation of less than 1/2 of an index point around our bias, meaning that our proxy is quite robust. We can use this proxy to find at-the-money options strikes, find a continuous index level outside of regular trading hours, and compute realized volatility on a tick-by-tick basis. # Filter options by moneyness ## Overview In this example, we'll calculate the moneyness of SPXW options and filter to strikes within 5% of the ATM price. We'll calculate moneyness for each option by combining SPXW option definitions and statistics from Databento's [OPRA dataset]($PLATFORM_HOSTNAME/portal/catalog/opra) with SPX index values from the [Databento Core Indices dataset]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE). ## Moneyness Moneyness describes where an option's strike sits relative to the current price of the underlying. The moneyness of an option is separated into three trading states: - **In-the-money (ITM)**: The option has positive intrinsic value. - **At-the-money (ATM)**: The option's strike price is almost identical to the market price of the underlying asset. - **Out-of-the-money (OTM)**: The option has no intrinsic value. Options closer to ATM typically have higher trading volume. For any option: \[ \text{Moneyness} = \frac{K}{S} - 1 \] Where: - \(K\) is the strike price of the contract - \(S\) is the current price of the underlying asset A negative value places the strike below spot, which is in-the-money for a call and out-of-the-money for a put. The opposite is true for a positive value. Both the sign and the contract type determine the state of an option. For easier readability, we will multiply this value by 100 to express moneyness as a percentage rather than a decimal. ## Calculating moneyness To calculate moneyness, we'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) to retrieve the necessary information for each option, such as `expiration`, `strike_price`, and `instrument_class`. We'll then pull the spot price at market open using the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md). ```python import databento as db import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Fetch data for all expirations for the week of Monday, April 27 start = "2026-04-27" # Fetch the SPX index price from market open spx_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1m", symbols=["SPX"], start=start + "T13:30", end=start + "T13:31", ).to_df() spx = spx_df[spx_df["symbol"] == "SPX"]["open"].iloc[0] # Fetch SPXW options definitions spxw_definitions = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Filter to outright contracts expiring by the end of the week friday_expiry = pd.to_datetime(start, utc=True) + pd.Timedelta(days=4) spxw_definitions = spxw_definitions[spxw_definitions["expiration"] <= friday_expiry] spxw_definitions = spxw_definitions[spxw_definitions["instrument_class"].isin(["P", "C"])] # Drop all unnecessary rows for this process spxw_metadata = spxw_definitions[["instrument_id", "strike_price", "expiration", "instrument_class"]] # Calculate Moneyness per contract (For the sake of this example, we convert to a percentage by multiplying by 100) spxw_metadata["moneyness"] = (spxw_metadata["strike_price"] / spx - 1) * 100 print(spxw_metadata.head()) ``` ```plaintext instrument_id strike_price expiration instrument_class moneyness ts_recv 2026-04-27 10:30:04.059606168+00:00 1191185088 6500.0 2026-05-01 00:00:00+00:00 C -9.125480 2026-04-27 10:30:04.069618621+00:00 1191185095 7150.0 2026-05-01 00:00:00+00:00 C -0.038027 2026-04-27 10:30:04.069618621+00:00 1191185096 5200.0 2026-05-01 00:00:00+00:00 C -27.300384 2026-04-27 10:30:04.079627286+00:00 1191185103 5950.0 2026-05-01 00:00:00+00:00 P -16.814862 2026-04-27 10:30:04.079627286+00:00 1191185104 6300.0 2026-05-01 00:00:00+00:00 C -11.921619 ``` ## Visualizing activity by strike Now that we have calculated the moneyness of each strike price, we can visualize the trading activity within 5% of the opening ATM price. A single SPXW expiration lists strikes across a range far wider than the underlying is likely to move, and most of them barely trade, so any workflow that consumes the chain could benefit from cutting it down to an active region. ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Fetch data for all expirations for the week of Monday, April 27 start = "2026-04-27" # Fetch the SPX index price from market open spx_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1m", symbols=["SPX"], start=start + "T13:30", end=start + "T13:31", ).to_df() spx = spx_df[spx_df["symbol"] == "SPX"]["open"].iloc[0] # Fetch SPXW options definitions spxw_definitions = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Filter to outright contracts expiring by the end of the week friday_expiry = pd.to_datetime(start, utc=True) + pd.Timedelta(days=4) spxw_definitions = spxw_definitions[spxw_definitions["expiration"] <= friday_expiry] spxw_definitions = spxw_definitions[spxw_definitions["instrument_class"].isin(["P", "C"])] # Drop all unnecessary rows for this process spxw_metadata = spxw_definitions[["instrument_id", "strike_price", "expiration", "instrument_class"]] # Calculate Moneyness per contract (For the sake of this example, we convert to a percentage by multiplying by 100) spxw_metadata["moneyness"] = (spxw_metadata["strike_price"] / spx - 1) * 100 # Fetch SPXW options statistics spxw_statistics = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="statistics", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Include only daily volume statistics spxw_volumes = spxw_statistics[spxw_statistics["stat_type"] == db.StatType.CLEARED_VOLUME][["instrument_id", "quantity"]] spxw_volumes = spxw_volumes.groupby("instrument_id").last().reset_index().rename(columns={"quantity": "volume"}) # Merge the metadata and volume dataframes spxw = pd.merge(spxw_volumes, spxw_metadata, on="instrument_id") # Create a reusable function to plot the volume data for each expiration date def volume_plots(axes, plot_df, expiries, alpha=1): for ax, expiry in zip(axes, expiries): # Filter the dataframe to only include options for the current expiration date expiry_df = plot_df[plot_df["expiration"] == expiry] puts = expiry_df[expiry_df["instrument_class"] == "P"] calls = expiry_df[expiry_df["instrument_class"] == "C"] x_values = np.sort(expiry_df["moneyness"].unique()) bar_width = np.median(np.diff(x_values)) * 0.85 # Bar width will be 85% of the median price difference between strikes # Plot the volume for calls and puts as bars, with calls above the x-axis and puts below the x-axis ax.bar(calls["moneyness"], calls["volume"], width=bar_width, label="Calls", alpha=alpha, color="C0") ax.bar(puts["moneyness"], -puts["volume"], width=bar_width, label="Puts", alpha=alpha, color="C1") # Add a vertical line at moneyness = 0 and a horizontal line at volume = 0 ax.axvline(0, linestyle="--", linewidth=1) ax.axhline(0, linewidth=0.8, color="black") ax.set_title(pd.Timestamp(expiry).strftime("%a %b %d"), loc="left") ax.set_ylabel("Trading Volume") # Filter to only include options that are within 5% of the SPX index price plot_df = spxw[spxw["moneyness"].between(-5, 5)] # Create a list of unique expiration dates for the options expiries = sorted(plot_df["expiration"].unique()) # Create the subplots for each expiration date fig, axes = plt.subplots(nrows=len(expiries), ncols=1, figsize=(15, 12), sharex=True) volume_plots(axes, plot_df, expiries) axes[-1].set_xlabel("Moneyness (%)") axes[0].legend() fig.suptitle("SPXW Volume by Expiration", fontsize=16) plt.show() ``` ![Strikes by volume](https://databento.com/docs/option-moneyness-2-0.svg.md) # Filter SPXW strikes by VIX-implied expected move ## Overview In this example, we will use the VIX to define a volatility-adjusted active strike range across SPXW expirations from the [OPRA feed]($PLATFORM_HOSTNAME/portal/catalog/opra) using SPX and VIX index values from Databento's [Databento Core Indices dataset]($PLATFORM_HOSTNAME/portal/catalog/dbix/DBIX.CORE). > **Info** > > This article is a continuation of our [Filter options by moneyness example](https://databento.com/docs/examples/indices/option-moneyness.md). For more details on moneyness, visit that page first. ## The VIX The VIX, or Cboe Volatility Index, is a market index that measures the S&P 500's anticipated volatility in the near future. VIX is derived from the SPX option chain and expresses expected 30-day volatility in annualized terms. For this example, we will focus on 0DTE-5DTE options contracts, so we will instead use shorter-term versions of the VIX: VIX1D and VIX9D, which measure the S&P 500's expected volatility for the current trading day and the next 9 days respectively. Then, we will use these two points to interpolate the implied total variance for each expiration. We use variance instead of volatility here because variance is additive over time. To do this, we first need to convert volatility to annualized variance by squaring: \[ Var_{1D} = (\text{VIX1D})^2 \] \[ Var_{9D} = (\text{VIX9D})^2 \] Next, we will account for the time fraction of each index to get the actual total variance expected over their respective horizons: \[ \text{Total Var}_{1D} = Var_{1D} \times \frac{1}{365} \] \[ \text{Total Var}_{9D} = Var_{9D} \times \frac{9}{365} \] We then linearly interpolate for the target day by calculating where it sits proportionally between day 1 and day 9: \[\text{Total\ Var}_{ND}=\text{Total\ Var}_{1D}+\left(\frac{N-1}{9-1}\right)\times (\text{Total\ Var}_{9D}-\text{Total\ Var}_{1D})\] Finally, we take the square root of the total variance to get the expected move over n days. This converts the variance back into a percentage move, the same unit as moneyness: \[ \text{Expected Move}_{ND} = \sqrt{\text{Total Var}_{ND}} \] We can achieve this very simply in Python: ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Fetch data for all expirations for the week of Monday, April 27 start = "2026-04-27" # Get the opening values for the VIX (not available exactly at open, since the VIX is calculated from SPX options) vix_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1s", symbols=["VIX1D", "VIX9D"], start=start + "T13:31", end=start + "T13:32", ).to_df() vix1d = vix_df[vix_df["symbol"] == "VIX1D"]["open"].iloc[0] vix9d = vix_df[vix_df["symbol"] == "VIX9D"]["open"].iloc[0] # Square the indices to get annualized variances var_1d = vix1d ** 2 var_9d = vix9d ** 2 # Convert to annualized variances for the respective horizons total_var_1d = var_1d * (1 / 365) total_var_9d = var_9d * (9 / 365) # Interpolate days 1 through 5 expected_moves = {} for n in range(1, 6): weight = (n - 1) / (9 - 1) total_var_n = (total_var_1d + weight * (total_var_9d - total_var_1d)) # Convert to expected move and map to expiration expected_moves[pd.to_datetime(start, utc=True) + pd.Timedelta(days=n - 1)] = total_var_n ** 0.5 # Now that we have interpolated the expected moves for each expiry, # we can filter for all strikes within one standard deviation of the spot. # Under normal assumptions, there is a ~68% chance that the spot price will settle within this band. # Fetch the SPX index price from market open to calculate moneyness spx_df = client.timeseries.get_range( dataset="DBIX.CORE", schema="ohlcv-1m", symbols=["SPX"], start=start + "T13:30", end=start + "T13:31", ).to_df() spx = spx_df[spx_df["symbol"] == "SPX"]["open"].iloc[0] # Fetch SPXW options definitions spxw_definitions = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Filter to outright contracts expiring by the end of the week friday_expiry = pd.to_datetime(start, utc=True) + pd.Timedelta(days=4) spxw_definitions = spxw_definitions[spxw_definitions["expiration"] <= friday_expiry] spxw_definitions = spxw_definitions[spxw_definitions["instrument_class"].isin(["P", "C"])] # Drop all unnecessary rows for this process spxw_metadata = spxw_definitions[["instrument_id", "strike_price", "expiration", "instrument_class"]] # Calculate Moneyness per contract (For the sake of this example, we convert to a percentage by multiplying by 100) spxw_metadata["moneyness"] = (spxw_metadata["strike_price"] / spx - 1) * 100 # Fetch SPXW options statistics spxw_statistics = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="statistics", symbols=["SPXW.OPT"], stype_in="parent", start=start, ).to_df() # Include only daily volume statistics spxw_volumes = spxw_statistics[spxw_statistics["stat_type"] == db.StatType.CLEARED_VOLUME][["instrument_id", "quantity"]] spxw_volumes = spxw_volumes.groupby("instrument_id").last().reset_index().rename(columns={"quantity": "volume"}) # Merge the metadata and volume dataframes spxw = pd.merge(spxw_volumes, spxw_metadata, on="instrument_id") # Create a reusable function to plot the volume data for each expiration date def volume_plots(axes, plot_df, expiries, alpha=1): for ax, expiry in zip(axes, expiries): # Filter the dataframe to only include options for the current expiration date expiry_df = plot_df[plot_df["expiration"] == expiry] puts = expiry_df[expiry_df["instrument_class"] == "P"] calls = expiry_df[expiry_df["instrument_class"] == "C"] x_values = np.sort(expiry_df["moneyness"].unique()) bar_width = np.median(np.diff(x_values)) * 0.85 # Bar width will be 85% of the median price difference between strikes # Plot the volume for calls and puts as bars, with calls above the x-axis and puts below the x-axis ax.bar(calls["moneyness"], calls["volume"], width=bar_width, label="Calls", alpha=alpha, color="C0") ax.bar(puts["moneyness"], -puts["volume"], width=bar_width, label="Puts", alpha=alpha, color="C1") # Add a vertical line at moneyness = 0 and a horizontal line at volume = 0 ax.axvline(0, linestyle="--", linewidth=1) ax.axhline(0, linewidth=0.8, color="black") ax.set_title(pd.Timestamp(expiry).strftime("%a %b %d"), loc="left") ax.set_ylabel("Trading Volume") # Filter to only include options that are within 5% of the SPX index price plot_df = spxw[spxw["moneyness"].between(-5, 5)] # Create a list of unique expiration dates for the options expiries = sorted(plot_df["expiration"].unique()) # Create a mask for options that are within the expected move range and those that are outside of it inside_range_mask = abs(plot_df["moneyness"]) < plot_df["expiration"].map(expected_moves) outside_range_mask = abs(plot_df["moneyness"]) > plot_df["expiration"].map(expected_moves) # Use the function defined above to plot the options that are within the expected move range # and those that are outside of it, with different alpha values for each fig, axes = plt.subplots(nrows=len(expiries), ncols=1, figsize=(15, 12), sharex=True) volume_plots(axes, plot_df[inside_range_mask], expiries, alpha=1) axes[0].legend() # Avoid duplicate legends by only adding it to the first subplot volume_plots(axes, plot_df[outside_range_mask], expiries, alpha=0.3) # Add vertical lines to indicate the expected move range for each expiration date for i in range(len(expiries)): axes[i].axvspan(-expected_moves[expiries[i]], expected_moves[expiries[i]], alpha=0.08) axes[i].axvline(-expected_moves[expiries[i]], linestyle=":") axes[i].axvline( expected_moves[expiries[i]], linestyle=":") axis_title = f"{pd.Timestamp(expiries[i]).strftime('%a %b %d')} -- Volatility range: ±{expected_moves[expiries[i]]:.2f}%" axes[i].set_title(axis_title, loc="left") axes[-1].set_xlabel("Moneyness (%)") fig.suptitle("SPXW Volume by Expiration", fontsize=16) plt.show() ``` ![Strikes filtered by volatility](https://databento.com/docs/spx-and-the-vix-0.svg.md) Each shaded band now scales with the expiration's expected volatility rather than by a fixed percentage. Because we have the moneyness mapped to each `instrument_id`, we can easily subscribe to only the contracts we have identified as liquid for each expiration. ## Options | Title | Subtitle | Image | Supported languages | | [Equity options: Introduction](https://databento.com/docs/examples/options/equity-options-introduction.md) | Get started with stock options data. Fetch options chains, strikes, expirations, NBBO. | ![example1](https://databento.com/docs/equity-options-introduction.svg.md) | Python, Options, Historical | | [Options on futures: Introduction](https://databento.com/docs/examples/options/options-on-futures-introduction.md) | Get started with options on futures data. Fetch options chains, strikes, expirations. | ![example2](https://databento.com/docs/options-on-futures-introduction.svg.md) | Python, Options, Historical | | [All options with a given underlying](https://databento.com/docs/examples/options/options-with-underlying.md) | Find all the quarterly, monthly (serial), weekly, and daily options of an underlying. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Options, Historical | | [Join options with underlying prices](https://databento.com/docs/examples/options/options-and-futures.md) | Get front month outright options and join with the latest underlying futures quotes. | ![Image placeholder](https://databento.com/docs/options-and-futures-thumbnail.svg.md) | Python, Options, Historical | | [US equity options volume by venue](https://databento.com/docs/examples/options/options-venues.md) | Compare volume across all venues in the US equity options market using pandas. | ![Image placeholder](https://databento.com/docs/options-venues-thumbnail.svg.md) | Python, Options, Historical | | [Resample US equity options NBBO](https://databento.com/docs/examples/options/nbbo-resampling.md) | Fetch NBBO and downsample to 1-minute intervals for all options in a chain. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Options, Historical | | [Estimate implied volatility](https://databento.com/docs/examples/options/estimating-implied-volatility.md) | Compute and graph implied volatility from options prices. | ![example6](https://databento.com/docs/estimating-implied-volatility-thumbnail.svg.md) | Python, Options, Historical | | [Get symbols for 0DTE options](https://databento.com/docs/examples/options/zero-dte-options.md) | Get a list of 0DTE options for a product. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Options, Historical, Live | | [Daily statistics for equity options](https://databento.com/docs/examples/options/equity-open-interest.md) | Calculate volume and open interest for an equity options chain. | ![Image placeholder](https://databento.com/docs/equity-open-interest-thumbnail.svg.md) | Python, Options, Historical | | [Get end of day option spreads](https://databento.com/docs/examples/options/option-spreads.md) | Calculate end of day options spread by expirations. | ![Image placeholder](https://databento.com/docs/option-spreads-thumbnail.svg.md) | Python, Options, Historical | # Equity options: Introduction > **Info** > > This article introduces additional concepts to get started with equity options data > on Databento. If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) > guide. ## Overview In this example, we'll introduce: - The OPRA dataset - Using instrument definitions to get symbols, strike prices, and expirations - Options Clearing Corporation (OCC) symbology for raw symbols - Using parent instrument symbology to fetch an option chain - Stream live options trades We'll also highlight any special conventions of equity options datasets on Databento that differ from those of other asset classes. ## Finding an equity options dataset To use equity options data on Databento, first identify the dataset that you want from our [data catalog]($PLATFORM_HOSTNAME) and go to its detail page. Here, you can find its dataset ID at the top left of the page. ![Dataset ID](https://databento.com/docs/equity-options-dataset-id.png.md) For this example, we'll use the [OPRA]($PLATFORM_HOSTNAME/datasets/OPRA.PILLAR) dataset, whose dataset ID is `OPRA.PILLAR`. You'll need to pass this in as the `dataset` parameter of any API or client method. ## OPRA OPRA provides consolidated last sale and national BBO across all US equity options exchanges. Various types of options are covered, including: - Single name stock options (e.g. AAPL, TSLA) - ETF options (e.g. SPY, QQQ) - Index options (e.g. SPX/SPXW, VIX) > **Info** > > The "Pillar" suffix in the `OPRA.PILLAR` dataset ID is a reference to the SIAC Pillar > SIP, the latest infrastructure behind the binary OPRA feed since 2021. ## Using instrument definitions to get symbols, strike prices, and expirations A full list of instrument definitions can be fetched with the `definition` schema and passing in `symbols="ALL_SYMBOLS"` to the [timeseries.get\_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md?lang=python) method. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="definition", symbols="ALL_SYMBOLS", start="2024-08-09", limit=10000, ) df = data.to_df() print(df[["raw_symbol", "strike_price", "expiration"]]) ``` ```plaintext raw_symbol strike_price expiration ts_recv 2024-08-09 10:30:00.446956866+00:00 BKNG 250117P01770000 1770.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.446956866+00:00 BKNG NaN NaT 2024-08-09 10:30:00.446972667+00:00 BLK 250117C00430000 430.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.446972667+00:00 BLK NaN NaT 2024-08-09 10:30:00.448575941+00:00 UNH 250117P00640000 640.0 2025-01-17 00:00:00+00:00 ... ... ... ... 2024-08-09 10:30:00.586069426+00:00 SPY 240814P00561000 561.0 2024-08-14 00:00:00+00:00 2024-08-09 10:30:00.586076435+00:00 SMCI 250117C01080000 1080.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.586081478+00:00 SMCI 250117C01440000 1440.0 2025-01-17 00:00:00+00:00 2024-08-09 10:30:00.586087572+00:00 QQQ 250331C00415000 415.0 2025-03-31 00:00:00+00:00 2024-08-09 10:30:00.586107573+00:00 SHY 250117P00085000 85.0 2025-01-17 00:00:00+00:00 [10000 rows x 3 columns] ``` Note that we used a timeseries method here and the results are indexed in time. These are called **point-in-time instrument definitions**. Many data providers treat instrument listings and definitions as a static table, so this may initially feel unintuitive if you're new to this approach. However, for options, this is important as new instruments are added intraday as the underlying moves, and knowing the strike prices of these instruments may result in lookahead effects. ## OCC symbology Databento's raw symbols for equity options are based on OCC symbology, also known as Options Symbology Initiative (OSI) symbology. The [OPRA dataset specification](https://databento.com/docs/datasets/opra-pillar.md#symbology) provides an overview of this symbology format. You can specify the symbology type to use for your input symbols with the `stype_in` parameter. To demonstrate this, let's fetch the raw symbol `SPY 241115P00525000` with `stype_in="raw_symbol"`: the SPY 525.00 put expiring on 2024-11-15. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="cmbp-1", stype_in="raw_symbol", symbols=["SPY 241115P00525000"], start="2024-08-09T09:30-04:00", end="2024-08-09T10:00-04:00", ) df = data.to_df() print(df[["symbol", "bid_px_00", "ask_px_00"]].head()) ``` ```plaintext symbol bid_px_00 ask_px_00 ts_recv 2024-08-09 13:30:00.025137089+00:00 SPY 241115P00525000 16.33 16.53 2024-08-09 13:30:00.167110712+00:00 SPY 241115P00525000 16.33 16.52 2024-08-09 13:30:00.182316856+00:00 SPY 241115P00525000 16.34 16.51 2024-08-09 13:30:00.238455572+00:00 SPY 241115P00525000 16.34 16.50 2024-08-09 13:30:00.340439305+00:00 SPY 241115P00525000 16.34 16.51 ``` ## Using parent symbology to fetch an option chain Working with individual raw symbols can be tedious. Many users find it easier to specify the underlying's symbol and fetch all option symbols for that underlying in an **option chain** at once. This can be done using our [parent symbology](https://databento.com/docs/conventions/symbology.md#parent) by passing in `stype_in="parent"` to your request. For example, we can print all tick-by-tick trades of options on AAPL for a given time range as follows: ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="trades", stype_in="parent", symbols=["AAPL.OPT"], start="2024-08-09T09:30-04:00", end="2024-08-09T10:00-04:00", ) df = data.to_df() print(df[["symbol", "price"]]) ``` ```plaintext symbol price ts_recv 2024-08-09 13:30:00.008689345+00:00 AAPL 240809C00217500 0.06 2024-08-09 13:30:00.021880156+00:00 AAPL 240816C00230000 0.10 2024-08-09 13:30:00.032886565+00:00 AAPL 240920P00175000 0.77 2024-08-09 13:30:00.036991612+00:00 AAPL 240809P00202500 0.08 2024-08-09 13:30:00.109699333+00:00 AAPL 240816C00215000 2.23 ... ... ... 2024-08-09 13:59:58.801315993+00:00 AAPL 240816C00207500 6.80 2024-08-09 13:59:58.962219427+00:00 AAPL 240809C00212500 1.24 2024-08-09 13:59:59.036621416+00:00 AAPL 240816C00240000 0.02 2024-08-09 13:59:59.124512360+00:00 AAPL 240809C00212500 1.24 2024-08-09 13:59:59.858997181+00:00 AAPL 240809C00220000 0.02 [22451 rows x 2 columns] ``` ## Streaming live equity options trades For this example, we will stream the [TCBBO schema](https://databento.com/docs/schemas/tbbo.md#consolidated-bbo-on-trade-tcbbo) for all AAPL options. This schema shows every trade event along with the NBBO immediately *before* the effect of each trade. Over 99.9% of events on the OPRA feed are CMBP-1 updates. It's significantly easier to work with another schema like [TCBBO](https://databento.com/docs/schemas/tbbo.md#consolidated-bbo-on-trade-tcbbo) or [trades](https://databento.com/docs/schemas/trades.md). While the OPRA feed does not disseminate the trade aggressor side for a trade (`side` will be always be `N`), you can use the NBBO before the trade executes to help make a determination on whether the trade was initiated by the buyer or seller. > **Info** > > This example requires a live license to `OPRA.PILLAR`. Visit our [live data portal]($PLATFORM_HOSTNAME/portal/live-data#manage) > to sign up. ```python import databento as db # Enable some basic logging db.enable_logging("INFO") # Create a live client and connect live_client = db.Live("$YOUR_API_KEY", compression=db.Compression.ZSTD) # Subscribe to the TCBBO schema for all active AAPL options live_client.subscribe( dataset="OPRA.PILLAR", schema="tcbbo", symbols="AAPL.OPT", stype_in="parent", ) # We add a print callback to view each record live_client.add_callback(print) # Start the live client to begin data streaming live_client.start() # Run the stream for 5 seconds before closing live_client.block_for_close(timeout=5) ``` > **Info** > > If you do not see any output, it could be because the markets are closed. See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) for using [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=16777235, ts_event=1769047399503117383, pretty_ts_event='2026-01-22T02:03:19.503117383Z', stype_in=None, stype_in_symbol='AAPL.OPT', stype_out=None, stype_out_symbol='AAPL 270115P00380000', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') ... Cmbp1Msg(ts_recv=1769005800009884453, pretty_ts_recv='2026-01-21T14:30:00.009884453Z', rtype=, publisher_id=OPRA.PILLAR.XPHL (35), instrument_id=16787902, ts_event=1769005800009683740, pretty_ts_event='2026-01-21T14:30:00.009683740Z', action='T', side='N', price=260000000, pretty_price=0.26, size=7, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=260000000, pretty_bid_px_00=0.26, ask_px_00=300000000, pretty_ask_px_00=0.3, bid_sz_00=13, ask_sz_00=3, bid_pb_00=35, ask_pb_00=35) Cmbp1Msg(ts_recv=1769005800010490784, pretty_ts_recv='2026-01-21T14:30:00.010490784Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16787347, ts_event=1769005800010290179, pretty_ts_event='2026-01-21T14:30:00.010290179Z', action='T', side='N', price=4950000000, pretty_price=4.95, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=9223372036854775807, pretty_bid_px_00=NaN, ask_px_00=9223372036854775807, pretty_ask_px_00=NaN, bid_sz_00=0, ask_sz_00=0, bid_pb_00=0, ask_pb_00=0) Cmbp1Msg(ts_recv=1769005800023091148, pretty_ts_recv='2026-01-21T14:30:00.023091148Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16787347, ts_event=1769005800022890199, pretty_ts_event='2026-01-21T14:30:00.022890199Z', action='T', side='N', price=5000000000, pretty_price=5, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=4900000000, pretty_bid_px_00=4.9, ask_px_00=5000000000, pretty_ask_px_00=5, bid_sz_00=2, ask_sz_00=1, bid_pb_00=26, ask_pb_00=26) ... Cmbp1Msg(ts_recv=1769005800202002589, pretty_ts_recv='2026-01-21T14:30:00.202002589Z', rtype=, publisher_id=OPRA.PILLAR.XBXO (33), instrument_id=16794630, ts_event=1769005800201802008, pretty_ts_event='2026-01-21T14:30:00.201802008Z', action='T', side='N', price=1920000000, pretty_price=1.92, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=1920000000, pretty_bid_px_00=1.92, ask_px_00=2030000000, pretty_ask_px_00=2.03, bid_sz_00=6, ask_sz_00=6, bid_pb_00=27, ask_pb_00=27) Cmbp1Msg(ts_recv=1769005800215279430, pretty_ts_recv='2026-01-21T14:30:00.215279430Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16790992, ts_event=1769005800215078713, pretty_ts_event='2026-01-21T14:30:00.215078713Z', action='T', side='N', price=410000000, pretty_price=0.41, size=3, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=350000000, pretty_bid_px_00=0.35, ask_px_00=420000000, pretty_ask_px_00=0.42, bid_sz_00=3, ask_sz_00=6, bid_pb_00=37, ask_pb_00=37) Cmbp1Msg(ts_recv=1769005800215680274, pretty_ts_recv='2026-01-21T14:30:00.215680274Z', rtype=, publisher_id=OPRA.PILLAR.XISX (26), instrument_id=16780046, ts_event=1769005800215479700, pretty_ts_event='2026-01-21T14:30:00.215479700Z', action='T', side='N', price=480000000, pretty_price=0.48, size=6, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=460000000, pretty_bid_px_00=0.46, ask_px_00=910000000, pretty_ask_px_00=0.91, bid_sz_00=20, ask_sz_00=1, bid_pb_00=25, ask_pb_00=25) ``` ## Special conventions for equity options on Databento - **Empty levels and null prices.** Many option contracts are illiquid and may have an empty book, especially at the start of day. Empty levels will have null prices and zero depth. We represent null prices with type max, e.g. 2^63 - 1, as a sentinel value. Our client libraries will generally replace these sentinel values with a more idiomatic null representation--for example, if you pass `price_type="float"` (default) in to the [DBNStore.to\_df](https://databento.com/docs/api-historical/helpers/dbn-store-to-df.md?lang=python) method of the Python client, these sentinel values will be replaced with `NaN` in the resulting DataFrame. - **Bandwidth.** Due to the significant size of OPRA data, your live feed may get backed up if your client is not reading fast enough or network bandwidth is limited--this is a result of TCP flow control. Hence, we recommend setting up [dedicated connectivity](https://databento.com/docs/architecture/dedicated-connectivity-guide.md) if you intend to consume a large number of symbols or the entire dataset in real-time. - **Intraday listings.** Our instrument definitions are timestamped on both historical and live data. This point-in-time treatment is especially useful if you intend to subscribe to and trade new strikes as they get listed; to backtest this, and to avoid lookahead effects of knowing the underlying's future price changes. ## Numeric publisher IDs The source of each trade and quote is identified by a numeric publisher ID. > **See also** > > [US equity options volume by venue](https://databento.com/docs/examples/options/options-venues.md) to see how to > identify exchange from publisher ID. # Options on futures: Introduction > **Info** > > This article builds off [Futures: Introduction](https://databento.com/docs/examples/futures/futures-introduction.md) and > introduces additional concepts to get started with options on futures. > If you're new to Databento, see the [Quickstart](https://databento.com/docs/quickstart.md) guide. ## Overview In this example, we'll demonstrate: - Using instrument definitions to get symbols, strike prices, and expirations. - Using parent instrument symbology to fetch an option chain. ## Finding an options on futures dataset Several futures exchanges also provide options on futures. You can follow the [Futures: Introduction](https://databento.com/docs/examples/futures/futures-introduction.md) guide to get the dataset ID of such an exchange. For this example, we'll use the [CME Globex MDP 3.0]($PLATFORM_HOSTNAME/datasets/GLBX.MDP3) dataset, whose dataset ID is `GLBX.MDP3`. ## Using instrument definitions to get symbols, strike prices, and expirations A full list of instrument definitions can be fetched with the `definition` schema and passing in `symbols='ALL_SYMBOLS'` to the [timeseries.get\_range](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md?lang=python) method. ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="definition", symbols="ALL_SYMBOLS", start="2023-08-25", limit=10000, ) # Convert to DataFrame df = data.to_df() print(df[["raw_symbol", "strike_price", "expiration"]]) ``` ```plaintext raw_symbol strike_price expiration ts_recv 2023-08-25 00:00:00+00:00 OZSH4 C1520 1520.00 2024-02-23 19:20:00+00:00 2023-08-25 00:00:00+00:00 MEOF4-MEOK4 NaN 2024-01-31 19:30:00+00:00 2023-08-25 00:00:00+00:00 LOX4 P250 2.50 2024-10-17 18:30:00+00:00 2023-08-25 00:00:00+00:00 AHMG4 NaN 2024-02-01 04:59:00+00:00 2023-08-25 00:00:00+00:00 OBK4 C29300 2.93 2024-04-25 18:30:00+00:00 ... ... ... ... 2023-08-25 00:00:00+00:00 L0BZ3 C-0070 -7.00 2023-12-01 19:00:00+00:00 2023-08-25 00:00:00+00:00 OZNV3 C1135 113.50 2023-09-22 21:00:00+00:00 2023-08-25 00:00:00+00:00 R2AU3 P1970 1970.00 2023-09-11 20:00:00+00:00 2023-08-25 00:00:00+00:00 G1MV3 P2100 2100.00 2023-10-02 17:30:00+00:00 2023-08-25 00:00:00+00:00 CHIG5-WDCG5 NaN 2025-02-24 20:00:00+00:00 [10000 rows x 3 columns] ``` Note that we used a timeseries method here and the results are indexed in time. These are called **point-in-time instrument definitions**. Many data providers treat instrument listings and definitions as a static table, so this may initially feel unintuitive if you're new to this approach. However, for options, this is important as new instruments are added intraday as the underlying moves, and knowing the strike prices of these instruments may result in lookahead effects. ## Using parent symbology to fetch an option chain Working with individual raw symbols can be tedious. Many users find it easier to specify the underlying's symbol and fetch all option symbols for that underlying in an **option chain** at once. This can be done using our [parent symbology](https://databento.com/docs/conventions/symbology.md#parent) by passing in `stype_in="parent"` to your request. For example, we can print all tick-by-tick trades of quarterly options on ES for a given time range as follows: ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request trades data data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols=["ES.OPT"], start="2023-08-25T12:00", end="2023-08-25T13:00", ) # Convert to DataFrame df = data.to_df() print(df[["symbol", "price"]].tail(10)) ``` ```plaintext symbol price ts_recv 2023-08-25 12:58:44.446885515+00:00 ESU3 P4390 53.25 2023-08-25 12:58:44.447489339+00:00 ESU3 P4410 61.50 2023-08-25 12:58:58.034159513+00:00 ESU3 C4460 30.50 2023-08-25 12:58:58.052849806+00:00 ESU3 P4390 53.25 2023-08-25 12:59:16.789312686+00:00 ESU3 C4500 17.25 2023-08-25 12:59:27.661545003+00:00 ESU3 C4420 49.00 2023-08-25 12:59:28.337200208+00:00 ESU3 P4410 61.50 2023-08-25 12:59:30.609532264+00:00 ESU3 C4460 30.50 2023-08-25 12:59:50.095229136+00:00 ESU3 C4440 39.25 2023-08-25 12:59:50.141519076+00:00 ESU3 P4380 49.25 ``` ## Special conventions for options on futures on Databento Most of the [conventions for futures](https://databento.com/docs/examples/futures/futures-introduction.md#special-conventions-for-futures-on-databento) will also apply to options on futures. - **Empty levels and null prices.** Many option contracts are illiquid and may have an empty book, especially at the start of day. Empty levels will have null prices and zero depth. We represent null prices with type max, e.g. 2^63 - 1, as a sentinel value. Our client libraries will generally replace these sentinel values with a more idiomatic null representation--for example, if you pass `price_type="float"` (default) in to the [DBNStore.to\_df](https://databento.com/docs/api-historical/helpers/dbn-store-to-df.md?lang=python) method of the Python client, these sentinel values will be replaced with `NaN` in the resulting DataFrame. - **Intraday listings.** Our instrument definitions are timestamped on both historical and live data. This point-in-time treatment is especially useful if you intend to subscribe to and trade new strikes as they get listed; to backtest this, and to avoid lookahead effects of knowing the underlying's future price changes. > **See also** > > [All options with a given underlying](https://databento.com/docs/examples/options/options-with-underlying.md). # All options with a given underlying ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve instrument definitions for all options on futures with a given underlying. This is useful for complex assets such as the options on [E-mini S&P 500 Futures](https://www.cmegroup.com/markets/equities/sp/e-mini-sandp500.html) which have different symbols for their quarterly, monthly (serial), weekly, and daily options. We'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties such as the security type and underlying of the instrument. We'll request the definitions for all symbols, and then filter the instruments using the `security_type` and `underlying` fields. We'll show the `asset` field, which refers to the parent symbol for that option, as well as `strike price` and `expiration` . > **Info** > > See the [Options on E-Mini S&P 500 Futures](https://www.cmegroup.com/trading/equity-index/weekly-eom-options-faq.html) > FAQ page for more information on the ES Options symbology. ## Example ```python import databento as db # Set underlying instrument symbol = "ESH5" # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Get definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="definition", symbols="ALL_SYMBOLS", start="2025-03-10", ) # Convert to DataFrame df = data.to_df() # Filter for instruments with a security type of "OOF" (options-on-futures) df = df[df["security_type"] == "OOF"] # Select only options for the specified underlying df = df[df["underlying"] == symbol] # Sort by strike price and select specific columns df = df.sort_values("strike_price") df = df[["raw_symbol", "underlying", "asset", "instrument_class", "strike_price", "expiration"]] print(f"{len(df):,d}", f"option(s) for {symbol}") print(df) ``` ```plaintext 6,494 option(s) for ESH5 ``` > **See also** > > Check out the [0DTE symbols](https://databento.com/docs/examples/options/zero-dte-options.md) example on how to filter these > options based on `expiration`. ```plaintext raw_symbol underlying asset instrument_class strike_price expiration ts_recv 2025-03-10 00:00:00+00:00 E2AH5 P100 ESH5 E2A P 100.0 2025-03-10 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 C100 ESH5 E3D C 100.0 2025-03-20 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3AH5 C100 ESH5 E3A C 100.0 2025-03-17 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 P100 ESH5 E3D P 100.0 2025-03-20 20:00:00+00:00 2025-03-10 00:00:00+00:00 E2DH5 P100 ESH5 E2D P 100.0 2025-03-13 20:00:00+00:00 ... ... ... ... ... ... ... 2025-03-10 00:00:00+00:00 ESH5 P8500 ESH5 ES P 8500.0 2025-03-21 13:30:00+00:00 2025-03-10 00:00:00+00:00 ESH5 P8600 ESH5 ES P 8600.0 2025-03-21 13:30:00+00:00 2025-03-10 00:00:00+00:00 ESH5 C8600 ESH5 ES C 8600.0 2025-03-21 13:30:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 C10100 ESH5 E3D C 10100.0 2025-03-20 20:00:00+00:00 2025-03-10 00:00:00+00:00 E3DH5 P10100 ESH5 E3D P 10100.0 2025-03-20 20:00:00+00:00 [6494 rows x 6 columns] ``` # Join options with underlying prices > **Info** > > This example builds off the [Join schemas on instrument ID](https://databento.com/docs/examples/basics-historical/joining-schemas.md) example. ## Overview In this example, we'll use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to get data for futures and options. First, we'll request [trades schema](https://databento.com/docs/schemas/trades.md) data for all options on the volume-based continuous contract and join them with the most recent quote from the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) for the underlying future. We'll also request [BBO-1m schema](https://databento.com/docs/schemas/bbo.md) data for the underlying future and a single option and plot the midprice for a full session. ## Example ```python import databento as db import matplotlib.dates as mdates import matplotlib.pyplot as plt import pandas as pd def get_bbo_data( dataset: str, instrument_id: int, start: str, end: str, ) -> pd.DataFrame: mbp_df = client.timeseries.get_range( dataset=dataset, schema="bbo-1m", symbols=instrument_id, stype_in="instrument_id", start=start, end=end, ).to_df(map_symbols=False) return mbp_df[["instrument_id", "bid_px_00", "ask_px_00"]] def plot_option_with_underlying( opt_df: pd.DataFrame, fut_df: pd.DataFrame, ) -> None: opt_df["mid_price"] = (opt_df["bid_px_00"] + opt_df["ask_px_00"]) / 2 fut_df["mid_price"] = (fut_df["bid_px_00"] + fut_df["ask_px_00"]) / 2 full_index = pd.date_range( start=start, end=end, freq="1min", tz="UTC", inclusive="right", ) opt_df = opt_df.reindex(full_index).ffill().astype(opt_df.dtypes) fut_df = fut_df.reindex(full_index).ffill().astype(fut_df.dtypes) opt_symbol = opt_df["raw_symbol"].iloc[0] fut_symbol = opt_df["underlying"].iloc[0] fig, ax1 = plt.subplots(figsize=(14, 6)) ax1.plot(fut_df.index, fut_df["mid_price"], color="C0", label=fut_symbol) ax1.set_ylabel(f"Futures midprice ({fut_symbol})", color="C0") ax1.tick_params(axis="y", labelcolor="C0") strike = opt_df["strike_price"].iloc[0] ax1.axhline(y=strike, color="gray", linestyle="--") ax1.text( x=fut_df.index[0], y=strike, s=f"Strike price: {strike:.0f}", color="gray", va="bottom", ha="left", ) ax2 = ax1.twinx() ax2.plot(opt_df.index, opt_df["mid_price"], color="C1", label=opt_symbol) ax2.set_ylabel(f"Options midprice ({opt_symbol})", color="C1") ax2.tick_params(axis="y", labelcolor="C1") ax1.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M")) ax1.xaxis.set_major_locator(mdates.HourLocator(interval=1)) ax1.set_title("Futures vs. Options Midprice") ax1.set_xlabel("Time (UTC)") lines1, labels1 = ax1.get_legend_handles_labels() lines2, labels2 = ax2.get_legend_handles_labels() ax1.legend(lines1 + lines2, labels1 + labels2, loc="upper left") ax1.grid(True) ax2.grid(False) fig.tight_layout() plt.show() # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" symbol = "CL.v.0" start = "2026-04-12T22:00:00" end = "2026-04-13T21:00:00" # Get symbol mapping for the continuous contract symbol_map = client.symbology.resolve( dataset=dataset, symbols=symbol, stype_in="continuous", stype_out="instrument_id", start_date="2026-04-12", ) # Get instrument ID for the resolved futures contract fut_id = int(symbol_map["result"][symbol][0]["s"]) # Get all option definitions def_df = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="definition", start="2026-04-12", ).to_df() # Filter for options with the continuous contract as an underlying opt_def_df = def_df[ (def_df["user_defined_instrument"] == db.UserDefinedInstrument.NO) & (def_df["instrument_class"].isin((db.InstrumentClass.CALL, db.InstrumentClass.PUT))) & (def_df["underlying_id"] == fut_id) ] opt_def_df = opt_def_df[ [ "ts_event", "expiration", "raw_symbol", "instrument_id", "asset", "strike_price", "underlying", "underlying_id", "instrument_class", ] ] # Get trades data for the filtered options opt_trades_df = ( client.timeseries.get_range( dataset=dataset, schema="trades", symbols=[f"{x}.OPT" for x in opt_def_df["asset"].unique()], stype_in="parent", start=start, end=end, ) .to_df() .rename(columns={"size": "trade_size", "price": "trade_price", "side": "trade_side"}) ) # Join options with their definitions opt_df = ( opt_trades_df.reset_index() .merge( opt_def_df.reset_index(), on="instrument_id", how="inner", suffixes=("", "_def"), ) .set_index("ts_recv") .sort_index() ) fut_mbp_df = client.timeseries.get_range( dataset=dataset, schema="mbp-1", symbols=fut_id, stype_in="instrument_id", start=start, end=end, ).to_df() fut_mbp_df = fut_mbp_df.rename( columns={ "bid_px_00": "underlying_bid", "ask_px_00": "underlying_ask", }, )[["underlying_bid", "underlying_ask"]] # Join most recent underlying bid/ask with options trades df = pd.merge_asof( opt_df, fut_mbp_df, left_index=True, right_index=True, direction="backward", ) print( df[ [ "ts_event", "instrument_id", "raw_symbol", "trade_price", "trade_size", "trade_side", "strike_price", "instrument_class", "expiration", "underlying_bid", "underlying_ask", ] ], ) # 42963414 is the `instrument_id` for NL2J6 P10000 opt_id = 42963414 opt_bbo_df = get_bbo_data(dataset, opt_id, start, end) opt_bbo_df = ( opt_bbo_df.reset_index().merge(opt_def_df, on="instrument_id", how="left").set_index("ts_recv") ) fut_bbo_df = get_bbo_data(dataset, fut_id, start, end) plot_option_with_underlying(opt_bbo_df, fut_bbo_df) ``` ## Result We can see all trades for options that have the volume-based continuous contract as an underlying. ```plaintext ts_event instrument_id raw_symbol trade_price trade_size ... strike_price instrument_class expiration underlying_bid underlying_ask ts_recv ... 2026-04-12 22:00:00.356333832+00:00 2026-04-12 22:00:00+00:00 42157669 LOK6 C9900 6.01 1 ... 99.00 C 2026-04-16 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.356370138+00:00 2026-04-12 22:00:00+00:00 42180553 LOK6 P8200 0.39 1 ... 82.00 P 2026-04-16 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.357402871+00:00 2026-04-12 22:00:00+00:00 42511805 ML2J6 C10000 2.40 2 ... 100.00 C 2026-04-13 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.357526014+00:00 2026-04-12 22:00:00+00:00 42639465 ML2J6 P9100 0.74 1 ... 91.00 P 2026-04-13 18:30:00+00:00 NaN NaN 2026-04-12 22:00:00.357609934+00:00 2026-04-12 22:00:00+00:00 42963412 NL2J6 P9600 0.99 1 ... 96.00 P 2026-04-14 18:30:00+00:00 NaN NaN ... ... ... ... ... ... ... ... ... ... ... ... 2026-04-13 20:57:18.223377118+00:00 2026-04-13 20:57:18.222967937+00:00 42308894 LOK6 P8500 0.27 1 ... 85.00 P 2026-04-16 18:30:00+00:00 97.96 97.98 2026-04-13 20:57:18.223891862+00:00 2026-04-13 20:57:18.223423419+00:00 42308894 LOK6 P8500 0.27 1 ... 85.00 P 2026-04-16 18:30:00+00:00 97.96 97.98 2026-04-13 20:57:19.267462126+00:00 2026-04-13 20:57:19.267057293+00:00 42631933 WL3J6 C9775 3.03 1 ... 97.75 C 2026-04-15 18:30:00+00:00 97.97 97.99 2026-04-13 20:58:04.991697756+00:00 2026-04-13 20:58:04.983728493+00:00 42958534 NL2J6 C10100 0.91 1 ... 101.00 C 2026-04-14 18:30:00+00:00 97.94 97.96 2026-04-13 20:58:38.292464827+00:00 2026-04-13 20:58:38.292053561+00:00 379004 LOK6 P7000 0.02 1 ... 70.00 P 2026-04-16 18:30:00+00:00 97.97 97.99 [17262 rows x 11 columns] ``` Now we'll plot the midprice of the underlying future and a single option for the duration of a full session. In this example, the option `NL2J6 P10000` expires on 2026-04-14. ![Futures and options midprice](https://databento.com/docs/options-and-futures-0.svg.md) # US equity options volume by venue ## Overview This example will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve data from the entire US equity options market and compare volume across execution venues. We'll use the [OHLCV-1m](https://databento.com/docs/schemas/ohlcv.md) schema and look at data over a 30-minute window. This example could be modified to look at the entire session by changing the `start` and `end` variables, or by using the OHLCV-1d schema. The OPRA datafeed provides consolidated options data for the 20 US equity options venues. The `publisher_id` field indicates which venue the record is associated with. Using this information, we will create a table and plot similar to the data found in [Cboe's Market Volume Summary](https://www.cboe.com/us/options/market_statistics/) report. > **See also** > > For a full list of the OPRA venues, see the [OPRA documentation](https://databento.com/docs/datasets/opra-pillar.md#venues). ## Example ```python import databento as db import matplotlib.pyplot as plt from matplotlib.patches import Patch # Set parameters dataset = "OPRA.PILLAR" start = "2025-03-26T14:00" end = "2025-03-26T14:30" # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Request OHLCV-1m data for ALL_SYMBOLS and convert to DataFrame df = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1m", symbols="ALL_SYMBOLS", start=start, end=end, ).to_df() # Aggregate volume by venue df = df.groupby("publisher_id", as_index=False)["volume"].sum() df["percentage"] = df["volume"] / df["volume"].sum() * 100 # Rename publisher_id and columns for readability publishers = client.metadata.list_publishers() venue_name_map = { x["publisher_id"]: db.Publisher.from_int(x["publisher_id"]).venue.description for x in publishers } df["publisher_id"] = df["publisher_id"].map(venue_name_map) df = df.rename(columns={"publisher_id": "Venue", "volume": "Volume", "percentage": "% of Total"}) # Calculate volume by group and sort df["Group"] = df["Venue"].str.split(" ").str[0] df["Group Volume"] = df.groupby("Group")["Volume"].transform("sum") df = df.sort_values(by=["Group Volume", "Volume"], ascending=[False, False]) df = df[["Group", "Venue", "Volume", "% of Total"]] print(df.set_index(["Group", "Venue"]).to_string(float_format="%0.2f")) # Now, let's plot this data fig, ax = plt.subplots(figsize=(14, 6)) group_handles = [] # Legend handles for group_idx, group in enumerate(df["Group"].unique()): group_data = df[df["Group"] == group] # Create handle for plot legend group_pct = group_data["% of Total"].sum() group_handles.append(Patch(facecolor=f"C{group_idx}", label=f"{group}: {group_pct:.2f}%")) last_venue_pct = 0 for i, (_, venue_data) in enumerate(group_data.iterrows()): venue_pct = venue_data["% of Total"] # Used as height for each venue section # Add section for venue bar = ax.bar( x=group, height=venue_pct, bottom=last_venue_pct, color=f"C{group_idx}", alpha=1.0 - i * 0.15, ) # Add the venue label ax.text( x=group_idx, y=last_venue_pct + venue_pct / 2, s=" ".join(venue_data["Venue"].split(" ")[:2]), ha="center", va="center", ) last_venue_pct += venue_pct ax.legend(handles=group_handles) ax.set_title("U.S. Options Market Volume") ax.set_ylabel("% of Total Volume") plt.tight_layout() plt.show() ``` ```plaintext Volume % of Total Group Venue Cboe Cboe Options 954511 14.47 Cboe EDGX Options 381901 5.79 Cboe BZX Options 261275 3.96 Cboe C2 Options 152628 2.31 Nasdaq Nasdaq PHLX 435190 6.60 Nasdaq ISE 381951 5.79 Nasdaq Options 327166 4.96 Nasdaq GEMX 272005 4.12 Nasdaq MRX 182480 2.77 Nasdaq Texas Options 105090 1.59 NYSE NYSE American Options 865360 13.12 NYSE Arca Options 664865 10.08 MIAX MIAX Options 447819 6.79 MIAX Emerald 230339 3.49 MIAX Pearl 201093 3.05 MIAX Sapphire 124162 1.88 BOX BOX Options 386105 5.85 MEMX MEMX Options 222973 3.38 ``` ![Options venues volume](https://databento.com/docs/options-venues-0.svg.md) # Resample US equity options NBBO ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve the regional and consolidated best bid and offer (BBO) for the US equities market from the OPRA dataset. From this, we'll filter out the NBBO and then downsample at a 1-minute interval. To do this we will use pandas, specifically the `DataFrame.groupby` method. Note that sometimes the BBO is referred to as a *quote*, and *ask* and *offer* are synonyms referring to the sell price. ## MBP-1 schema We'll use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) in this example, which contains the price and size for both the bid and the ask. For datasets such as OPRA that cover multiple execution venues (exchanges), the `publisher_id` field indicates which venue the record is associated with. There will be records for each individual venue to indicate the *regional* BBO, and then an additional record to represent the *consolidated* NBBO. For the OPRA dataset, which is the SIP for US equity options, the consolidated `publisher_id` is `30` (venue `OPRA`). Trades, which are included in the MBP-1 schema as records with `action=T`, also use `publisher_id` to indicate which venue they occurred on. > **See also** > > [API reference - Historical - metadata.list\_publishers](https://databento.com/docs/api-historical/metadata/metadata-list-publishers.md) ## Example ```python import databento as db import pandas as pd # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, request MBP-1 for the SPIKE options chain (a volatility index) df = client.timeseries.get_range( dataset="OPRA.PILLAR", start="2023-07-03", schema="mbp-1", stype_in="parent", symbols="SPIKE.OPT", limit=20_000, ).to_df() # Only include NBBO records (`publisher_id == 30`) df = df[df["publisher_id"] == db.Publisher.OPRA_PILLAR_OPRA.to_int()] # Resample the data at a fixed one minute interval, per symbol df_1m = df.groupby([pd.Grouper(freq="1T"), "symbol"]).first() # Print the NBBO print(df_1m[["bid_px_00", "ask_px_00"]]) ``` ## Result ```plaintext bid_px_00 ask_px_00 ts_recv symbol 2023-07-03 13:30:00+00:00 SPIKE 230719C00010000 5.25 5.40 SPIKE 230719C00010500 4.75 4.90 SPIKE 230719C00011000 4.25 4.40 SPIKE 230719C00011500 3.75 3.90 SPIKE 230719C00012000 3.25 3.40 ... ... ... 2023-07-03 13:34:00+00:00 SPIKE 231018C00047500 0.16 0.70 SPIKE 231018C00050000 0.12 0.66 SPIKE 231018C00055000 0.06 0.59 SPIKE 231018P00014000 0.15 0.69 SPIKE 231115C00050000 0.20 0.75 [1657 rows x 2 columns] ``` # Estimate implied volatility ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to process instrument definition and MBP-1 data to graph implied volatility by strike price for the front-month [E-mini S&P 500 Futures (ES)](https://www.cmegroup.com/markets/equities/sp/e-mini-sandp500.html) contract. To do this we will request top-of-book data for the front-month ES futures contract using our [continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous) and the entire ES options chain using our [parent symbology](https://databento.com/docs/conventions/symbology.md#parent). We will then filter the options by their underlying, to only consider options for the front-month contract. Finally, we will extract prices for these instruments and use the [Black-76 model](https://en.wikipedia.org/wiki/Black_model) to graph implied volatility by strike price for the nearest quarterly E-mini S&P 500 (ES) options expiration. ## Implied volatility Implied Volatility (IV) is a key concept in the field of options trading and financial markets. It represents the market's forecast of a likely movement in a security's price. Rather than reflecting historical price changes, implied volatility looks forward into the market's expectations and is often used as an indicator of market sentiment for options trading. Naturally, a higher implied volatility indicates larger price movements are expected. ## Black-76 model The most famous model for calculating implied volatility is the [Black-Scholes model](https://en.wikipedia.org/wiki/Black%E2%80%93Scholes_model). This model is primarily used for pricing equities options. In this example, we will be calculating the implied volatility using options on futures, so we will use an alternative model called the [Black-76 model](https://en.wikipedia.org/wiki/Black_model). This model is more appropriate for options on futures, bonds, and other interest rate products. Typically, the implied volatility is assumed and is used to calculate the price of an option. In this exercise we will invert this operation, and use historical prices to solve for implied volatility, usually denoted as σ (sigma). We cannot solve for this analytically, so we will use the [scipy](https://pypi.org/project/scipy/) module to find sigma numerically using [scipy.optimize.root\_scalar](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.root_scalar.html). ## Dependencies This example will use the [scipy](https://pypi.org/project/scipy/) module to perform the implied volatility calculation. The [matplotlib](https://pypi.org/project/matplotlib/) package will be used for charting. These dependencies can be installed with the following: ```bash $ pip install scipy matplotlib ``` ## Example ```python from collections.abc import Iterable import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd from matplotlib.ticker import PercentFormatter from scipy.optimize import root_scalar from scipy.stats import norm dataset = "GLBX.MDP3" def black_76( S: float, K: float, T: float, sigma: float, r: float = 0.05, is_call_option: bool = True, ) -> float: """ Calculate an option price from the Black-76 model. Parameters ---------- S : float Current underlying price. K : float Option strike price. T : float Time to expiration in years. sigma : float Implied volatility, annualized. r : float, default 0.05 The risk free interest rate, annualized. is_call_option : bool, default True Flag to indicate the option is a call or put. Returns ------- float """ d1 = (np.log(S / K) + (sigma**2 / 2) * T) / (sigma * np.sqrt(T)) d2 = d1 - sigma * np.sqrt(T) discount_factor = np.exp(-r * T) cp = 1 if is_call_option else -1 return discount_factor * cp * (S * norm.cdf(cp * d1) - K * norm.cdf(cp * d2)) def get_front_month_symbol( parent: str, start: pd.Timestamp, ) -> str: """ Get the symbol of the front month futures contract of a given parent symbol. Parameters ---------- parent : str The parent symbol, such as ES. start : pd.Timestamp The date to obtain the front month symbol for. Returns ------- str """ front_def = client.timeseries.get_range( dataset=db.Dataset.GLBX_MDP3, schema=db.Schema.DEFINITION, symbols=[f"{parent}.c.0"], stype_in=db.SType.CONTINUOUS, start=start.date(), end=(start + pd.Timedelta(days=1)).date(), ).to_df() return front_def.iloc[0]["raw_symbol"] def get_options_chain( parent: str, underlying: str, start: pd.Timestamp, ) -> pd.DataFrame: """ Retrieve the definitions of all options contract of the given parent symbol with for the underlying. Parameters ---------- parent : str The parent symbol, such as ES. underlying : str The underlying contract for the option. start : pd.Timestamp The date to obtain the definitions for. Returns ------- pd.DataFrame """ options_def = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=f"{parent}.OPT", stype_in="parent", start=start.date(), ) df = options_def.to_df() df = df[df["underlying"] == underlying] df = df[df["instrument_class"].isin(("C", "P"))] df["years_to_expiration"] = (df["expiration"] - start).dt.days / 365 return df.sort_values("strike_price") def get_midprice( symbols: Iterable[str], start: pd.Timestamp, end: pd.Timestamp, ) -> pd.Series: """ Get the last top-of-book midprice for one or more symbols between the `start` and `end` times. Parameters ---------- symbols : Iterable[str] A collection of symbols to retrieve the midprices for. start : pd.Timestamp The start time. end : pd.Timestamp The end time (exclusive). Returns ------- pd.Series """ price_df = client.timeseries.get_range( dataset=dataset, schema="mbp-1", symbols=symbols, start=start, end=end, ).to_df() price_df = price_df.groupby("symbol").last() price_df["midprice"] = np.mean(price_df[["bid_px_00", "ask_px_00"]], axis=1) return price_df["midprice"] def find_sigma( row: pd.Series, ) -> float: """ Find the roots of the Black-76 model by varying sigma, implied volatility. This function is for use with `pandas.Dataframe.apply`. Each row should contain a column for "strike_price", "years_to_expiration", "instrument_class", "midprice", and "underlying_price", If the optimization fails, `numpy.nan` is returned. Parameters ---------- row : pd.Series A series of data to process. Returns ------- float | numpy.nan """ def f(sigma: float) -> float: return row["midprice"] - black_76( S=row["underlying_price"], K=row["strike_price"], T=row["years_to_expiration"], sigma=sigma, is_call_option=row["instrument_class"] == "C", ) result = root_scalar(f, x0=0.1, x1=0.5) if result.converged: return result.root print( f"Could not find sigma for {row['raw_symbol']} with midprice {row['midprice']}", ) return np.nan # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Now, define some parameters for the calculation parent_symbol = "ES" start = pd.Timestamp("2024-04-04T17:00:00", tz="US/Central") end = pd.Timestamp("2024-04-04T18:00:00", tz="US/Central") # Get the front month futures contract for the parent symbol front_month_symbol = get_front_month_symbol( parent=parent_symbol, start=start, ) # Get the definitions for all options with the front-month future as the underlying options_chain = get_options_chain( parent=parent_symbol, underlying=front_month_symbol, start=start, ) # Retrieve the midprices for the options and front-month future midprices = get_midprice( symbols=[front_month_symbol, *options_chain["raw_symbol"]], start=start, end=end, ) # Then, join the options definitions with the midprices underlying_price = midprices.loc[front_month_symbol] options_chain = options_chain.join(midprices, on="symbol").dropna(subset="midprice") options_chain["underlying_price"] = underlying_price # Remove options with strike prices far from the underlying price options_chain = options_chain[ abs(options_chain["strike_price"] - underlying_price) <= underlying_price * 0.1 ] # Next, find the implied volatility for each option options_chain["sigma"] = options_chain.apply( find_sigma, axis=1, ) # Separate options into calls and puts call_options = options_chain[options_chain["instrument_class"] == "C"] put_options = options_chain[options_chain["instrument_class"] == "P"] # Finally, plot each curve of implied volatility by strike price plt.axvline( underlying_price, label="underlying price", linestyle=":", ) plt.plot( "strike_price", "sigma", data=call_options, label="call", ) plt.plot( "strike_price", "sigma", data=put_options, label="put", ) plt.title(f"{front_month_symbol} Implied Volatility by Strike - {end}") plt.ylabel("Implied Volatility ($\\sigma$)") plt.xlabel("Strike Price") plt.gca().yaxis.set_major_formatter(PercentFormatter(1)) plt.legend() plt.show() ``` ## Results ![Results](https://databento.com/docs/estimating-implied-volatility-0.svg.md) # Get symbols for 0DTE options ## Overview In this example we will show how to get symbols for 0DTE options for a product. 0DTE (Zero **D**ays **T**o **E**xpiration) options are contracts that expire at the end of the current session. ## Definition schema The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) contains instrument definitions and properties. In these examples, we are primarily interested in the `raw_symbol` and `expiration` fields. We will retrieve definition data for all options that trade under a parent symbol. We will then filter for 0DTE contracts based on `expiration`. We can then use `raw_symbol` to pass a list of symbols into another data query. ## Historical example First, we'll take a look at how to do this with the Historical client. We'll use a pandas DataFrame to filter for this example. ```python import datetime import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "OPRA.PILLAR" parent_symbol = "SPXW" # SPX Weekly Options date = datetime.date(2025, 9, 10) # Download definition data for a parent symbol def_data = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=f"{parent_symbol}.OPT", stype_in="parent", start=date, ) # Convert to DataFrame and filter for 0DTE def_df = def_data.to_df() def_df = def_df[def_df["expiration"].dt.date == date] # Print out raw_symbols, expiration, and strike price print(def_df[["raw_symbol", "expiration", "strike_price"]]) # Download some TCBBO data with these symbols tcbbo_data = client.timeseries.get_range( dataset=dataset, schema="tcbbo", symbols=def_df["raw_symbol"].to_list(), start=date, limit=10000, ) # Convert to DataFrame tcbbo_df = tcbbo_data.to_df() # Print TCBBO DataFrame print(tcbbo_df) ``` ## Result ```plaintext raw_symbol expiration strike_price ts_recv 2025-09-10 10:30:02.624717419+00:00 SPXW 250910P04000000 2025-09-10 00:00:00+00:00 4000.0 2025-09-10 10:30:02.634731457+00:00 SPXW 250910P06700000 2025-09-10 00:00:00+00:00 6700.0 2025-09-10 10:30:02.644740147+00:00 SPXW 250910P06200000 2025-09-10 00:00:00+00:00 6200.0 2025-09-10 10:30:02.644740147+00:00 SPXW 250910C05775000 2025-09-10 00:00:00+00:00 5775.0 2025-09-10 10:30:02.662006482+00:00 SPXW 250910P06450000 2025-09-10 00:00:00+00:00 6450.0 ... ... ... ... 2025-09-10 10:30:29.833577055+00:00 SPXW 250910C05970000 2025-09-10 00:00:00+00:00 5970.0 2025-09-10 10:30:29.844558808+00:00 SPXW 250910C06145000 2025-09-10 00:00:00+00:00 6145.0 2025-09-10 10:30:29.869477754+00:00 SPXW 250910C06065000 2025-09-10 00:00:00+00:00 6065.0 2025-09-10 10:30:29.883667299+00:00 SPXW 250910C05960000 2025-09-10 00:00:00+00:00 5960.0 2025-09-10 10:30:29.994986746+00:00 SPXW 250910P06710000 2025-09-10 00:00:00+00:00 6710.0 [406 rows x 3 columns] ``` ```plaintext ts_event rtype publisher_id instrument_id side ... bid_sz_00 ask_sz_00 bid_pb_00 ask_pb_00 symbol ts_recv ... 2025-09-10 13:30:02.764440257+00:00 2025-09-10 13:30:02.764236273+00:00 194 22 1241523583 N ... 0 0 0 0 SPXW 250910P06350000 2025-09-10 13:30:02.764443670+00:00 2025-09-10 13:30:02.764240780+00:00 194 22 1241523583 N ... 0 0 0 0 SPXW 250910P06350000 2025-09-10 13:30:02.764725051+00:00 2025-09-10 13:30:02.764522162+00:00 194 22 1241515705 N ... 0 0 0 0 SPXW 250910P06400000 2025-09-10 13:30:02.764725051+00:00 2025-09-10 13:30:02.764522162+00:00 194 22 1241515705 N ... 0 0 0 0 SPXW 250910P06400000 2025-09-10 13:30:02.764734633+00:00 2025-09-10 13:30:02.764531350+00:00 194 22 1241515705 N ... 0 0 0 0 SPXW 250910P06400000 ... ... ... ... ... ... ... ... ... ... ... ... 2025-09-10 13:31:50.928232746+00:00 2025-09-10 13:31:50.928030554+00:00 194 22 1241531922 N ... 385 364 22 22 SPXW 250910C06610000 2025-09-10 13:31:50.956014355+00:00 2025-09-10 13:31:50.955812019+00:00 194 22 1241520104 N ... 134 200 22 22 SPXW 250910P06495000 2025-09-10 13:31:50.961115681+00:00 2025-09-10 13:31:50.960913601+00:00 194 22 1241528118 N ... 234 200 22 22 SPXW 250910C06555000 2025-09-10 13:31:50.962087120+00:00 2025-09-10 13:31:50.961883488+00:00 194 22 1241516275 N ... 136 60 22 22 SPXW 250910C06545000 2025-09-10 13:31:50.962087120+00:00 2025-09-10 13:31:50.961883488+00:00 194 22 1241519790 N ... 54 188 22 22 SPXW 250910C06550000 [10000 rows x 15 columns] ``` ## Live example Next, we'll take a look at how to do this with the Live client. We'll use the `pretty_expiration` attribute which converts `expiration` into a datetime object to filter. > **Info** > > You will need a live license for `OPRA.PILLAR` to run this example. See our [plans and live data page]($PLATFORM_HOSTNAME/docs/portal/live-data) for more information. ```python import datetime import databento as db # Create a live client live_client = db.Live("$YOUR_API_KEY", compression=db.Compression.ZSTD) # Set parameters dataset = "OPRA.PILLAR" parent_symbol = "SPXW" # SPX Weekly Options date = datetime.datetime.now(tz=datetime.timezone.utc).date() # Subscribe to the definition schema for the parent symbol for this session live_client.subscribe( dataset=dataset, schema="definition", symbols=f"{parent_symbol}.OPT", stype_in="parent", start=0, ) # Create a list of 0DTE symbols symbols = [] def append_symbol_list(msg): if isinstance(msg, db.InstrumentDefMsg): # Filter for 0DTE if msg.pretty_expiration.date() == date: symbols.append(msg.raw_symbol) # Add the callback and start the stream live_client.add_callback(append_symbol_list) live_client.start() # Listen to the stream for 15 seconds to get all messages live_client.block_for_close(timeout=15) # Print out the symbol count and the first 5 symbols print(f"Total symbol count for {dataset} = {len(symbols)}\nFirst 5 symbols...{symbols[:5]}") # Re-subscribe to the TCBBO schema with 0DTE symbols live_client.subscribe( dataset=dataset, schema="tcbbo", symbols=symbols, ) # Add a print callback and start the stream live_client.add_callback(print) live_client.start() # Listen to the stream for 10 seconds live_client.block_for_close(timeout=10) ``` ## Result ```plaintext Total symbol count for OPRA.PILLAR = 139 First 5 symbols...['SPXW 260122P02800000', 'SPXW 260122P08200000', 'SPXW 260122C05800000', 'SPXW 260122C04800000', 'SPXW 260122C03400000'] ``` ```plaintext SymbolMappingMsg(rtype=, publisher_id=0, instrument_id=1174406426, ts_event=1769047672270047890, pretty_ts_event='2026-01-22T02:07:52.270047890Z', stype_in=None, stype_in_symbol='SPXW 260122P02800000', stype_out=None, stype_out_symbol='SPXW 260122P02800000', start_ts=18446744073709551615, pretty_start_ts='2554-07-21T23:34:33.709551615Z', end_ts=18446744073709551615, pretty_end_ts='2554-07-21T23:34:33.709551615Z') ... Cmbp1Msg(ts_recv=1769027025118777730, pretty_ts_recv='2026-01-21T20:23:45.118777730Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410419, ts_event=1769027025118577558, pretty_ts_event='2026-01-21T20:23:45.118577558Z', action='T', side='N', price=2420000000, pretty_price=2.42, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=2400000000, pretty_bid_px_00=2.4, ask_px_00=2450000000, pretty_ask_px_00=2.45, bid_sz_00=24, ask_sz_00=33, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027025118782732, pretty_ts_recv='2026-01-21T20:23:45.118782732Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410419, ts_event=1769027025118582557, pretty_ts_event='2026-01-21T20:23:45.118582557Z', action='T', side='N', price=2420000000, pretty_price=2.42, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=2400000000, pretty_bid_px_00=2.4, ask_px_00=2450000000, pretty_ask_px_00=2.45, bid_sz_00=24, ask_sz_00=33, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027026311010903, pretty_ts_recv='2026-01-21T20:23:46.311010903Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174415604, ts_event=1769027026310810688, pretty_ts_event='2026-01-21T20:23:46.310810688Z', action='T', side='N', price=19140000000, pretty_price=19.14, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=19000000000, pretty_bid_px_00=19, ask_px_00=19300000000, pretty_ask_px_00=19.3, bid_sz_00=19, ask_sz_00=28, bid_pb_00=22, ask_pb_00=22) ... Cmbp1Msg(ts_recv=1769027042926074875, pretty_ts_recv='2026-01-21T20:24:02.926074875Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410087, ts_event=1769027042925867805, pretty_ts_event='2026-01-21T20:24:02.925867805Z', action='T', side='N', price=10720000000, pretty_price=10.72, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=10600000000, pretty_bid_px_00=10.6, ask_px_00=10800000000, pretty_ask_px_00=10.8, bid_sz_00=13, ask_sz_00=11, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027043433405443, pretty_ts_recv='2026-01-21T20:24:03.433405443Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410087, ts_event=1769027043433197884, pretty_ts_event='2026-01-21T20:24:03.433197884Z', action='T', side='N', price=10700000000, pretty_price=10.7, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=10600000000, pretty_bid_px_00=10.6, ask_px_00=10800000000, pretty_ask_px_00=10.8, bid_sz_00=14, ask_sz_00=26, bid_pb_00=22, ask_pb_00=22) Cmbp1Msg(ts_recv=1769027043433405682, pretty_ts_recv='2026-01-21T20:24:03.433405682Z', rtype=, publisher_id=OPRA.PILLAR.XCBO (22), instrument_id=1174410087, ts_event=1769027043433202613, pretty_ts_event='2026-01-21T20:24:03.433202613Z', action='T', side='N', price=10700000000, pretty_price=10.7, size=1, flags=LAST | TOB (192), ts_in_delta=0, bid_px_00=10600000000, pretty_bid_px_00=10.6, ask_px_00=10800000000, pretty_ask_px_00=10.8, bid_sz_00=14, ask_sz_00=26, bid_pb_00=22, ask_pb_00=22) ``` # Daily statistics for equity options ## Overview In this example, we'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate volume and open interest statistics for an equity options chain. First, we'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties, to get information such as `expiration`, `strike_price`, and `instrument_class`. Next, we'll use the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md) to get the total volume for each option. Finally, we'll use the [statistics schema](https://databento.com/docs/schemas/statistics.md) to request open interest statistics. OPRA publishes start-of-day open interest statistics before the regular trading hours (RTH) open at 9:30 ET. ## Example ```python import datetime as dt from zoneinfo import ZoneInfo import databento as db import matplotlib.pyplot as plt import pandas as pd from matplotlib.patches import Patch from matplotlib.ticker import MaxNLocator def get_volume_data( client: db.Historical, dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """Get volume data for all options for a given symbol.""" volume_df = client.timeseries.get_range( dataset=dataset, symbols=f"{symbol}.OPT", schema="ohlcv-1d", stype_in="parent", start=date, ).to_df() return volume_df.groupby("symbol")["volume"].sum().reset_index() def get_oi_data( client: db.Historical, dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """Get open interest data for all options for a given symbol.""" # Start-of-day open interest is published before the RTH open at 9:30 ET end_time = dt.time(9, 30, 0, tzinfo=ZoneInfo("America/New_York")) stats_df = client.timeseries.get_range( dataset=dataset, symbols=f"{symbol}.OPT", schema="statistics", stype_in="parent", start=date, end=dt.datetime.combine(date, end_time), ).to_df() stats_df = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST] stats_df = stats_df.drop_duplicates("symbol", keep="last") stats_df["open_interest"] = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST][ "quantity" ] return stats_df[["open_interest", "symbol"]] def get_definition_data( client: db.Historical, dataset: str, symbol: str, date: dt.date, ) -> pd.DataFrame: """Get definition data for all options for a given symbol.""" def_df = client.timeseries.get_range( dataset=dataset, symbols=f"{symbol}.OPT", schema="definition", stype_in="parent", start=date, ).to_df() def_df["days_to_expiration"] = (def_df["expiration"] - def_df.index.normalize()).dt.days def_df["expiration"] = def_df["expiration"].dt.normalize().dt.date return def_df[ ["symbol", "strike_price", "instrument_class", "expiration", "days_to_expiration"] ] def plot_oi_by_strike( df: pd.DataFrame, expiration_date: dt.date, ) -> None: df = df[df["expiration"] == expiration_date] trade_date = df["trade_date"].iloc[0] days_to_expiration = df["days_to_expiration"].iloc[0] df_strikes = df.groupby(["strike_price", "instrument_class"])["open_interest"].sum().unstack() # Plot bar chart _, ax = plt.subplots(figsize=(14, 6)) df_strikes.plot( ax=ax, kind="bar", xlabel="Strike price", ylabel="Open interest", ) # Adjust X ticks and labels ax.xaxis.set_major_locator(MaxNLocator(15)) ax.set_xticklabels(ax.get_xticklabels(), rotation=45, ha="right") # Set title and legend ax.set_title(f"{symbol} open interest \n {trade_date} ({days_to_expiration} DTE)") ax.legend(handles=[Patch(facecolor="C0", label="Call"), Patch(facecolor="C1", label="Put")]) plt.tight_layout() plt.show() # Set parameters dataset = "OPRA.PILLAR" symbol = "NVDA" start = dt.date(2025, 8, 19) # Create a historical client client = db.Historical("$YOUR_API_KEY") # Get strike price, expiration, and instrument class in the definition schema def_df = get_definition_data(client, dataset, symbol, start) # Volume from the OHLCV-1d schema volume_df = get_volume_data(client, dataset, symbol, start) # Start-of-day open interest is published in the morning in the statistics schema stats_df = get_oi_data(client, dataset, symbol, start) # Merge different schemas together df = def_df.merge(volume_df, on="symbol", how="left") df = df.merge(stats_df, on="symbol", how="left") df["trade_date"] = start df["volume"] = df["volume"].fillna(0).astype(int) df["open_interest"] = df["open_interest"].fillna(0).astype(int) # Print out top 10 contracts by open interest print(df.nlargest(10, "open_interest").to_string(index=False)) # Plot open interest by expiration date plot_oi_by_strike(df, dt.date(2025, 8, 22)) ``` ## Result You could use the resulting data to filter for the contracts with the highest open interest. Additionally, you could plot the open interest distribution for a given expiration. ```plaintext symbol strike_price instrument_class expiration days_to_expiration volume open_interest trade_date NVDA 250919C00160000 160.0 C 2025-09-19 31 52206 173864 2025-08-19 NVDA 260618P00075000 75.0 P 2026-06-18 303 1 109796 2025-08-19 NVDA 250919C00170000 170.0 C 2025-09-19 31 4636 106760 2025-08-19 NVDA 250822C00185000 185.0 C 2025-08-22 3 158486 102111 2025-08-19 NVDA 250919C00185000 185.0 C 2025-09-19 31 61860 101843 2025-08-19 NVDA 250919C00190000 190.0 C 2025-09-19 31 59766 92379 2025-08-19 NVDA 250822C00190000 190.0 C 2025-08-22 3 46489 91666 2025-08-19 NVDA 250919C00165000 165.0 C 2025-09-19 31 759 90851 2025-08-19 NVDA 250919C00175000 175.0 C 2025-09-19 31 5862 85548 2025-08-19 NVDA 260116C00100000 100.0 C 2026-01-16 150 161 85414 2025-08-19 ``` ![Options venues volume](https://databento.com/docs/equity-open-interest-0.svg.md) # Get end of day option spreads ## Overview In this example, we'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to get the end of day spread for an options chain. First, we'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties, to get information such as `expiration`, `strike_price`, and `instrument_class`. We'll also use the [CBBO-1m schema](https://databento.com/docs/schemas/bbo.md#consolidated-bbo-on-interval-cbbo) to get the bid/ask spread for each option. We'll plot the end-of-day midprice and spread in basis-points (BPS) across all strikes for multiple expirations. ## Example ```python import datetime as dt from zoneinfo import ZoneInfo import databento as db import matplotlib.pyplot as plt import pandas as pd def get_options_data( client: db.Historical, symbol: str, schema: str, start: dt.datetime, end: dt.datetime | None = None, ) -> pd.DataFrame: """Fetch options data between timestamps""" return client.timeseries.get_range( dataset="OPRA.PILLAR", schema=schema, stype_in="parent", symbols=f"{symbol}.OPT", start=start, end=end, ).to_df() def combine_data( trade_date: dt.date, df_def: pd.DataFrame, df_close: pd.DataFrame, ) -> pd.DataFrame: """Combine open and close data with proper column naming""" df_close = df_close.groupby("symbol")[["bid_px_00", "ask_px_00"]].last() df_def = df_def[["symbol", "instrument_class", "expiration", "strike_price"]] df_def["dte"] = (df_def["expiration"].dt.tz_localize(None) - pd.Timestamp(trade_date)).dt.days df_def = df_def.set_index("symbol") df = df_close.join(df_def, on="symbol", how="left").reset_index().sort_values("symbol") # Calculate spreads in basis points df["mid_px"] = (df["bid_px_00"] + df["ask_px_00"]) / 2 df["spread"] = df["ask_px_00"] - df["bid_px_00"] df["spread_bps"] = (df["spread"] / df["mid_px"]) / 1e-4 df["trade_date"] = trade_date return df def plot_spreads( symbol: str, df: pd.DataFrame, target_dtes: tuple[int], ) -> None: """Plot bid-ask spreads for selected expirations""" fig, axes = plt.subplots(len(target_dtes), 1, figsize=(12, 12)) selected_expirations = [ df.loc[(df["dte"] - t).abs().idxmin(), "expiration"] for t in target_dtes ] subset = df[df["expiration"].isin(selected_expirations)] y_min = subset["mid_px"].min() y_max = subset["mid_px"].max() y_range = y_max - y_min x_min = subset["strike_price"].min() x_max = subset["strike_price"].max() x_padding = (x_max - x_min) * 0.02 y_padding = y_range * 0.1 # 10% of y_range = 1000 bps y_spread_scale = (y_range * 0.10) / 1000 for idx, (ax, exp) in enumerate(zip(axes, selected_expirations)): df_exp = df[df["expiration"] == exp].copy() # Scale spreads for visualization df_exp["scaled_spread"] = df_exp["spread_bps"].apply(lambda x: x * y_spread_scale) def plot_options( instrument_class: db.InstrumentClass, color: str, ) -> None: data = df_exp[df_exp["instrument_class"] == instrument_class].sort_values( "strike_price", ) if len(data) > 0: half = data["scaled_spread"] / 2 ax.vlines( x=data["strike_price"], ymin=data["mid_px"] - half, ymax=data["mid_px"] + half, colors=color, linewidth=2, alpha=0.7, label=instrument_class.name.capitalize(), ) for y_offset in (half, -half): ax.hlines( y=data["mid_px"] + y_offset, xmin=data["strike_price"] - 0.5, xmax=data["strike_price"] + 0.5, colors=color, linewidth=1.5, alpha=0.7, ) ax.plot(data["strike_price"], data["mid_px"], "s", color=color, markersize=3) plot_options(db.InstrumentClass.CALL, "C0") plot_options(db.InstrumentClass.PUT, "C1") ref_x = x_max - (x_max - x_min) * 0.05 ref_y_base = y_min * 0.3 s1000 = y_spread_scale * 1000 ax.plot([ref_x, ref_x], [ref_y_base, ref_y_base + s1000], "k-", linewidth=2) for y_off, label in [(0, "0 bps"), (s1000, "1,000 bps")]: ax.hlines( y=ref_y_base + y_off, xmin=ref_x - 1, xmax=ref_x + 1, colors="black", linewidth=1.5, ) ax.text( ref_x - 3, ref_y_base + y_off, label, fontsize=8, va="center", ha="right", ) ax.annotate( f"DTE = {df_exp['dte'].iloc[0]} days", xy=(0.02, 0.98), xycoords="axes fraction", fontsize=11, ha="left", va="top", bbox=dict(boxstyle="round,pad=0.3", fc="white", ec="gray", alpha=0.8), ) ax.set_xlim(x_min - x_padding, x_max + x_padding) ax.set_ylim(y_min - y_padding, y_max + y_padding) ax.set_ylabel("Option Price ($)", fontsize=11) ax.grid(True, alpha=0.3) ax.legend(loc="upper right") fig.suptitle(f"{symbol} Options End of Day NBBO ({df['trade_date'].iloc[0]})") fig.supxlabel("Strike Price ($)", fontsize=11) plt.tight_layout() plt.show() symbol = "AAPL" start_date = dt.date(2025, 11, 19) close_time = dt.time(16, 0, 0, tzinfo=ZoneInfo("US/Eastern")) close_dt = dt.datetime.combine(start_date, close_time) client = db.Historical("$YOUR_API_KEY") df_close = get_options_data( client, symbol, "cbbo-1m", close_dt - dt.timedelta(minutes=15), close_dt, ) df_definition = get_options_data(client, symbol, "definition", start_date) df = combine_data(start_date, df_definition, df_close) print(df) # Plot spreads for targeted DTEs plot_spreads(symbol, df, target_dtes=(0, 30, 60)) ``` ## Result ```plaintext symbol bid_px_00 ask_px_00 instrument_class expiration ... dte mid_px spread spread_bps trade_date 0 AAPL 251121C00100000 167.30 168.55 C 2025-11-21 00:00:00+00:00 ... 2 167.925 1.25 74.437993 2025-11-19 1 AAPL 251121C00105000 162.30 165.25 C 2025-11-21 00:00:00+00:00 ... 2 163.775 2.95 180.125172 2025-11-19 2 AAPL 251121C00110000 157.25 160.25 C 2025-11-21 00:00:00+00:00 ... 2 158.750 3.00 188.976378 2025-11-19 3 AAPL 251121C00115000 152.25 155.25 C 2025-11-21 00:00:00+00:00 ... 2 153.750 3.00 195.121951 2025-11-19 4 AAPL 251121C00120000 147.25 150.25 C 2025-11-21 00:00:00+00:00 ... 2 148.750 3.00 201.680672 2025-11-19 ... ... ... ... ... ... ... ... ... ... ... ... 2690 AAPL 280121P00490000 219.00 224.00 P 2028-01-21 00:00:00+00:00 ... 793 221.500 5.00 225.733634 2025-11-19 2691 AAPL 280121P00500000 229.00 234.00 P 2028-01-21 00:00:00+00:00 ... 793 231.500 5.00 215.982721 2025-11-19 2692 AAPL 280121P00510000 239.00 244.00 P 2028-01-21 00:00:00+00:00 ... 793 241.500 5.00 207.039337 2025-11-19 2693 AAPL 280121P00520000 249.00 254.00 P 2028-01-21 00:00:00+00:00 ... 793 251.500 5.00 198.807157 2025-11-19 2694 AAPL 280121P00530000 259.00 264.00 P 2028-01-21 00:00:00+00:00 ... 793 261.500 5.00 191.204589 2025-11-19 [2695 rows x 11 columns] ``` ![Options venues volume](https://databento.com/docs/option-spreads-0.svg.md) ## Live data | Title | Subtitle | Supported languages | | [Handle multiple record types](https://databento.com/docs/examples/basics-live/live-dispatch.md) | Handle different record types using callbacks. | Python, C++, Rust, Live | | [Stream live data to a file](https://databento.com/docs/examples/basics-live/live-stream-to-file.md) | Write and save live data to a file. | Python, C++, Rust, Live | | [Estimate Databento feed latency](https://databento.com/docs/examples/basics-live/live-latency.md) | Get time between our servers and your real-time application. | Python, C++, Rust, Live | | [Calculate TICK and TRIN indicators](https://databento.com/docs/examples/basics-live/live-tick-trin.md) | Compute market TICK and TRading INdex on real-time trades. | Python, C++, Rust, Live | | [Subscribe to MBO snapshot](https://databento.com/docs/examples/basics-live/live-snapshot.md) | Get the latest order book state without replaying the entire trading session. | Python, C++, Rust, Live | | [Compare on-exchange and off-exchange trade volume](https://databento.com/docs/examples/basics-live/live-nasdaq-trf.md) | Compare the volumes of the different venues of Nasdaq Basic with NLS Plus. | Python, Live | # Handle multiple record types ## Overview In this example we will use the [LiveThreaded client](https://databento.com/docs/api-live/client/live-threaded.md) to process order book data using a callback function. We'll use the `GetIf` templated method on `Record` which returns a pointer to the specified type if the record is that type, otherwise `nullptr`. We'll chain calls to this method to handle the various record types we might receive from the live gateway. ## MBO schema We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) in this example but any schema's records can be dispatched this way. ## Example ```cpp #include // seconds #include // LiveBuilder #include // Px #include // Dataset #include // Record, SymbolMappingMsg, MboMsg, ErrorMsg #include #include namespace db = databento; // First, we create our callback function, pulling out different record types // with GetIf db::KeepGoing RecordHandler(const db::Record& record) { if (auto* sym = record.GetIf()) { std::cout << sym->STypeInSymbol() << " has an instrument ID of " << sym->hd.instrument_id << '\n'; } else if (auto* mbo = record.GetIf()) { std::cout << mbo->order_id << '\t' << mbo->side << '\t' << mbo->size << '\t' << std::setprecision(6) << db::pretty::Px{mbo->price} << '\n'; } else if (auto* error = record.GetIf()) { std::cerr << error->Err() << '\n'; } else if (auto* system = record.GetIf()) { std::cout << system->Msg() << '\n'; } return db::KeepGoing::Continue; } int main() { auto client = db::LiveThreaded::Builder() .SetKey("$YOUR_API_KEY") .SetDataset(db::Dataset::GlbxMdp3) .BuildThreaded(); // First we subscribe to the MBO schema client.Subscribe({"ZN.c.0"}, db::Schema::Mbo, db::SType::Continuous); // Now, add our callback function and start the streaming session client.Start(RecordHandler); // Finally, we will wait for 5 seconds before closing the connection client.BlockForStop(std::chrono::seconds{5}); return 0; } ``` ## Result ```plaintext Subscription request 1 for mbo data succeeded ZN.c.0 has an instrument ID of 42066706 8416556618490 Bid 1 112.015625 8416556618521 Bid 35 112.046875 8416556618538 Bid 4 112.046875 ``` # Handle multiple record types ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to process order book data using [a callback](https://databento.com/docs/api-live/client/add-callback.md) function. We will use Python's [functools.singledispatch](https://docs.python.org/3/library/functools.html#functools.singledispatch) to make handling different record types easier. ## MBO schema We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) in this example but any schema's records can be dispatched this way. ## Example ```python from functools import singledispatch import databento as db # First, we will create our callback function using singledispatch @singledispatch def dbn_handler(_: db.DBNRecord): pass @dbn_handler.register def _(sym: db.SymbolMappingMsg): instrument_id = sym.instrument_id symbol = sym.stype_in_symbol print(f"{symbol} has an instrument ID of {instrument_id}") @dbn_handler.register def _(mbo: db.MBOMsg): order_id = f"{mbo.order_id:13d}" side = mbo.side size = mbo.size price = f"${mbo.pretty_price:,.06f}" print(order_id, side, size, price, sep="\t") @dbn_handler.register def _(sys: db.SystemMsg): print(f"{sys.msg}") @dbn_handler.register def _(error: db.ErrorMsg): print(f"ERROR: {error.err}") # Then, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the MBO schema live_client.subscribe( dataset="GLBX.MDP3", schema="mbo", stype_in="continuous", symbols="ZN.c.0", ) # Now, add our callback function and start the streaming session live_client.add_callback(dbn_handler) live_client.start() # Finally, we will wait for 5 seconds before closing the connection live_client.block_for_close(timeout=5) ``` ## Result ```plaintext Subscription request 1 for mbo data succeeded ZN.c.0 has an instrument ID of 42066706 8416556628429 A 4 $112.062500 8416556628472 A 3 $112.046875 8416556625196 B 1 $112.046875 8416556625202 B 1 $112.046875 8416556625204 B 1 $112.046875 8416556625196 B 1 $112.046875 8416556625202 B 1 $112.046875 8416556625204 B 1 $112.046875 8416556628476 A 1 $112.062500 8416556628477 A 1 $112.062500 8416556628479 A 1 $112.062500 8416556628482 A 1 $112.125000 8416556628483 A 1 $112.125000 8416556628484 B 1 $112.046875 8416556628128 B 5 $112.046875 8416556628379 B 4 $112.046875 ... ``` # Handle multiple record types ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to process order book data using a callback function. We'll use the generic `get` method on `RecordRef` which returns a reference to the specified record type `T`, otherwise `None`. We'll chain calls to this method with `if let` to handle the various record types we might receive from the live gateway. ## MBO schema We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) in this example but any schema's records can be dispatched this way. ## Example ```rust use databento::{ LiveClient, dbn::{ Dataset::GlbxMdp3, ErrorMsg, MboMsg, SType, Schema, SymbolMappingMsg, SystemMsg, pretty, }, live::Subscription, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = LiveClient::builder() .key_from_env()? .dataset(GlbxMdp3) .build() .await?; // First we subscribe to the MBO schema client .subscribe( Subscription::builder() .schema(Schema::Mbo) .symbols("ZN.c.0") .stype_in(SType::Continuous) .build(), ) .await?; client.start().await?; // Then, we iterate over the first 5 records for _ in 0..5 { let Some(rec) = client.next_record().await? else { break; }; // We handle different record types by chaining calls to `rec.get()` if let Some(sym) = rec.get::() { println!( "{} has an instrument ID of {}", sym.stype_in_symbol()?, sym.hd.instrument_id, ); } else if let Some(mbo) = rec.get::() { println!( "{}\t{:?}\t{}\t{:.6}", mbo.order_id, mbo.side()?, mbo.size, pretty::Px(mbo.price) ); } else if let Some(error) = rec.get::() { eprintln!("{}", error.err()?); } else if let Some(system) = rec.get::() { println!("{}", system.msg()?); } } Ok(()) } ``` ## Result ```plaintext ZN.c.0 has an instrument ID of 42002321 8414557391806 Ask 1 113.328125000 8414557387367 Bid 5 113.234375000 8414557378440 Bid 5 113.234375000 8414557375529 Bid 5 113.234375000 ``` # Stream live data to a file ## Overview In this example we will use the [LiveBlocking client](https://databento.com/docs/api-live/client/live-blocking.md) to save live data to a file using an [output stream](https://databento.com/docs/api-live/client/add-stream.md). We will then open this file as a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md). This allows for live data to be archived for replay and testing purposes. ## Databento Binary Encoding (DBN) The binary data will be written to the stream in the [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md) format. This is a zero-copy format that is extremely fast and efficient on bandwidth. The easiest way to read data encoded in DBN is through our client libraries or the [dbn-cli](https://crates.io/crates/dbn-cli) tool. ## Example ```cpp #include #include #include #include #include // OutFileStream #include // LiveBuilder #include #include // Dataset #include // Record #include // PitSymbolMap #include #include namespace db = databento; int main() { auto client = db::LiveBlocking::Builder() .SetKey("$YOUR_API_KEY") .SetDataset(db::Dataset::GlbxMdp3) .BuildBlocking(); // First, we subscribe to the OHLCV-1s schema for a few symbols client.Subscribe({"ES.c.0", "ES.c.1", "CL.c.0", "CL.c.1"}, db::Schema::Ohlcv1S, db::SType::Continuous); // We start streaming db::Metadata metadata = client.Start(); { // We open a file for writing and encode the metadata db::OutFileStream out_file{"example.dbn"}; db::DbnEncoder encoder{metadata, &out_file}; // Wait for at least one record and 5 seconds before closing the connection const std::chrono::seconds timeout{5}; const auto start = std::chrono::steady_clock::now(); while (auto* record = client.NextRecord(timeout)) { encoder.EncodeRecord(*record); if (std::chrono::steady_clock::now() - start >= timeout) { break; } } } // Finally, we open the DBN file db::DbnStore store{"example.dbn"}; std::cout << std::left << std::setw(32) << "ts_event" << std::right << std::setw(16) << "instrument_id" << std::setw(12) << "open" << std::setw(12) << "high" << std::setw(12) << "low" << std::setw(12) << "close" << std::setw(10) << "volume" << " " << "symbol\n"; db::PitSymbolMap symbol_map; while (auto* record = store.NextRecord()) { symbol_map.OnRecord(*record); if (auto* ohlcv = record->GetIf()) { std::cout << std::left << std::setw(32) << db::pretty::Ts{ohlcv->hd.ts_event} << std::right << std::setw(16) << ohlcv->hd.instrument_id << std::setprecision(2) << std::setw(12) << db::pretty::Px{ohlcv->open} << std::setw(12) << db::pretty::Px{ohlcv->high} << std::setw(12) << db::pretty::Px{ohlcv->low} << std::setw(12) << db::pretty::Px{ohlcv->close} << std::setw(10) << ohlcv->volume << " " << symbol_map.At(*ohlcv) << '\n'; } } return 0; } ``` ## Result ```plaintext ts_event instrument_id open high low close volume symbol 2026-04-20T16:27:46.000000000Z 42140864 7137.25 7137.25 7137.00 7137.00 57 ESM6 2026-04-20T16:27:47.000000000Z 670557 87.22 87.22 87.22 87.22 3 CLM6 2026-04-20T16:27:48.000000000Z 670557 87.23 87.23 87.22 87.22 3 CLM6 2026-04-20T16:27:48.000000000Z 42140864 7137.00 7137.00 7137.00 7137.00 2 ESM6 2026-04-20T16:27:49.000000000Z 42140864 7137.00 7137.00 7137.00 7137.00 23 ESM6 2026-04-20T16:27:50.000000000Z 670557 87.23 87.23 87.21 87.22 5 CLM6 2026-04-20T16:27:50.000000000Z 42140864 7137.00 7137.00 7137.00 7137.00 1 ESM6 2026-04-20T16:27:51.000000000Z 670557 87.21 87.21 87.21 87.21 1 CLM6 ``` # Stream live data to a file ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to save live data to a file using an [output stream](https://databento.com/docs/api-live/client/add-stream.md). We will then open this file as a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md). This allows for live data to be archived for replay and testing purposes. ## Databento Binary Encoding (DBN) The binary data will be written to the stream in the [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md) format. This is a zero-copy format that is extremely fast and efficient on bandwidth. The easiest way to read data encoded in DBN is through our client libraries or the [dbn-cli](https://crates.io/crates/dbn-cli) tool. ## Example ```python import databento as db # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the OHLCV-1s schema for a few symbols live_client.subscribe( dataset="GLBX.MDP3", schema="ohlcv-1s", stype_in="continuous", symbols=["ES.c.0", "ES.c.1", "CL.c.0", "CL.c.1"], ) # Now, we will open a file for writing and start streaming live_client.add_stream("example.dbn") live_client.start() # We will listen for 10 seconds before closing the connection # Any ohlcv-1s bars received during this time will be saved live_client.block_for_close(timeout=10) # Finally, we will open the DBN file dbn_store = db.DBNStore.from_file("example.dbn") print(dbn_store.to_df(schema="ohlcv-1s")) ``` ## Result ```plaintext rtype publisher_id instrument_id open high low close volume symbol ts_event 2023-12-18 14:40:19+00:00 32 1 17077 4785.75 4785.75 4785.75 4785.75 143 ESH4 2023-12-18 14:40:19+00:00 32 1 686071 74.19 74.19 74.19 74.19 6 CLG4 2023-12-18 14:40:20+00:00 32 1 17077 4785.75 4785.75 4785.50 4785.75 8 ESH4 2023-12-18 14:40:20+00:00 32 1 686071 74.19 74.20 74.19 74.20 6 CLG4 2023-12-18 14:40:21+00:00 32 1 17077 4785.75 4785.75 4785.75 4785.75 105 ESH4 2023-12-18 14:40:21+00:00 32 1 686071 74.20 74.21 74.20 74.21 8 CLG4 ``` # Stream live data to a file ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to save live data to a file using an [output stream](https://databento.com/docs/api-live/client/add-stream.md). We will then open this file as a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md). This allows for live data to be archived for replay and testing purposes. ## Databento Binary Encoding (DBN) The binary data will be written to the stream in the [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md) format. This is a zero-copy format that is extremely fast and efficient on bandwidth. The easiest way to read data encoded in DBN is through our client libraries or the [dbn-cli](https://crates.io/crates/dbn-cli) tool. ## Example ```rust use std::time::Instant; use databento::{ LiveClient, dbn::{ Dataset::GlbxMdp3, OhlcvMsg, PitSymbolMap, SType, Schema, decode::AsyncDbnDecoder, encode::{AsyncDbnEncoder, AsyncEncodeRecord, AsyncEncodeRecordRef}, pretty::{Px, Ts}, }, live::Subscription, }; use tokio::{fs::File, io::BufWriter}; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = LiveClient::builder() .key_from_env()? .dataset(GlbxMdp3) .build() .await?; // First, we subscribe to the OHLCV-1s schema for a few symbols client .subscribe( Subscription::builder() .schema(Schema::Ohlcv1S) .symbols(vec!["ES.c.0", "ES.c.1", "CL.c.0", "CL.c.1"]) .stype_in(SType::Continuous) .build(), ) .await?; // We start streaming let metadata = client.start().await?; // We open a file for writing and encode the metadata let mut encoder = AsyncDbnEncoder::new( BufWriter::new(File::create("example.dbn").await?), &metadata, ) .await?; // Wait for at least one record and 5 seconds before closing the connection let now = Instant::now(); while let Some(record) = client.next_record().await? { encoder.encode_record_ref(record).await?; if now.elapsed().as_secs() >= 5 { break; } } // Next, we shutdown the encoder to ensure any buffered output // is written to the file encoder.shutdown().await?; // Finally, we open the DBN file let mut dbn_file = AsyncDbnDecoder::new(File::open("example.dbn").await?).await?; let mut symbol_map = PitSymbolMap::new(); println!( "{:<32}{:>13}{:>11}{:>11}{:>11}{:>11}{:>8} symbol", "ts_event", "instrument_id", "open", "high", "low", "close", "volume" ); while let Some(record) = dbn_file.decode_record_ref().await? { symbol_map.on_record(record)?; if let Some(ohlcv) = record.get::() { let symbol = &symbol_map[ohlcv]; println!( "{:<32}{:>13}{:>11.4}{:>11.4}{:>11.4}{:>11.4}{:>8} {symbol}", Ts(ohlcv.hd.ts_event), ohlcv.hd.instrument_id, Px(ohlcv.open), Px(ohlcv.high), Px(ohlcv.low), Px(ohlcv.close), ohlcv.volume ); } } Ok(()) } ``` ## Result ```plaintext ts_event instrument_id open high low close volume symbol 2025-06-29T22:00:00.000000000Z 14160 6223.2500 6228.2500 6223.2500 6224.7500 282 ESU5 2025-06-29T22:00:00.000000000Z 48059 63.7500 63.7800 63.7000 63.7100 9 CLU5 2025-06-29T22:00:00.000000000Z 209884 65.1500 65.1800 65.0800 65.1000 44 CLQ5 2025-06-29T22:00:01.000000000Z 14160 6224.5000 6225.5000 6224.5000 6225.0000 48 ESU5 2025-06-29T22:00:01.000000000Z 209884 65.0900 65.0900 65.0900 65.0900 1 CLQ5 2025-06-29T22:00:01.000000000Z 48059 63.6800 63.6800 63.6700 63.6700 3 CLU5 2025-06-29T22:00:02.000000000Z 14160 6224.5000 6225.5000 6224.2500 6224.7500 24 ESU5 2025-06-29T22:00:02.000000000Z 209884 65.0900 65.0900 65.0600 65.0900 14 CLQ5 2025-06-29T22:00:03.000000000Z 14160 6224.5000 6225.7500 6224.5000 6225.7500 24 ESU5 2025-06-29T22:00:03.000000000Z 209884 65.1000 65.1200 65.0700 65.0700 3 CLQ5 ``` # Estimate Databento feed latency ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to calculate latency of incoming records. We will use the `ts_recv` field to make a measurement of the time between Databento's servers receiving a message from the exchange, to that message being received by the client. To do this we will request order events for a few symbols using [parent symbology](https://databento.com/docs/conventions/symbology.md#parent) and calculate the latency for each message. You can read more about Databento's timestamps in this [article](https://databento.com/docs/architecture/timestamping-guide.md). ## MBO schema We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) for this example. This schema provides records for every individual order event. ## Example ```cpp #include // sort #include // nanoseconds #include // UnixNanos #include // LiveBuilder #include // Dataset #include // MboMsg #include // kAllSymbols #include namespace db = databento; int main() { auto client = db::LiveBlocking::Builder() .SetDataset(db::Dataset::GlbxMdp3) .SetKeyFromEnv() .BuildBlocking(); // First we subscribe to all orders client.Subscribe({"ES.FUT"}, db::Schema::Mbo, db::SType::Parent); client.Start(); // Then, we iterate over the records std::vector samples; constexpr size_t num_samples = 1000; while (samples.size() < num_samples) { auto record = client.NextRecord(); if (const auto* mbo = record.GetIf()) { auto delta = db::UnixNanos::clock::now() - mbo->ts_recv; samples.emplace_back(delta); } } // Finally, we calculate the latency percentiles std::sort(samples.begin(), samples.end()); for (auto percentile : {0, 50, 90, 99}) { auto sample = samples.at(percentile * samples.size() / 100); std::chrono::duration millis{sample}; std::cout << percentile << "p " << millis.count() << "ms\n"; } return 0; } ``` ## Result Latency results will depend upon the quality of your system clock and internet connection. Your individual results might be very large or even negative given these factors. ```plaintext 0p 4.49448ms 50p 5.66887ms 90p 10.7711ms 99p 47.5738ms ``` # Estimate Databento feed latency ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to calculate latency of incoming records. We will use the `ts_recv` field to make a measurement of the time between Databento's servers receiving a message from the exchange, to the message being received by the client. To do this, we will request [MBP-1](https://databento.com/docs/schemas/mbp-1.md) data for a [parent symbol](https://databento.com/docs/conventions/symbology.md#parent) and calculate the latency for each message. You can read our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md) to learn more about our architecture. Latency results will depend upon the quality of your system clock and internet connection. Your individual results might be very large or even negative given these factors. To obtain the most accurate results, we suggest you sync your clocks to our [NTP service](https://databento.com/docs/architecture/ntp-service.md). ## Example ```python import time import databento as db import matplotlib.pyplot as plt import numpy as np # Set parameters dataset = "GLBX.MDP3" product = "ES" num_samples = 5000 percentiles = [50, 90, 95, 99] num_bins = 100 # Create a live client live_client = db.Live(key="$YOUR_API_KEY") # Subscribe to MBP-1 data live_client.subscribe( dataset="GLBX.MDP3", schema="mbp-1", symbols=f"{product}.FUT", stype_in="parent", ) # Iterate over the live client and calculate latency latency_ms = [] for record in live_client: if isinstance(record, db.MBP1Msg): delta = time.time_ns() - record.ts_recv latency_ms.append(delta / 1e6) if len(latency_ms) >= num_samples: break fig, ax = plt.subplots(figsize=(10, 6)) x_min = min(latency_ms) * 0.95 x_max = max(latency_ms) * 1.05 # Create histogram with log bins log_bins = np.logspace(np.log10(x_min), np.log10(x_max), num_bins) n, bins, patches = ax.hist(latency_ms, bins=log_bins, rwidth=0.85, color="C3") y_max = max(n) * 1.25 percentile_handles = [] percentile_labels = [] # Create percentile lines for p in percentiles: p_value = np.percentile(latency_ms, p) line = ax.axvline(p_value, color="C0", linestyle="--", linewidth=1) ax.text(p_value, y_max * 0.9, f"{p}th", color="C0", rotation=-90) percentile_handles.append(line) percentile_labels.append(f"p{p} - {round(p_value, 2)} ms") # Set log scale and labels tick_locations = np.logspace(np.log10(x_min), np.log10(x_max), 10) ax.set_xscale("log") ax.set_xticks(tick_locations) ax.set_xticklabels([str(round(x)) for x in tick_locations]) ax.tick_params(axis="x", which="minor", bottom=False, labelbottom=False) # Set limits and labels ax.set_xlim(x_min * 0.95, x_max * 1.05) ax.set_ylim(0, y_max) ax.set_xlabel("Latency (ms)", fontsize=12) ax.set_ylabel("Frequency", fontsize=12) ax.set_title("Latency Distribution", fontsize=14) # Add legend and display ax.legend(handles=percentile_handles, labels=percentile_labels) plt.tight_layout() plt.show() ``` ![Latency plot](https://databento.com/docs/live-latency-0.svg.md) # Estimate Databento feed latency ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to calculate latency of incoming records. We will use the `ts_recv` field to make a measurement of the time between Databento's servers receiving a message from the exchange, to that message being received by the client. To do this we will request order events for a few symbols using [parent symbology](https://databento.com/docs/conventions/symbology.md#parent) and calculate the latency for each message. You can read more about Databento's timestamps in this [article](https://databento.com/docs/architecture/timestamping-guide.md). ## MBO schema We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) for this example. This schema provides records for every individual order event. ## Example ```rust use databento::{ LiveClient, dbn::{MboMsg, SType, Schema}, live::Subscription, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = LiveClient::builder() .key_from_env()? .dataset("GLBX.MDP3") .build() .await?; // First we subscribe to all orders client .subscribe( Subscription::builder() .schema(Schema::Mbo) .symbols("ES.FUT") .stype_in(SType::Parent) .build(), ) .await?; client.start().await?; // Then, we iterate over the records const NUM_SAMPLES: usize = 1000; let mut samples = Vec::new(); while let Some(record) = client.next_record().await? { if let Some(mbo) = record.get::() { let delta = time::OffsetDateTime::now_utc() - mbo.ts_recv().unwrap(); samples.push(delta); if samples.len() >= NUM_SAMPLES { break; } } } // Finally, we calculate the latency percentiles samples.sort(); for percentile in [0, 50, 90, 99] { let sample = &samples[percentile * samples.len() / 100]; let latency_ms = sample.whole_milliseconds(); let latency_ns = sample.subsec_nanoseconds() % 1_000_000; println!("{percentile:02}p {latency_ms:3}.{latency_ns:06}ms"); } Ok(()) } ``` ## Result Latency results will depend upon the quality of your system clock and internet connection. Your individual results might be very large or even negative given these factors. ```plaintext 00p 4.658912ms 50p 5.818663ms 90p 8.842160ms 99p 37.491661ms ``` # Calculate TICK and TRIN indicators ## Overview In this example we will use the [LiveThreaded client](https://databento.com/docs/api-live/client/live-threaded.md) to calculate two technical indicators: market tick (TICK) and the traders index (TRIN). To do this we will request trades for all symbols and calculate these indicators for every execution. ## Trades schema These indicators are updated on every execution for all symbols on the venue. To get this, we will use the [trades schema](https://databento.com/docs/schemas/trades.md), which provides records for every trade execution. ## Example ```cpp #include #include // dataset #include // LiveBuilder #include // TradeMsg #include // kAllSymbols #include #include namespace db = databento; class TickTrinCalculator { public: TickTrinCalculator() = default; db::KeepGoing HandleRecord(const db::Record& record) { if (const auto* trade = record.GetIf()) { if (trade->side == db::Side::Ask) { ++downtick_; downvol_ += trade->size; } else if (trade->side == db::Side::Bid) { ++uptick_; upvol_ += trade->size; } double tick = static_cast(uptick_) / static_cast(downtick_); double trin_base = static_cast(upvol_) / static_cast(downvol_); double trin = tick / trin_base; // Only print if one of the indicators is finite if (std::isfinite(tick) || std::isfinite(trin)) { std::cout << std::fixed << std::setprecision(4) << tick << ", " << trin << '\n'; ++measurements_count_; if (measurements_count_ == 20) { return db::KeepGoing::Stop; } } } return db::KeepGoing::Continue; } private: std::size_t measurements_count_{}; std::uint32_t uptick_{}; std::uint32_t downtick_{}; std::uint32_t upvol_{}; std::uint32_t downvol_{}; }; int main() { TickTrinCalculator calculator; auto client = db::LiveThreaded::Builder() .SetKey("$YOUR_API_KEY") .SetDataset(db::Dataset::GlbxMdp3) .BuildThreaded(); client.Subscribe({"CL.c.0"}, db::Schema::Trades, db::SType::Continuous); client.Start([&calculator](const db::Record& record) { return calculator.HandleRecord(record); }); // Wait for callback to return `Stop` client.BlockForStop(); return 0; } ``` ## Result ```plaintext 1.0000, 0.5000 2.0000, 0.6667 3.0000, 0.7500 4.0000, 0.6667 5.0000, 0.7143 2.5000, 1.0714 3.0000, 1.1250 3.5000, 1.1667 4.0000, 1.2000 4.5000, 1.2273 5.0000, 1.0000 5.0000, 1.0000 5.5000, 1.0312 6.0000, 1.0000 4.0000, 1.1111 3.0000, 1.5000 2.4000, 1.4667 2.6000, 1.5053 2.6000, 1.5053 2.6000, 1.5053 ``` # Calculate TICK and TRIN indicators ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to calculate two technical indicators: market tick (TICK) and the traders index (TRIN). ## Trades schema These indicators are updated on every execution for all symbols on the venue. To get this, we will use the [trades schema](https://databento.com/docs/schemas/trades.md), which provides records for every trade execution. ## Example We can pass in `symbols="ALL_SYMBOLS"` to request all symbols of a dataset at once. ```python import databento as db import numpy as np # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to trades for all symbols live_client.subscribe( dataset="GLBX.MDP3", schema="trades", symbols="ALL_SYMBOLS", ) # Then, we setup our initial values for TICK/TRIN uptick = downtick = upvol = downvol = 0 # We will consume 20 records and then close for this example trades = 0 for record in live_client: if isinstance(record, db.TradeMsg): side = record.side if side == "A": downtick += 1 downvol += record.size elif side == "B": uptick += 1 upvol += record.size tick = uptick / downtick if downtick != 0 else np.nan trin_base = upvol / downvol if downvol != 0 else np.nan trin = tick / trin_base if trin_base != 0 else np.nan print(f"{tick:.4f}", f"{trin:.4f}") trades += 1 if trades >= 20: break elif isinstance(record, db.ErrorMsg): print(f"Error: {record.err}") ``` ## Result ```plaintext 1.0000 1.0000 0.5000 1.0000 0.3333 1.0000 0.2500 1.0000 0.2000 2.2000 0.1667 2.0000 0.1429 1.8571 0.1250 1.8750 0.1111 2.1111 0.2222 2.1111 0.3333 1.5833 0.4444 1.0556 0.4000 1.0000 0.5000 1.0000 0.6000 1.0909 0.5455 1.1405 0.5000 1.0909 0.4615 1.0909 0.4286 1.0909 ``` # Calculate TICK and TRIN indicators ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live-threaded.md) to calculate two technical indicators: market tick (TICK) and the traders index (TRIN). To do this we will request trades for all symbols and calculate these indicators for every execution. ## Trades schema These indicators are updated on every execution for all symbols on the venue. To get this, we will use the [trades schema](https://databento.com/docs/schemas/trades.md), which provides records for every trade execution. ## Example ```rust use databento::{ LiveClient, Symbols, dbn::{Dataset, Schema, Side, TradeMsg}, live::Subscription, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = LiveClient::builder() .key_from_env()? .dataset(Dataset::GlbxMdp3) .build() .await?; // First, we subscribe to trades for all symbols client .subscribe( Subscription::builder() .schema(Schema::Trades) .symbols(Symbols::All) .build(), ) .await?; client.start().await?; // Next, we setup our initial values for TICK and TRIN let mut measurements_count = 0; let mut uptick = 0.0; let mut downtick = 0.0; let mut upvol = 0.0; let mut downvol = 0.0; // Then, we consume 20 records calculating as we go while let Some(record) = client.next_record().await? { if let Some(trade) = record.get::() { if trade.side()? == Side::Ask { downtick += 1.0; downvol += trade.size as f64; } else if trade.side()? == Side::Bid { uptick += 1.0; upvol += trade.size as f64; } let tick = uptick / downtick; let trin_base = upvol / downvol; let trin = tick / trin_base; if tick.is_finite() || trin.is_finite() { println!("{tick:.4}, {trin:.4}"); measurements_count += 1; if measurements_count == 20 { break; } } } } // Finally, we close the connection client.close().await?; Ok(()) } ``` ## Result ```plaintext 0.0000, NaN 0.0000, NaN 0.0000, NaN 0.0000, NaN 0.0000, NaN 0.0000, NaN 0.1667, 1.0000 0.3333, 0.5000 0.5000, 0.6667 0.4286, 0.6190 0.3750, 0.6250 0.3333, 0.6667 0.3000, 0.7000 0.2727, 0.7273 0.2500, 0.8611 0.3333, 0.4697 0.3077, 0.4476 0.3846, 0.5351 0.4615, 0.3992 0.5385, 0.4534 ``` # Subscribe to MBO snapshot ## Overview This example shows how to use the [Live client](https://databento.com/docs/api-live/client/live.md) to request a snapshot for [MBO schema](https://databento.com/docs/schemas/mbo.md), which can be used to reconstruct the [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) without replaying the entire trading session. More details about live snapshot can be found in this [article](https://databento.com/docs/conventions/mbo-snapshot.md). ## Requesting live snapshot To request a subscription with snapshot, we use [SubscribeWithSnapshot](https://databento.com/docs/api-live/client/live-blocking-subscribe-with-snapshot.md) method from the live client. Once the session has started, it streams all the snapshot records (with the `F_SNAPSHOT` [flag](https://databento.com/docs/conventions/common-fields-enums-types.md#flags) set), followed by real-time records. We stop iterating over the records when encountering a record with the last flag set, to make sure the book is in a valid state. ## Example ```cpp #include // dataset #include // LiveBuilder #include // MboMsg #include namespace db = databento; int main() { // First, create a live client and connect auto client = db::LiveBlocking::Builder() .SetKey("$YOUR_API_KEY") .SetDataset(db::Dataset::GlbxMdp3) .BuildBlocking(); // Next, we will subscribe to the MBO snapshot and start the session client.SubscribeWithSnapshot({"ES.c.0"}, db::Schema::Mbo, db::SType::Continuous); client.Start(); constexpr auto timeout = std::chrono::seconds{5}; // Finally, we will process all snapshot records, and stop at the first record // with F_LAST flag, which indicates that the snapshot is complete and the // order book is in a valid state while (auto* record = client.NextRecord(timeout)) { if (auto* msg = record->GetIf()) { if (msg->flags.IsSnapshot()) { std::cout << "Snapshot: " << *msg << '\n'; } else { std::cout << "Live " << *msg << '\n'; } if (msg->flags.IsLast()) { std::cout << "Snapshot is complete" << '\n'; break; } } } return 0; } ``` ## Result ```plaintext Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-23T10:06:12.163585630Z }, order_id = 0, price = kUndefPrice, size = 0, flags = SNAPSHOT | BAD_TS_RECV (40), channel_id = 0, action = Clear, side = None, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 0 } Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-21T16:28:16.070204729Z }, order_id = 6413514530616, price = 5547.500000000, size = 8, flags = SNAPSHOT | BAD_TS_RECV (40), channel_id = 0, action = Add, side = Bid, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 1022 } Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-21T16:28:16.070204729Z }, order_id = 6413514530330, price = 5547.250000000, size = 7, flags = SNAPSHOT | BAD_TS_RECV (40), channel_id = 0, action = Add, side = Bid, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 1022 } Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-21T16:28:16.070204729Z }, order_id = 6413514530315, price = 5547.000000000, size = 9, flags = SNAPSHOT | BAD_TS_RECV (40), channel_id = 0, action = Add, side = Bid, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 1022 } Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-21T16:28:16.070204729Z }, order_id = 6413514528530, price = 5546.750000000, size = 8, flags = SNAPSHOT | BAD_TS_RECV (40), channel_id = 0, action = Add, side = Bid, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 1021 } Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-21T16:28:16.070204729Z }, order_id = 6413514517321, price = 5546.500000000, size = 7, flags = SNAPSHOT | BAD_TS_RECV (40), channel_id = 0, action = Add, side = Bid, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 1020 } ... Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-23T10:06:10.357094773Z }, order_id = 6413522250995, price = 5605.250000000, size = 3, flags = SNAPSHOT | BAD_TS_RECV (42), channel_id = 0, action = Add, side = Bid, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 12967152 } Snapshot: MboMsg { hd = RecordHeader { length = 14, rtype = Mbo, publisher_id = 1, instrument_id = 118, ts_event = 2024-07-23T10:06:10.493868863Z }, order_id = 6413522250996, price = 5606.250000000, size = 1, flags = LAST | SNAPSHOT | BAD_TS_RECV (170), channel_id = 0, action = Add, side = Bid, ts_recv = 2024-07-23T10:06:12.163585630Z, ts_in_delta = 0, sequence = 12967153 } Snapshot is complete ``` # Subscribe to MBO snapshot ## Overview This example shows how to use the [Live client](https://databento.com/docs/api-live/client/live.md) to request a snapshot for [MBO schema](https://databento.com/docs/schemas/mbo.md), which can be used to reconstruct the [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) without replaying the entire trading session. More details about live snapshot can be found in this [article](https://databento.com/docs/conventions/mbo-snapshot.md). ## Requesting live snapshot To request a subscription with snapshot, we provide `snapshot=True` and leave `start` default to `None`. Once the session has started, it streams all the snapshot records (with the `F_SNAPSHOT` [flag](https://databento.com/docs/conventions/common-fields-enums-types.md#flags) set), followed by real-time records. We stop iterating over the records when encountering a record with the last flag set, to make sure the book is in a valid state. ## Example ```python import databento as db # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the MBO snapshot live_client.subscribe( dataset="GLBX.MDP3", schema="mbo", stype_in="continuous", symbols="ES.c.0", snapshot=True, ) # Finally, we will process all snapshot records, and stop at the first record # with F_LAST flag, which indicates that the snapshot is complete and the # order book is in a valid state for record in live_client: if isinstance(record, db.MBOMsg): if record.flags & db.RecordFlags.F_SNAPSHOT: print(f"Snapshot: {record}") else: print(f"Live: {record}") if record.flags & db.RecordFlags.F_LAST: print("Snapshot is complete") break elif isinstance(record, db.ErrorMsg): print(f"Error {record.err}") break ``` ## Result ```plaintext Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1769043341317912993, pretty_ts_event='2026-01-22T00:55:41.317912993Z', action='R', side='N', price=9223372036854775807, pretty_price=NaN, size=0, channel_id=0, order_id=0, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=0) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6811250000000, pretty_price=6811.25, size=1, channel_id=0, order_id=6416171004846, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=761) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6810000000000, pretty_price=6810, size=2, channel_id=0, order_id=6416138613427, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6805000000000, pretty_price=6805, size=2, channel_id=0, order_id=6416108605777, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=756) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=50, channel_id=0, order_id=6416091817328, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=754) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=10, channel_id=0, order_id=6416102672407, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=755) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=5, channel_id=0, order_id=6416117277784, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=756) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6800000000000, pretty_price=6800, size=20, channel_id=0, order_id=6416141452066, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=758) ... Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6795000000000, pretty_price=6795, size=25, channel_id=0, order_id=6416122261431, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6791000000000, pretty_price=6791, size=1, channel_id=0, order_id=6416120339416, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6780500000000, pretty_price=6780.5, size=1, channel_id=0, order_id=6416121479238, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) Snapshot: MboMsg(ts_recv=1769043341317912993, pretty_ts_recv='2026-01-22T00:55:41.317912993Z', rtype=, publisher_id=GLBX.MDP3.GLBX (1), instrument_id=42140878, ts_event=1768746891991522549, pretty_ts_event='2026-01-18T14:34:51.991522549Z', action='A', side='B', price=6775000000000, pretty_price=6775, size=1, channel_id=0, order_id=6416127400070, flags=SNAPSHOT | BAD_TS_RECV (40), ts_in_delta=0, sequence=757) ``` # Subscribe to MBO snapshot ## Overview This example shows how to use the [Live client](https://databento.com/docs/api-live/client/live.md) to request a snapshot for [MBO schema](https://databento.com/docs/schemas/mbo.md), which can be used to reconstruct the [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) without replaying the entire trading session. More details about live snapshot can be found in this [article](https://databento.com/docs/conventions/mbo-snapshot.md). ## Requesting live snapshot To request a subscription with snapshot, we call `use_snapshot(true)` method on the subscription builder and leave `start` unset. Once the session has started, it streams all the snapshot records (with the `F_SNAPSHOT` [flag](https://databento.com/docs/conventions/common-fields-enums-types.md#flags) set), followed by real-time records. We stop iterating over the records when encountering a record with the last flag set, to make sure the book is in a valid state. ## Example ```rust use databento::{ LiveClient, dbn::{Dataset, MboMsg, SType, Schema}, live::Subscription, }; #[tokio::main] async fn main() -> anyhow::Result<()> { // First, create a live client and connect let mut client = LiveClient::builder() .key_from_env()? .dataset(Dataset::GlbxMdp3) .build() .await?; // Next, we will subscribe to the MBO snapshot and start the session client .subscribe( Subscription::builder() .schema(Schema::Mbo) .symbols("ES.c.0") .stype_in(SType::Continuous) .use_snapshot(true) .build(), ) .await?; client.start().await?; // Then, we will process all snapshot records, and stop at the first record // with F_LAST flag, which indicates that the snapshot is complete and the // order book is in a valid state while let Some(record) = client.next_record().await? { if let Some(mbo) = record.get::() { if mbo.flags.is_snapshot() { println!("Snapshot: {mbo:?}"); } else { println!("Live: {mbo:?}"); } if mbo.flags.is_last() { println!("Snapshot is complete"); break; } } } // Finally, we close the connection client.close().await?; Ok(()) } ``` ## Result ```plaintext Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721730014428582820 }, order_id: 0, price: UNDEF_PRICE, size: 0, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'R', side: 'N', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 0 } Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514530616, price: 5547.500000000, size: 8, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 1022 } Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514530330, price: 5547.250000000, size: 7, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 1022 } Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514530315, price: 5547.000000000, size: 9, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 1022 } Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514528530, price: 5546.750000000, size: 8, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 1021 } Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721579296070204729 }, order_id: 6413514517321, price: 5546.500000000, size: 7, flags: SNAPSHOT | BAD_TS_RECV (40), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 1020 } ... Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721730013056310351 }, order_id: 6413522263444, price: 5607.250000000, size: 1, flags: SNAPSHOT | BAD_TS_RECV (42), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 12990544 } Snapshot: MboMsg { hd: RecordHeader { length: 14, rtype: Mbo, publisher_id: GlbxMdp3Glbx, instrument_id: 118, ts_event: 1721730013859038005 }, order_id: 6413522263447, price: 5607.500000000, size: 1, flags: LAST | SNAPSHOT | BAD_TS_RECV (170), channel_id: 0, action: 'A', side: 'B', ts_recv: 1721730014428582820, ts_in_delta: 0, sequence: 12990552 } Snapshot is complete ``` # Compare on-exchange and off-exchange trade volume ## Overview In this example, we will stream live data and compare the volume of trades on the different venues that compose the [Nasdaq Basic with NLS Plus](https://databento.com/docs/datasets/xnas-basic.md) (`XNAS.BASIC`) dataset. The following venues are included in this dataset: - Nasdaq (XNAS) - Nasdaq Texas (XBOS) - Nasdaq PSX (XPSX) - FINRA/Nasdaq TRF Carteret (FINN) - FINRA/Nasdaq TRF Chicago (FINC) The [FINRA/Nasdaq TRFs](https://www.nasdaq.com/solutions/post-trade-reporting) aggregate data from most of the 30 ATSs and account for approximately 45% to 49% of the average daily volume (ADV) in all exchange-listed securities. As of January 2025, `XNAS.BASIC` represented about 63% of ADV, which includes both on-exchange and off-exchange trades. ## Trades schema We will use the [trades schema](https://databento.com/docs/schemas/trades.md) for this example. [NLS Plus](https://data.nasdaq.com/databases/NLS) reports the size, price, and exchange/TRF for every trade. The trade aggressor side is not available on this dataset, so the `side` field will always be set to `N`. The `side` field is populated on our depth-of-book feeds, such as [Nasdaq TotalView-ITCH](https://databento.com/docs/datasets/xnas-itch.md) (`XNAS.ITCH`). > **Info** > > Real-time `XNAS.BASIC` data is included with a Plus or Unlimited subscription through our Databento US Equities service. > Visit our [pricing page]($PLATFORM_HOSTNAME/pricing/) for more information. ## Example ```python import databento as db # First, create the respective clients historical_client = db.Historical(key="$YOUR_API_KEY") live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to the trades schema for the designated symbol symbol = "TSLA" live_client.subscribe( dataset="XNAS.BASIC", schema="trades", symbols=symbol, ) # We will stream live data to file for further use live_client.add_stream(f"{symbol}_trades.dbn") live_client.start() # Let the connection run for 15 seconds. All trades in this time will be saved # to the file above live_client.block_for_close(timeout=15) # Now we will open the file for our analysis and create a Pandas DataFrame dbn_store = db.DBNStore.from_file(f"{symbol}_trades.dbn") df = dbn_store.to_df(schema="trades") # Calculate statistics by venue venue_df = df.groupby("publisher_id", as_index=False)["size"].agg(["sum", "mean"]) venue_df["mean"] = venue_df["mean"].round().astype(int) total_volume = venue_df["sum"].sum() venue_df["percentage"] = (venue_df["sum"] / total_volume * 100).round(2) # Clean up for readability publishers = historical_client.metadata.list_publishers() publisher_venues = {x["publisher_id"]: x["description"].partition("-")[2] for x in publishers} venue_df["publisher_id"] = venue_df["publisher_id"].map(publisher_venues) venue_df = venue_df.sort_values("sum", ascending=False) venue_df.columns = ["Venue", "Total Volume", "Average Size", "% of Dataset Total"] print(venue_df) ``` ## Result > **Info** > > If you do not see any output, it could be because the markets are closed. See the `start` parameter in [Live.subscribe](https://databento.com/docs/api-live/client/subscribe.md) for using [intraday replay](https://databento.com/docs/api-live/basics/intraday-replay.md). ```plaintext Venue Total Volume Average Size % of Dataset Total 1 FINRA/Nasdaq TRF Carteret 86591 56 68.93 0 Nasdaq 36218 48 28.83 2 FINRA/Nasdaq TRF Chicago 2408 51 1.92 3 Nasdaq BX 348 58 0.28 4 Nasdaq PSX 57 11 0.05 ``` ## Historical data | Title | Subtitle | Supported languages | | [Programmatic batch downloads](https://databento.com/docs/examples/basics-historical/programmatic-batch-download.md) | Request and download batch files programmatically. | Python, C++, Rust, Historical | | [Best bid, best offer, and midprice](https://databento.com/docs/examples/basics-historical/midprice.md) | Calculate midprice using BBO from MBP-1 or TBBO data. | Python, C++, Rust, Historical | | [Custom OHLCV bars from trades](https://databento.com/docs/examples/basics-historical/custom-ohlcv.md) | Resample trades data to construct custom OHLCV bars. | Python, Rust, Historical | | [Join schemas on instrument ID](https://databento.com/docs/examples/basics-historical/joining-schemas.md) | Augment price data with instrument definitions by joining on instrument ID. | Python, Historical | | [Plot a candlestick chart](https://databento.com/docs/examples/basics-historical/candlestick-plot.md) | Visualize OHLCV data on a candlestick chart. | Python, Historical | | [Calculate VWAP and RSI](https://databento.com/docs/examples/basics-historical/technical-indicators.md) | Compute the volume-weighted average price and relative strength indicator. | Python, Historical | | [End-of-day pricing and portfolio valuation](https://databento.com/docs/examples/basics-historical/eod.md) | Get mark-to-market valuation of a portfolio at end of day. | Python, C++, Rust, Historical | | [Benchmark portfolio performance](https://databento.com/docs/examples/basics-historical/benchmarks.md) | Calculate metrics like risk-adjusted return and Sharpe ratio of a portfolio. | Python, Historical | | [Market halts, volatility interrupts, and price bands](https://databento.com/docs/examples/basics-historical/halts.md) | Handle trading halts using status messages. | Python, Historical | | [Resample OHLCV from 1-minute to 5-minute](https://databento.com/docs/examples/basics-historical/ohlcv-resampling.md) | Forward-fill gaps from intervals with no trades, then resample to 5-minute bars. | Python, Historical | | [Convert DBN to other encoding formats](https://databento.com/docs/examples/basics-historical/encodings.md) | Request data in DBN format and convert to CSV, JSON, or Parquet. | Python, Historical | | [Request a large number of symbols](https://databento.com/docs/examples/basics-historical/requesting.md) | Get multiple symbols or a firehose of all symbols traded on a venue. | Python, Rust, Historical | # Programmatic batch downloads ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to submit a batch request and download files programmatically. To do this we will request five days of trade data, wait for our request to process, and then download the files locally and load them with a [DbnStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. > **See also** > > [Streaming vs. batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md). ## Example ```cpp #include #include #include #include #include #include namespace db = databento; int main() { auto client = db::Historical::Builder().SetKey("$YOUR_API_KEY").Build(); db::BatchJob new_job = client.BatchSubmitJob( db::dataset::kGlbxMdp3, {"OZN.OPT"}, db::Schema::Trades, {"2022-12-12T00:00", "2022-12-12T08:00"}, {}, db::Compression::Zstd, {}, {}, {}, {}, db::SplitDuration::Day, {}, db::Delivery::Download, db::SType::Parent, db::SType::InstrumentId, {}); while (true) { db::BatchJob job_details = client.BatchGetJobDetails(new_job.id); if (job_details.state == db::JobState::Done) { break; } std::this_thread::sleep_for(std::chrono::seconds{1}); } auto downloaded_files = client.BatchDownload(".", new_job.id); for (const auto& file : downloaded_files) { if (file.stem().extension() == ".dbn") { db::DbnStore dbn_store{file}; std::size_t count = 0; dbn_store.Replay([&count](const db::Record&) { ++count; return db::KeepGoing::Continue; }); std::cout << file << " contains " << count << " records\n"; } } return 0; } ``` ## Result ```plaintext ./GLBX-20250116-9YULLS4UJH/glbx-mdp3-20221212.trades.dbn.zst contains 1662 records ./GLBX-20250116-9YULLS4UJH/glbx-mdp3-20221213.trades.dbn.zst contains 3511 records ./GLBX-20250116-9YULLS4UJH/glbx-mdp3-20221214.trades.dbn.zst contains 2544 records ./GLBX-20250116-9YULLS4UJH/glbx-mdp3-20221215.trades.dbn.zst contains 2460 records ./GLBX-20250116-9YULLS4UJH/glbx-mdp3-20221216.trades.dbn.zst contains 2191 records ``` # Programmatic batch downloads ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to submit a batch request and download files programmatically. To do this we will request five days of trade data, wait for our request to process, and then download the files locally and load them into a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) object. > **See also** > > [Streaming vs. batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md). ## Example ```python import pathlib import time import databento as db # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will submit a batch job new_job = client.batch.submit_job( dataset="GLBX.MDP3", start="2022-12-12T00:00:00", end="2022-12-17T00:00:00", symbols="OZN.OPT", schema="trades", split_duration="day", stype_in="parent", ) # Retrieve the new job ID new_job_id: str = new_job["id"] # Now, we have to wait for our batch job to complete while True: job_details = client.batch.get_job_details(new_job_id) if job_details["state"] == "done": break # Exit the loop to continue time.sleep(1.0) # Once complete, we will download the files downloaded_files = client.batch.download( job_id=new_job_id, output_dir=pathlib.Path.cwd(), ) # Finally, we can load the data into a DBNStore for analysis for file in sorted(downloaded_files): if file.name.endswith(".dbn.zst"): data = db.DBNStore.from_file(file) # Convert the data to a pandas.DataFrame df = data.to_df() print(f"{file.name} contains {len(df):,d} records") ``` ## Result ```plaintext glbx-mdp3-20221212.trades.dbn.zst contains 2,055 records glbx-mdp3-20221213.trades.dbn.zst contains 4,112 records glbx-mdp3-20221214.trades.dbn.zst contains 2,936 records glbx-mdp3-20221215.trades.dbn.zst contains 3,034 records glbx-mdp3-20221216.trades.dbn.zst contains 2,540 records ``` # Programmatic batch downloads ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to submit a batch request and download files programmatically. To do this we will request five days of trade data, wait for our request to process, and then download the files locally and load them with a `AsyncDbnDecoder` object. > **See also** > > [Streaming vs. batch download](https://databento.com/docs/faqs/streaming-vs-batch-download.md). ## Example ```rust use std::time::Duration; use databento::{ HistoricalClient, dbn::{Dataset::GlbxMdp3, SType, Schema, decode::AsyncDynReader}, historical::{ batch::{DownloadParams, JobState, SubmitJobParams}, timeseries::AsyncDbnDecoder, }, }; use time::macros::datetime; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let job = client .batch() .submit_job( &SubmitJobParams::builder() .dataset(GlbxMdp3) .date_time_range(datetime!(2022-10-12 00:00 UTC)..datetime!(2022-10-17 00:00 UTC)) .symbols("OZN.OPT") .stype_in(SType::Parent) .schema(Schema::Trades) .build(), ) .await?; // Now we wait for the job to complete loop { let job_details = client.batch().get_job_details(&job.id).await?; if job_details.state == JobState::Done { break; } tokio::time::sleep(Duration::from_secs(1)).await; } // Once complete, we download the files let mut downloaded_files = client .batch() .download( &DownloadParams::builder() .output_dir(std::env::current_dir()?) .job_id(job.id) .build(), ) .await?; downloaded_files.sort(); for downloaded_file in downloaded_files { let Some(file_name) = downloaded_file.file_name().and_then(|name| name.to_str()) else { continue; }; if file_name.ends_with(".dbn") || file_name.ends_with(".dbn.zst") { // `DynReader` infers the compression (zstd or uncompressed) from the file let mut decoder = AsyncDbnDecoder::new(AsyncDynReader::from_file(&downloaded_file).await?).await?; let mut count = 0; while decoder.decode_record_ref().await?.is_some() { count += 1; } println!("{file_name} contains {count} records"); } } Ok(()) } ``` ## Result ```plaintext glbx-mdp3-20221012.trades.dbn.zst contains 3041 records glbx-mdp3-20221013.trades.dbn.zst contains 7956 records glbx-mdp3-20221014.trades.dbn.zst contains 3956 records glbx-mdp3-20221016.trades.dbn.zst contains 54 records ``` # Best bid, best offer, and midprice In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve the best bid and offer (BBO) and use these data to calculate the midprice. To do this we will request a subset of the entire order book data and then average the bid and ask prices to calculate the midprice. Note that _ask_ and _offer_ are synonyms referring to the sell price. ## TBBO Schema We'll use the [TBBO schema](https://databento.com/docs/schemas/tbbo.md) to demonstrate this. This schema contains every trade execution event together with the best bid and offer immediately before the trade. ## Example ```cpp #include // kUndefPrice #include #include // Px #include #include namespace db = databento; int main() { auto client = db::Historical::Builder().SetKey("$YOUR_API_KEY").Build(); auto store = client.TimeseriesGetRange( "GLBX.MDP3", {"2022-10-28T20:30:00", "2022-10-28T21:00:00"}, {"ESH3"}, db::Schema::Tbbo); std::cout << std::setprecision(3); while (const auto* record = store.NextRecord()) { auto bbo = record->Get(); auto ask_px = bbo.levels[0].ask_px; auto bid_px = bbo.levels[0].bid_px; // Skip records where either side of the book is empty if (ask_px == db::kUndefPrice || bid_px == db::kUndefPrice) { continue; } auto mid_px = (ask_px + bid_px) / 2; std::cout << bbo.hd.instrument_id << '\t' << db::pretty::Px{ask_px} << '\t' << db::pretty::Px{mid_px} << '\t' << db::pretty::Px{bid_px} << '\n'; } return 0; } ``` ## Result ```plaintext 206299 3955.250 3955.000 3954.750 206299 3955.750 3955.375 3955.000 206299 3955.750 3955.375 3955.000 206299 3955.500 3955.125 3954.750 206299 3956.250 3956.125 3956.000 206299 3955.000 3954.750 3954.500 206299 3956.250 3955.500 3954.750 206299 3956.000 3954.875 3953.750 206299 3956.000 3955.250 3954.500 ``` # Best bid, best offer, and midprice ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve the best bid and offer (BBO) and use these data to calculate the midprice. To do this we will request a subset of the entire order book data and then average the bid and ask prices to calculate the midprice. Note that _ask_ and _offer_ are synonyms referring to the sell price. ## TBBO schema We'll use the [TBBO schema](https://databento.com/docs/schemas/tbbo.md) to demonstrate this. This schema contains every trade execution event together with the best bid and offer immediately before the trade. ## Example ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request TBBO data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2022-10-28T20:30:00", end="2022-10-28T21:00:00", symbols="ESH3", stype_in="raw_symbol", schema="tbbo", ) # Convert to DataFrame tbbo_data = data.to_df() # Calculate the midprice for each row tbbo_data["mid_px"] = tbbo_data[["ask_px_00", "bid_px_00"]].mean(axis=1) # We now have a DataFrame that contains execution price, BBO, and midprice print(tbbo_data[["symbol", "side", "price", "ask_px_00", "mid_px", "bid_px_00"]]) ``` ## Result ```plaintext symbol side price ask_px_00 mid_px bid_px_00 ts_recv 2022-10-28 20:30:59.047138053+00:00 ESH3 B 3955.25 3955.25 3955.000 3954.75 2022-10-28 20:37:53.112494436+00:00 ESH3 A 3955.00 3955.75 3955.375 3955.00 2022-10-28 20:39:05.152071481+00:00 ESH3 A 3955.00 3955.75 3955.375 3955.00 2022-10-28 20:51:37.650473254+00:00 ESH3 B 3955.50 3955.50 3955.125 3954.75 2022-10-28 20:52:36.842346377+00:00 ESH3 A 3956.00 3956.25 3956.125 3956.00 2022-10-28 20:52:57.780797186+00:00 ESH3 B 3955.00 3955.00 3954.750 3954.50 2022-10-28 20:57:28.474236098+00:00 ESH3 A 3954.75 3956.25 3955.500 3954.75 2022-10-28 20:59:15.075191111+00:00 ESH3 A 3953.75 3956.00 3954.875 3953.75 2022-10-28 20:59:34.607239899+00:00 ESH3 A 3954.50 3956.00 3955.250 3954.50 ``` # Best bid, best offer, and midprice In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve the best bid and offer (BBO) and use these data to calculate the midprice. To do this we will request a subset of the entire order book data and then average the bid and ask prices to calculate the midprice. Note that _ask_ and _offer_ are synonyms referring to the sell price. ## TBBO Schema We'll use the [TBBO schema](https://databento.com/docs/schemas/tbbo.md?) to demonstrate this. This schema contains every trade execution event together with the best bid and offer immediately before the trade. ## Example ```rust use databento::{ HistoricalClient, dbn::{ Dataset::GlbxMdp3, Schema, TbboMsg, UNDEF_PRICE, decode::DbnMetadata, pretty::{Px, Ts}, }, historical::timeseries::GetRangeParams, }; use time::macros::datetime; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; // First, we request the TBBO data let mut decoder = client .timeseries() .get_range( &GetRangeParams::builder() .dataset(GlbxMdp3) .date_time_range(datetime!(2022-10-28 20:30 UTC)..datetime!(2022-10-28 21:00 UTC)) .symbols("ESH3") .schema(Schema::Tbbo) .build(), ) .await?; // Next, we extract the symbol mappings for date we queried let symbol_mappings = decoder .metadata() .symbol_map_for_date(decoder.metadata().start().date())?; println!( "{:<32}{:<8}{:<4}{:>9}{:>9}{:>9}{:>9}", "ts_recv", "symbol", "side", "price", "ask", "mid", "bid" ); // Finally, we calculate the midprice for each record while let Some(tbbo) = decoder.decode_record::().await? { let (bid_px, ask_px) = (tbbo.levels[0].bid_px, tbbo.levels[0].ask_px); let mid_px = if bid_px == UNDEF_PRICE || ask_px == UNDEF_PRICE { UNDEF_PRICE } else { bid_px + (ask_px - bid_px) / 2 }; println!( "{:<32}{:<8}{:?} {:>9.2}{:>9.2}{:>9.2}{:>9.2}", Ts(tbbo.ts_recv), symbol_mappings[tbbo], tbbo.side()?, Px(tbbo.price), Px(tbbo.levels[0].ask_px), Px(mid_px), Px(tbbo.levels[0].bid_px), ); } Ok(()) } ``` ## Result ```plaintext ts_recv symbol side price ask mid bid 2022-10-28T20:30:59.047138053Z ESH3 Bid 3955.25 3955.25 3955.00 3954.75 2022-10-28T20:37:53.112494436Z ESH3 Ask 3955.00 3955.75 3955.37 3955.00 2022-10-28T20:39:05.152071481Z ESH3 Ask 3955.00 3955.75 3955.37 3955.00 2022-10-28T20:51:37.650473254Z ESH3 Bid 3955.50 3955.50 3955.12 3954.75 2022-10-28T20:52:36.842346377Z ESH3 Ask 3956.00 3956.25 3956.12 3956.00 2022-10-28T20:52:57.780797186Z ESH3 Bid 3955.00 3955.00 3954.75 3954.50 2022-10-28T20:57:28.474236098Z ESH3 Ask 3954.75 3956.25 3955.50 3954.75 2022-10-28T20:59:15.075191111Z ESH3 Ask 3953.75 3956.00 3954.87 3953.75 2022-10-28T20:59:34.607239899Z ESH3 Ask 3954.50 3956.00 3955.25 3954.50 ``` # Custom OHLCV bars from trades ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to resample trade data at a fixed interval to construct 1-minute OHLCV bars. Our OHLCV schemas are directly derived from the [Trades schema](https://databento.com/docs/schemas/trades.md). This example will repeat this derivation to demonstrate that trades data can be resampled to construct the bars in the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). This will be done using the [pandas](https://pypi.org/project/pandas/) package, specifically the [DataFrame.resample](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html) method. ## Example ```python import databento as db import pandas as pd # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" start = "2022-06-06T20:50:00" end = "2022-06-06T21:00:00" symbols = ["ESM2", "ESU2"] # Request trades data for two instruments and convert to DataFrame trades_df = client.timeseries.get_range( dataset=dataset, start=start, end=end, symbols=symbols, schema="trades", ).to_df() # Groupby symbol and resample to 1-minute bars resampled_df = ( trades_df.groupby("symbol") .resample("1min", include_groups=False) .agg( open=("price", "first"), high=("price", "max"), low=("price", "min"), close=("price", "last"), volume=("size", "sum"), ) .dropna() .reset_index() .rename(columns={"ts_recv": "ts_event"}) .sort_values(by=["ts_event", "symbol"]) .set_index("ts_event") ) # Print out resampled data print(resampled_df) # We will validate our resampled data against the OHLCV-1m schema ohlcv_data = client.timeseries.get_range( dataset=dataset, start=start, end=end, symbols=symbols, schema="ohlcv-1m", ) # Sort the DataFrame rows by ts_event and symbol ohlcv_df = ( ohlcv_data.to_df().reset_index().sort_values(by=["ts_event", "symbol"]).set_index("ts_event") ) # Validate these two DataFrames are equal pd.testing.assert_frame_equal( resampled_df, ohlcv_df[["symbol", "open", "high", "low", "close", "volume"]], check_dtype=False, # ignore dtypes ) ``` ## Result ```plaintext symbol open high low close volume ts_event 2022-06-06 20:50:00+00:00 ESM2 4120.50 4122.25 4120.50 4122.25 428 2022-06-06 20:50:00+00:00 ESU2 4123.25 4124.50 4123.25 4124.50 9 2022-06-06 20:51:00+00:00 ESM2 4122.25 4122.25 4121.50 4122.00 254 2022-06-06 20:51:00+00:00 ESU2 4124.00 4124.00 4124.00 4124.00 2 2022-06-06 20:52:00+00:00 ESM2 4122.00 4122.25 4122.00 4122.25 118 2022-06-06 20:52:00+00:00 ESU2 4124.25 4124.25 4124.25 4124.25 1 2022-06-06 20:53:00+00:00 ESM2 4122.25 4122.50 4121.75 4121.75 94 2022-06-06 20:54:00+00:00 ESM2 4121.75 4122.50 4121.75 4122.25 125 2022-06-06 20:55:00+00:00 ESM2 4122.00 4122.50 4122.00 4122.00 172 2022-06-06 20:56:00+00:00 ESM2 4122.25 4122.50 4122.00 4122.25 34 2022-06-06 20:56:00+00:00 ESU2 4124.75 4124.75 4124.75 4124.75 6 2022-06-06 20:57:00+00:00 ESM2 4122.25 4122.50 4122.00 4122.00 121 2022-06-06 20:57:00+00:00 ESU2 4124.75 4124.75 4124.75 4124.75 1 2022-06-06 20:58:00+00:00 ESM2 4122.00 4122.00 4121.25 4121.50 131 2022-06-06 20:58:00+00:00 ESU2 4124.00 4124.25 4124.00 4124.25 2 2022-06-06 20:59:00+00:00 ESM2 4121.50 4121.75 4120.50 4120.75 313 2022-06-06 20:59:00+00:00 ESU2 4123.75 4123.75 4122.75 4122.75 9 ``` # Constructing custom OHLCV bars from trades ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to resample trade data at a fixed interval to construct 1-minute OHLCV bars. Our OHLCV schemas are directly derived from the [Trades schema](https://databento.com/docs/schemas/trades.md). This example will repeat this derivation to demonstrate that trades data can be resampled to construct the bars in the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). ## Example ```rust use std::collections::BTreeMap; use databento::{ HistoricalClient, dbn::{ Dataset, OhlcvMsg, RecordHeader, Schema, SymbolIndex, TradeMsg, decode::DbnMetadata, pretty::Px, rtype, }, historical::timeseries::GetRangeParams, }; use time::{OffsetDateTime, macros::datetime}; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; // First, we will request trades data let mut params = GetRangeParams::builder() .dataset(Dataset::GlbxMdp3) .schema(Schema::Trades) .symbols(["ESM2", "ESU2"]) .date_time_range(datetime!(2022-06-06 20:50 UTC)..datetime!(2022-06-06 21:10 UTC)) .build(); let mut data = client.timeseries().get_range(¶ms).await?; let symbol_map = data.metadata().symbol_map()?; // Bars by minute timestamp and symbol let mut ohlcv = BTreeMap::new(); while let Some(trade) = data.decode_record::().await? { let symbol = symbol_map.get_for_rec(trade).unwrap(); // Round down to minute let ts_bar = trade .hd .ts_event() .unwrap() .replace_second(0)? .replace_nanosecond(0)?; ohlcv .entry((ts_bar, symbol)) .and_modify(|bar| aggregate(bar, trade)) .or_insert_with(|| bar_from_trade(ts_bar, trade)); } println!( "{:<33}{:<6}{:>9}{:>9}{:>9}{:>9}{:>8}", "ts_event", "symbol", "open", "high", "low", "close", "volume" ); for ((ts, symbol), bar) in ohlcv.iter() { println!( "{ts:<33}{symbol:<6}{:>9.2}{:>9.2}{:>9.2}{:>9.2}{:>8}", Px(bar.open), Px(bar.high), Px(bar.low), Px(bar.close), bar.volume ); } // As an additional exercise, we will validate our resampled data against the // OHLCV-1m schema params.schema = Schema::Ohlcv1M; let mut data = client.timeseries().get_range(¶ms).await?; let mut record_count = 0; while let Some(bar) = data.decode_record::().await? { let symbol = symbol_map.get_for_rec(bar).unwrap(); assert_eq!( bar, ohlcv.get(&(bar.hd.ts_event().unwrap(), symbol)).unwrap() ); record_count += 1; } assert_eq!(record_count, ohlcv.len()); Ok(()) } fn bar_from_trade(ts_bar: OffsetDateTime, trade: &TradeMsg) -> OhlcvMsg { OhlcvMsg { hd: RecordHeader::new::( rtype::OHLCV_1M, trade.hd.publisher_id, trade.hd.instrument_id, ts_bar.unix_timestamp_nanos() as u64, ), open: trade.price, high: trade.price, low: trade.price, close: trade.price, volume: trade.size as u64, } } fn aggregate(bar: &mut OhlcvMsg, trade: &TradeMsg) { bar.high = bar.high.max(trade.price); bar.low = bar.low.min(trade.price); bar.close = trade.price; bar.volume += trade.size as u64; } ``` ## Result ```plaintext ts_event symbol open high low close volume 2022-06-06 20:50:00.0 +00:00:00 ESM2 4120.50 4122.25 4120.50 4122.25 428 2022-06-06 20:50:00.0 +00:00:00 ESU2 4123.25 4124.50 4123.25 4124.50 9 2022-06-06 20:51:00.0 +00:00:00 ESM2 4122.25 4122.25 4121.50 4122.00 254 2022-06-06 20:51:00.0 +00:00:00 ESU2 4124.00 4124.00 4124.00 4124.00 2 2022-06-06 20:52:00.0 +00:00:00 ESM2 4122.00 4122.25 4122.00 4122.25 118 2022-06-06 20:52:00.0 +00:00:00 ESU2 4124.25 4124.25 4124.25 4124.25 1 2022-06-06 20:53:00.0 +00:00:00 ESM2 4122.25 4122.50 4121.75 4121.75 94 2022-06-06 20:54:00.0 +00:00:00 ESM2 4121.75 4122.50 4121.75 4122.25 125 2022-06-06 20:55:00.0 +00:00:00 ESM2 4122.00 4122.50 4122.00 4122.00 172 2022-06-06 20:56:00.0 +00:00:00 ESM2 4122.25 4122.50 4122.00 4122.25 34 2022-06-06 20:56:00.0 +00:00:00 ESU2 4124.75 4124.75 4124.75 4124.75 6 2022-06-06 20:57:00.0 +00:00:00 ESM2 4122.25 4122.50 4122.00 4122.00 121 2022-06-06 20:57:00.0 +00:00:00 ESU2 4124.75 4124.75 4124.75 4124.75 1 2022-06-06 20:58:00.0 +00:00:00 ESM2 4122.00 4122.00 4121.25 4121.50 131 2022-06-06 20:58:00.0 +00:00:00 ESU2 4124.00 4124.25 4124.00 4124.25 2 2022-06-06 20:59:00.0 +00:00:00 ESM2 4121.50 4121.75 4120.50 4120.75 313 2022-06-06 20:59:00.0 +00:00:00 ESU2 4123.75 4123.75 4122.75 4122.75 9 ``` # Join schemas on instrument ID ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to get trades and definitions data for Henry Hub natural gas futures, excluding spreads. ## Example ```python import asyncio import databento as db async def fetch_data(): client = db.Historical(key="$YOUR_API_KEY") mbp_task = client.timeseries.get_range_async( dataset="GLBX.MDP3", schema="trades", symbols=["NG.FUT"], start="2023-04-02T00:00:00", end="2023-04-05T00:00:00", stype_in="parent", ) def_task = client.timeseries.get_range_async( dataset="GLBX.MDP3", schema="definition", symbols=["NG.FUT"], start="2023-04-02T00:00:00", end="2023-04-05T00:00:00", stype_in="parent", ) return await asyncio.gather(mbp_task, def_task) # Concurrently fetch definitions and trades mbp_store, def_store = asyncio.run(fetch_data()) mbp_df = mbp_store.to_df() def_df = def_store.to_df() # Join futures with their definitions on instrument ID combined_df = mbp_df.merge( def_df, on="instrument_id", how="inner", suffixes=("", "_def"), ).set_index("ts_event") # Remove spreads combined_df = combined_df.loc[combined_df["instrument_class"] == "F", :] print(combined_df[["raw_symbol", "expiration", "price"]].head()) ``` ## Result ```plaintext raw_symbol expiration price ts_event 2023-04-02 22:00:00+00:00 NGK3 2023-04-26 18:30:00+00:00 2.085 2023-04-02 22:00:00+00:00 NGK3 2023-04-26 18:30:00+00:00 2.085 2023-04-02 22:00:00+00:00 NGK3 2023-04-26 18:30:00+00:00 2.085 2023-04-02 22:00:00+00:00 NGM3 2023-05-26 18:30:00+00:00 2.345 2023-04-02 22:00:00+00:00 NGM3 2023-05-26 18:30:00+00:00 2.345 ``` > **See also** > > [List of fields by schema](https://databento.com/docs/schemas/whats-a-schema.md). # Plot a candlestick chart ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve AAPL pre-market one-minute OHLCV bars and plot a candlestick chart. A candlestick chart graphs the opening, high, low, and closing price at regular intervals. This is a common method for graphing price movements. ## OHLCV-1m schema We'll demonstrate this example using the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. We'll use an interval of one minute, a typical charting frequency, which is specified by the suffix `-1m`. ## Dependencies This example will use the [mplfinance](https://pypi.org/project/mplfinance/) package to plot a candlestick chart. This dependency can be installed with the following: ```bash $ pip install mplfinance ``` ## Example ```python import databento as db import mplfinance as mpf # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, we will request OHLCV-1m data for AAPL pre-market data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="ohlcv-1m", symbols=["AAPL"], start="2023-08-28T08:00:00-4", end="2023-08-28T09:30:00-4", ) # Then, convert to a pandas DataFrame df = data.to_df() # Convert the index to the US/Eastern timezone df.index = df.index.tz_convert("US/Eastern") # Finally, plot the candlestick chart mpf.plot( df, type="candle", volume=True, title="Nasdaq TotalView AAPL pre-market", ylabel="OHLCV-1M Candles", ylabel_lower="Volume", xlabel="Time", ) ``` ## Result ![AAPL pre-market candlestick](https://databento.com/docs/candlestick-plot-0.png.md) # Calculate VWAP and RSI ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate two technical indicators: the volume weighted average price (VWAP) and relative strength indicator (RSI). To do this we will request the open, high, low, and close prices in one minute intervals along with aggregated volume data to calculate these indicators at one minute intervals. ## OHLCV-1m schema We'll demonstrate this example using the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. We'll use an interval of one minute, a typical charting frequency, which is specified by the suffix `-1m`. ## Example ```python import databento as db # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request the OHLCV-1m data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2022-09-20", symbols="CLV2", stype_in="raw_symbol", schema="ohlcv-1m", ) # Then, convert the DBNStore to a DataFrame ohlcv_data = data.to_df() # Now, we will calculate the volume weighted average price (VWAP) ohlcv_data["pvt"] = ohlcv_data[["open", "high", "close"]].mean(axis=1) * ohlcv_data["volume"] ohlcv_data["vwap"] = ohlcv_data["pvt"].cumsum() / ohlcv_data["volume"].cumsum() # Finally, we will calculate the relative strength indicator (RSI) ohlcv_data["up"] = (ohlcv_data["close"] - ohlcv_data["open"]).clip(lower=0) ohlcv_data["down"] = (ohlcv_data["open"] - ohlcv_data["close"]).clip(lower=0) # We will use a simple moving average for this example ohlcv_data["rs"] = ohlcv_data["up"].rolling(5).mean() / ohlcv_data["down"].rolling(5).mean() ohlcv_data["rsi"] = 100 - (100 / (1 + ohlcv_data["rs"])) # We now have a DataFrame containing our technical indicators print(ohlcv_data[["symbol", "open", "close", "vwap", "rsi"]]) ``` ## Result ```plaintext symbol open close vwap rsi ts_event 2022-09-20 00:00:00+00:00 CLV2 85.85 85.85 85.850000 NaN 2022-09-20 00:08:00+00:00 CLV2 86.05 86.01 85.938889 NaN 2022-09-20 00:10:00+00:00 CLV2 86.03 86.03 85.950278 NaN 2022-09-20 00:11:00+00:00 CLV2 86.03 85.98 85.964516 NaN 2022-09-20 00:12:00+00:00 CLV2 86.01 86.01 85.965938 0.000000 ... ... ... ... ... ... 2022-09-20 18:25:00+00:00 CLV2 84.27 84.40 84.532655 53.488372 2022-09-20 18:26:00+00:00 CLV2 84.41 84.32 84.529908 51.111111 2022-09-20 18:27:00+00:00 CLV2 84.32 84.33 84.526147 64.864865 2022-09-20 18:28:00+00:00 CLV2 84.33 84.40 84.523789 77.500000 2022-09-20 18:29:00+00:00 CLV2 84.30 84.19 84.523212 51.219512 [176 rows x 5 columns] ``` # End-of-day pricing and portfolio valuation In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate the end-of-day (EOD) value of a hypothetical portfolio. To do this we will request the closing price for each symbol on a given date and use this price to determine the value of our portfolio. ## OHLCV Schemas We'll demonstrate this example using the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. Since we are interested in end-of-day evaluation, we'll use an interval of one day, which is specified by the suffix `1D`. ## Example ```cpp #include #include #include #include #include #include #include namespace db = databento; int main() { // hypothetical portfolio mapping symbols to positions std::unordered_map portfolio = { {"AAPL", 200}, {"AMZN", 200}, {"GOOG", 100}, {"META", 100}, {"NFLX", 114}, }; db::TsSymbolMap symbol_map; auto extract_symbol_map = [&symbol_map](db::Metadata metadata) { symbol_map = metadata.CreateSymbolMap(); }; std::int64_t value = 0; auto calc_eod = [&portfolio, &symbol_map, &value](const db::Record& record) { auto bar = record.Get(); auto symbol = symbol_map.At(bar); auto position = portfolio[symbol]; value += bar.close * position; return db::KeepGoing::Continue; }; std::vector symbols; for (const auto& [symbol, _] : portfolio) { symbols.emplace_back(symbol); } auto client = db::Historical::Builder().SetKey("$YOUR_API_KEY").Build(); client.TimeseriesGetRange( "XNAS.ITCH", db::DateTimeRange{"2022-10-19"}, symbols, db::Schema::Ohlcv1D, db::SType::RawSymbol, db::SType::InstrumentId, {}, extract_symbol_map, calc_eod); std::cout << '#### Result ```plaintext $105536.78 ``` << std::setprecision(2) << db::pretty::Px{value} << '\n'; return 0; } ``` ## Result # End-of-day pricing and portfolio valuation ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate the end-of-day (EOD) value of a hypothetical portfolio. To do this we will request the closing price for each symbol on a given date and use this price to determine the value of our portfolio. ## OHLCV-1d schema We'll demonstrate this example using the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. Since we are interested in end-of-day evaluation, we'll use an interval of one day, which is specified by the suffix `-1d`. Many users will prefer to use the official daily settlement prices found in the `statistics` schema for this purpose. ## Example ```python import databento as db # A hypothetical portfolio mapping symbols to a quantity of shares portfolio = { "AAPL": 200, "AMZN": 200, "GOOG": 100, "META": 100, "NFLX": 114, } # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1d data data = client.timeseries.get_range( dataset="XNAS.ITCH", start="2022-09-19", symbols=list(portfolio.keys()), stype_in="raw_symbol", schema="ohlcv-1d", ) # Convert to DataFrame eod_data = data.to_df() eod_data = eod_data.set_index("symbol") # Sum the products of the close prices and portfolio quantities eod_evaluation = sum( eod_data.at[symbol, "close"] * quantity for symbol, quantity in portfolio.items() ) print(f"${eod_evaluation:,.2f}") ``` ## Result ```plaintext $109,235.90 ``` # End-of-day pricing and portfolio valuation In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to calculate the end-of-day (EOD) value of a hypothetical portfolio. To do this we will request the closing price for each symbol on a given date and use this price to determine the value of our portfolio. ## OHLCV Schemas We'll demonstrate this example using the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. Since we are interested in end-of-day evaluation, we'll use an interval of one day, which is specified by the suffix `1D`. ## Example ```rust use std::collections::HashMap; use databento::{ HistoricalClient, dbn::{OhlcvMsg, Schema, decode::DbnMetadata, pretty::Px}, historical::timeseries::GetRangeParams, }; use time::macros::date; #[tokio::main] async fn main() -> anyhow::Result<()> { // A hypothetical portfolio mapping symbols to a quantity of shares let portfolio = HashMap::from([ ("AAPL", 200), ("AMZN", 200), ("GOOG", 100), ("META", 100), ("NFLX", 114), ]); let mut client = HistoricalClient::builder().key_from_env()?.build()?; // First, we request the OHLCV-1D data let mut decoder = client .timeseries() .get_range( &GetRangeParams::builder() .dataset("XNAS.ITCH") .date_time_range(date!(2022 - 10 - 19)) .symbols(portfolio.keys().cloned().collect::>()) .schema(Schema::Ohlcv1D) .build(), ) .await?; // Next, we extract the symbol mappings for the date we queried let symbol_mappings = decoder .metadata() .symbol_map_for_date(date!(2022 - 10 - 19))?; // Finally, we sum the products of the close prices and portfolio quantities let mut eod = 0; while let Some(ohlcv) = decoder.decode_record::().await? { eod += portfolio[symbol_mappings[ohlcv].as_str()] * ohlcv.close; } println!("${:.2}", Px(eod)); Ok(()) } ``` ## Result ```plaintext $105536.78 ``` # Benchmark portfolio performance > **Info** > > This example builds upon concepts introduced in the [Resampling data](https://databento.com/docs/examples/basics-historical/custom-ohlcv.md) > and [End-of-day pricing and portfolio evaluation](https://databento.com/docs/examples/basics-historical/eod.md) > examples. ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to benchmark a hypothetical portfolio. To do this, we will request daily open and close price data. We will then calculate the daily returns of our portfolio for one year and benchmark against a risk-free rate of 2.5% per year, and calculate the Sharpe ratio. ## OHLCV-1d schema We'll demonstrate this example using the [OHLCV-1d schema](https://databento.com/docs/schemas/ohlcv.md). The OHLCV family of schemas contain the opening, high, low and closing prices as well as the aggregated volume of trades within a time interval. Since we are interested in end-of-day evaluation, we'll use an interval of one day, which is specified by the suffix `-1d`. Many users will prefer to use the official daily settlement prices found in the `statistics` schema for this purpose. ## Example ```python import databento as db # A hypothetical portfolio mapping symbols to a quantity of shares portfolio = { "DVN": 300, "MRO": 200, "FTNT": 100, } # Assume a risk-free rate of 2.5% per year annual_benchmark_return = 0.025 daily_benchmark_return = ((1 + annual_benchmark_return) ** (1 / 252)) - 1 # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request the ohlcv-1d data data = client.timeseries.get_range( dataset="XNAS.ITCH", start="2020-12-31T00:00:00", end="2022-01-01T00:00:00", symbols=list(portfolio.keys()), stype_in="raw_symbol", schema="ohlcv-1d", ) # Then, convert the DBNStore to a pandas DataFrame daily_data = data.to_df() # Reshape the data with one column per symbol, propagating the previous price forward for dates without trades. # Note: this does not account for corporate actions such as dividends or splits. close_prices = daily_data.pivot(columns="symbol", values="close").fillna(method="ffill") portfolio_value = sum(close_prices[sym] * qty for sym, qty in portfolio.items()) # Calculate some simple properties of the strategy cost_basis = portfolio_value.iloc[0] final_value = portfolio_value.iloc[-1] total_return = (final_value - cost_basis) / cost_basis # Calculate the arithmetic mean and standard deviation of the daily returns daily_returns = portfolio_value.pct_change().dropna() avg_return = daily_returns.mean() std_return = daily_returns.std() # Finally, we calculate the annualized ex-post Sharpe ratio sharpe_ratio = (252**0.5) * (avg_return - daily_benchmark_return) / std_return print(f"Cost basis: ${cost_basis:,.02f}") print(f"Final portfolio value: ${final_value:,.02f}") print(f"Total return: {total_return:.1%}") print(f"Average of daily returns: {avg_return:.4%}") print(f"Stdev of daily returns: {std_return:.4%}") print(f"Annualized Sharpe ratio: {sharpe_ratio:.2f}") ``` ## Result ```plaintext Cost basis: $20,989.00 Final portfolio value: $52,326.00 Total return: 149.3% Average of daily returns: 0.3830% Stdev of daily returns: 2.0029% Annualized Sharpe ratio: 2.96 ``` # Market halts, volatility interrupts, and price bands Venues implement different types of market integrity controls to ensure a fair and efficient market. Some of these controls will limit how far price can move in one session, as well as how much price can move in a short period of time. When these events are triggered, it can result in a temporary halt in trading or quoting. ## Velocity Logic CME Globex has [Velocity Logic](https://www.cmegroup.com/education/demos-and-tutorials/understanding-velocity-logic.html), which prevents price from moving too far, too fast. During Velocity Logic, an instrument will transition to a "Pre-Open" state. While this state is normally seen before the full session open for CME Globex, it is also seen during these events. At the end of the event, the instrument will transition back to an "Open" state. ## Overview On 2025-04-09, a tariff pause announcement was made that led to significantly increased volatility on CME Globex. Multiple Velocity Logic events were triggered in Equity Index futures following the announcement. In this example, we'll take a look at just one of the Velocity Logic events after the announcement. We'll use the [status schema](https://databento.com/docs/schemas/status.md) to show the changes in market state during this event. Next, we'll plot the BBO and trades from the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) to show how quoting and trading were affected. Additionally, we'll use the [statistics schema](https://databento.com/docs/schemas/statistics.md) to show some exchange generated statistics. ```python import databento as db import matplotlib.pyplot as plt # Create historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" product = "NQ" start = "2025-04-09T17:18:40" end = "2025-04-09T17:18:55" # Request status data and convert to DataFrame status_data = client.timeseries.get_range( dataset=dataset, schema="status", start=start, end=end, symbols=f"{product}.v.0", stype_in="continuous", ) status_df = status_data.to_df() status_df = status_df[status_df["action"].isin([db.StatusAction.PRE_OPEN, db.StatusAction.TRADING])] status_df["action"] = status_df["action"].map( { db.StatusAction.PRE_OPEN: "Pre-Open", db.StatusAction.TRADING: "Open", }, ) def add_status_vlines(ax): for status_change in status_df.iterrows(): ax.axvline(status_change[0], color="dimgray", linestyle=":", linewidth=1.5) ax.text( status_change[0], ax.get_ylim()[1], status_change[1]["action"], color="dimgray", ha="center", ) # Request MBP-1 data and convert to DataFrame mbp1_data = client.timeseries.get_range( dataset=dataset, schema="mbp-1", start=start, end=end, symbols=f"{product}.v.0", stype_in="continuous", ) mbp1_df = mbp1_data.to_df() mbp1_df = mbp1_df.rename(columns={"bid_px_00": "Bid", "ask_px_00": "Ask"}) # Request statistics data and convert to DataFrame stats_data = client.timeseries.get_range( dataset=dataset, schema="statistics", start=start, end=end, symbols=f"{product}.v.0", stype_in="continuous", ) stats_df = stats_data.to_df() stats_df = stats_df[stats_df["stat_type"] == db.StatType.INDICATIVE_OPENING_PRICE] fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10.9, 10.9), sharey=True) # Plot trades trades_df = mbp1_df[mbp1_df["action"] == db.Action.TRADE] trades_df["price"].plot( ax=ax1, style=".", markersize=5, color="C4", xlabel="Time (UTC)", ylabel="Price", label="Trades", title=f"{product} Trades", ) add_status_vlines(ax1) ax1.legend() # Plot top of book data with Indicative Opening Price mbp1_df[["Bid", "Ask"]].plot(ax=ax2, drawstyle="steps-post") add_status_vlines(ax2) stats_df["price"].plot( ax=ax2, style=".", markersize=10, color="C2", xlabel="Time (UTC)", ylabel="Price", label="IOP", title=f"{product} Top of Book", ) ax2.legend() plt.tight_layout() plt.show() ``` In this first plot, you'll see how trading activity will temporarily halt when the market transitions to a Pre-Open state during Velocity Logic. The second plot takes a look at the BBO. While quoting is still allowed during these events, the bid-ask spread can become crossed. During these Pre-Open states, the exchange publishes a statistic called the [Indicative Opening Price](https://www.cmegroup.com/education/indicative-opening-price-overview.html), also known as the "IOP". This statistic, which can be updated multiple times during a Pre-Open state, provides the most probable price that trading will resume at when the market transitions back to an Open state. When the market transitions back to an Open state, the bid-ask spread will uncross, and trading will resume. ![Velocity Logic 1](https://databento.com/docs/halts-0.svg.md) # Resample OHLCV from 1-minute to 5-minute Databento follows a convention in the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md) to only publish a record when a trade occurs in the interval. This approach is adopted by most data vendors for two key reasons: - Multiple interpolation strategies exist and the optimal choice depends on the specific use case. Client-side interpolation keeps the strategy transparent. - This reduces storage and bandwidth requirements, especially for markets with many illiquid instruments like options. This example demonstrates one way to interpolate OHLCV-1m data to ensure exactly one row per minute. The interpolation strategy used in this example does the following: - Forward-fills the close price from the last known value. - Sets open, high, and low equal to the forward-filled close price. - Sets volume to 0 for interpolated periods. ## Overview In this example, we'll use the [historical client](https://databento.com/docs/api-historical/client.md) to request data from the [OHLCV-1m schema](https://databento.com/docs/schemas/ohlcv.md). First, we'll highlight how to interpolate missing rows in the 1-minute data. Next, we'll show how to resample the data to 5-minute bars. Note that while this example is showing these methods being used individually, you can also chain these methods together. ## Example ```python import databento as db import pandas as pd def interpolate_ohlcv( df: pd.DataFrame, start: pd.Timestamp, end: pd.Timestamp, interp_interval: str, ) -> pd.DataFrame: """ Interpolate OHLCV records between `start` and `end`, since Databento only sends an OHLCV record if a trade happens in that interval. """ def _interpolate_group(group): """Interpolate OHLCV records for each group""" # Reindex with a complete index using specified start/end times group = group.reindex( pd.date_range( start=start, end=end, freq=interp_interval, inclusive="left", ).rename(group.index.name), ) # Forward fill close prices (may remain NaN if no prior data exists) group["close"] = group["close"].ffill() # For intervals with no trades, set open/high/low equal to the close and volume to 0 group = group.fillna( { **{col: group["close"] for col in ["open", "high", "low"]}, "volume": 0, }, ) group["volume"] = group["volume"].astype(int) group = group.drop(columns=["rtype", "instrument_id"], errors="ignore") return group df = ( df.groupby(["publisher_id", "symbol"]) .apply(_interpolate_group, include_groups=False) .reset_index(["publisher_id", "symbol"]) .sort_values(["ts_event", "publisher_id", "symbol"]) ) return df def resample_ohlcv( df: pd.DataFrame, resample_interval: str, ) -> pd.DataFrame: """Resample OHLCV bars to the specified interval""" resampled_df = ( df.groupby(["publisher_id", "symbol"]) .resample(resample_interval) .agg( { "open": "first", "high": "max", "low": "min", "close": "last", "volume": "sum", }, ) .reset_index(["publisher_id", "symbol"]) .sort_values(["ts_event", "publisher_id", "symbol"]) ) return resampled_df # Set parameters start = pd.Timestamp("2024-12-17T09:30:00", tz="US/Eastern") end = pd.Timestamp("2024-12-17T16:00:00", tz="US/Eastern") # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1m data for all AAPL options and convert to DataFrame df = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="ohlcv-1m", symbols="AAPL.OPT", stype_in="parent", start=start, end=end, ).to_df(tz=start.tzinfo) # Interpolate missing rows. df1 = interpolate_ohlcv(df, start, end, "1min") print(df1) # Resample to 5-minute bars df2 = resample_ohlcv(df, "5min") print(df2) ``` ## Result ```plaintext publisher_id symbol open high low close volume ts_event 2024-12-17 09:30:00-05:00 20 AAPL 241220C00115000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00155000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00170000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00195000 NaN NaN NaN NaN 0 2024-12-17 09:30:00-05:00 20 AAPL 241220C00200000 NaN NaN NaN NaN 0 ... ... ... ... ... ... ... ... 2024-12-17 15:59:00-05:00 61 AAPL 261218C00240000 50.70 50.70 50.70 50.70 0 2024-12-17 15:59:00-05:00 61 AAPL 261218P00050000 0.29 0.29 0.29 0.29 0 2024-12-17 15:59:00-05:00 61 AAPL 270115C00240000 50.14 50.14 50.14 50.14 0 2024-12-17 15:59:00-05:00 61 AAPL 270115C00250000 44.34 44.34 44.34 44.34 0 2024-12-17 15:59:00-05:00 61 AAPL 270115C00440000 3.50 3.50 3.50 3.50 0 [3129360 rows x 7 columns] publisher_id symbol open high low close volume ts_event 2024-12-17 09:30:00-05:00 20 AAPL 241220C00220000 30.70 30.70 30.70 30.70 1 2024-12-17 09:30:00-05:00 20 AAPL 241220C00225000 25.95 25.95 25.95 25.95 2 2024-12-17 09:30:00-05:00 20 AAPL 241220C00242500 8.20 8.20 8.20 8.20 1 2024-12-17 09:30:00-05:00 20 AAPL 241220C00245000 6.30 6.30 6.17 6.17 4 2024-12-17 09:30:00-05:00 20 AAPL 241220C00247500 3.70 3.85 3.70 3.85 2 ... ... ... ... ... ... ... ... 2024-12-17 15:55:00-05:00 61 AAPL 250124C00255000 5.95 5.95 5.95 5.95 1 2024-12-17 15:55:00-05:00 61 AAPL 250124C00285000 0.30 0.30 0.30 0.30 47 2024-12-17 15:55:00-05:00 61 AAPL 250124P00235000 1.01 1.01 1.01 1.01 1 2024-12-17 15:55:00-05:00 61 AAPL 250321C00225000 33.90 33.90 33.90 33.90 1 2024-12-17 15:55:00-05:00 61 AAPL 250919P00105000 0.33 0.33 0.33 0.33 2 [231713 rows x 7 columns] ``` # Convert DBN to other encoding formats In this example, we'll walk through how to convert DBN to CSV, JSON, or Parquet encoding formats. ## DBN [DBN](https://databento.com/docs/conventions/databento-binary-encoding.md) is an extremely fast message encoding and storage format for normalized market data. DBN is the default encoding used across our API and client libraries. Historical data can be requested with [timeseries.get_range()](https://databento.com/docs/api-historical/timeseries/timeseries-get-range.md). This method returns a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md). > **See also** > > Historical data can also be requested in various encoding formats with batch downloads. > See the [programmatic batch downloads example](https://databento.com/docs/examples/basics-historical/programmatic-batch-download.md) for more information. ```python import databento as db client = db.Historical("$YOUR_API_KEY") dbn_store = client.timeseries.get_range( dataset="EQUS.SUMMARY", symbols=["AAPL", "NVDA", "NFLX", "META", "GOOGL"], schema="ohlcv-1d", start="2025-09-30", ) ``` You can save this data to a DBN file with [DBNStore.to_file()](https://databento.com/docs/api-historical/helpers/dbn-store-to-file.md), then read it back into a DBNStore with [DBNStore.from_file()](https://databento.com/docs/api-historical/helpers/dbn-store-from-file.md). ```python dbn_store.to_file("demo_data.dbn") dbn_store = db.DBNStore.from_file("demo_data.dbn") ``` You can inspect the symbology mappings for a [DBNStore](https://databento.com/docs/api-historical/helpers/dbn-store.md) with the `symbology` attribute. ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") print(dbn_store.symbology) ``` ```stdout {"symbols": ["AAPL", "NVDA", "NFLX", "META", "GOOGL"], "stype_in": "raw_symbol", "stype_out": "instrument_id", "start_date": "2025-09-30", "end_date": "2025-10-01", "partial": [], "not_found": [], "mappings": {"GOOGL": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "7152"}], "NVDA": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "11667"}], "AAPL": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "38"}], "NFLX": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "11275"}], "META": [{"start_date": datetime.date(2025, 9, 30), "end_date": datetime.date(2025, 10, 1), "symbol": "10451"}]}} ``` You can [iterate](https://databento.com/docs/api-historical/helpers/dbn-store-iter.md) over a DBNStore, which will yield DBN records. ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") for record in dbn_store: print(record) ``` ```stdout OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=38, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=254855000000, pretty_open=254.855, high=255919000000, pretty_high=255.919, low=253110000000, pretty_low=253.11, close=254630000000, pretty_close=254.63, volume=37704259) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=7152, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=242810000000, pretty_open=242.81, high=243290000000, pretty_high=243.29, low=239245000000, pretty_low=239.245, close=243100000000, pretty_close=243.1, volume=34724346) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=10451, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=742250000000, pretty_open=742.25, high=742970000000, pretty_high=742.97, low=726300000000, pretty_low=726.3, close=734380000000, pretty_close=734.38, volume=16226750) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=11275, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=1206410000000, pretty_open=1206.41, high=1208500000000, pretty_high=1208.5, low=1178000000000, pretty_low=1178, close=1198920000000, pretty_close=1198.92, volume=3830304) OhlcvMsg(rtype=, publisher_id=EQUS.SUMMARY.EQUS (90), instrument_id=11667, ts_event=1759190400000000000, pretty_ts_event='2025-09-30T00:00:00.000000000Z', open=182080000000, pretty_open=182.08, high=187350000000, pretty_high=187.35, low=181480000000, pretty_low=181.48, close=186580000000, pretty_close=186.58, volume=236981032) ``` You can convert a DBNStore into a Pandas DataFrame with [DBNStore.to_df()](https://databento.com/docs/api-historical/helpers/dbn-store-to-df.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") df = dbn_store.to_df() print(df) ``` ```stdout rtype publisher_id instrument_id open high low close volume symbol ts_event 2025-09-30 00:00:00+00:00 35 90 38 254.855 255.919 253.110 254.63 37704259 AAPL 2025-09-30 00:00:00+00:00 35 90 7152 242.810 243.290 239.245 243.10 34724346 GOOGL 2025-09-30 00:00:00+00:00 35 90 10451 742.250 742.970 726.300 734.38 16226750 META 2025-09-30 00:00:00+00:00 35 90 11275 1206.410 1208.500 1178.000 1198.92 3830304 NFLX 2025-09-30 00:00:00+00:00 35 90 11667 182.080 187.350 181.480 186.58 236981032 NVDA ``` ## CSV > **See also** > > Data can also be directly requested in CSV format by setting `encoding="csv"` in [batch.submit_job()](https://databento.com/docs/api-historical/batch/batch-submit-job.md). You can save a DBNStore to a CSV file with [DBNStore.to_csv()](https://databento.com/docs/api-historical/helpers/dbn-store-to-csv.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") dbn_store.to_csv("demo_data.csv") with open("demo_data.csv", "r") as f: print(f.read()) ``` ```stdout ts_event,rtype,publisher_id,instrument_id,open,high,low,close,volume,symbol 2025-09-30T00:00:00.000000000Z,35,90,38,254.855000000,255.919000000,253.110000000,254.630000000,37704259,AAPL 2025-09-30T00:00:00.000000000Z,35,90,7152,242.810000000,243.290000000,239.245000000,243.100000000,34724346,GOOGL 2025-09-30T00:00:00.000000000Z,35,90,10451,742.250000000,742.970000000,726.300000000,734.380000000,16226750,META 2025-09-30T00:00:00.000000000Z,35,90,11275,1206.410000000,1208.500000000,1178.000000000,1198.920000000,3830304,NFLX 2025-09-30T00:00:00.000000000Z,35,90,11667,182.080000000,187.350000000,181.480000000,186.580000000,236981032,NVDA ``` ## JSON > **See also** > > Data can also be directly requested in JSON format by setting `encoding="json"` in [batch.submit_job()](https://databento.com/docs/api-historical/batch/batch-submit-job.md). You can save a DBNStore to a JSONL file with [DBNStore.to_json()](https://databento.com/docs/api-historical/helpers/dbn-store-to-json.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") dbn_store.to_json("demo_data.json") with open("demo_data.json", "r") as f: print(f.read()) ``` ```stdout {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":38},"open":"254.855000000","high":"255.919000000","low":"253.110000000","close":"254.630000000","volume":"37704259","symbol":"AAPL"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":7152},"open":"242.810000000","high":"243.290000000","low":"239.245000000","close":"243.100000000","volume":"34724346","symbol":"GOOGL"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":10451},"open":"742.250000000","high":"742.970000000","low":"726.300000000","close":"734.380000000","volume":"16226750","symbol":"META"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":11275},"open":"1206.410000000","high":"1208.500000000","low":"1178.000000000","close":"1198.920000000","volume":"3830304","symbol":"NFLX"} {"hd":{"ts_event":"2025-09-30T00:00:00.000000000Z","rtype":35,"publisher_id":90,"instrument_id":11667},"open":"182.080000000","high":"187.350000000","low":"181.480000000","close":"186.580000000","volume":"236981032","symbol":"NVDA"} ``` ## Parquet You can save a DBNStore to a parquet file with [DBNStore.to_parquet()](https://databento.com/docs/api-historical/helpers/dbn-store-to-parquet.md). ```python dbn_store = db.DBNStore.from_file("demo_data.dbn") dbn_store.to_parquet("demo_data.parquet") ``` # Request a large number of symbols > **Info** > > If you're new to Databento, see the [Quickstart guide](https://databento.com/docs/quickstart.md) for a walkthrough on setting up our live and historical APIs. > Databento supports several ways for you to get multiple symbols in a single request. If you're interested in a large universe of symbols, we recommend you use any of these supported approaches and filter the symbols on client side as opposed to requesting one symbol at a time. This minimizes the number of requests, which will usually give you better performance. ## Getting all symbols with `"ALL_SYMBOLS"` The simplest way to do this is to get all symbols of a given dataset. Passing in `"ALL_SYMBOLS"` as the `symbols` argument will give you every listed symbol. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="trades", symbols="ALL_SYMBOLS", start="2024-05-21T15:00", end="2024-05-21T15:01", ) df = data.to_df() print(f"{len(df):,d} trade(s) for {len(df['instrument_id'].unique()):,d} instrument(s)") ``` ```plaintext 31,405 trade(s) for 2,854 instrument(s) ``` This works for live subscriptions as well, which gives you a firehose subscription to all symbols: ```python import databento as db client = db.Live("$YOUR_API_KEY") client.subscribe( dataset="XNAS.ITCH", schema="ohlcv-1s", symbols="ALL_SYMBOLS", ) ``` Our live data gateways are capable of delivering every book event on most venues--like a direct feed, except over internet. You'll usually run into bandwidth limitations long before our gateway runs into other practical limits. If your application requires real-time data of all symbols at the granularity of the `trades` schema or more, we recommend you to look into our supported options for [dedicated connectivity](https://databento.com/docs/architecture/dedicated-connectivity-guide.md). ## Getting multiple, specific symbols If you're only interested in a subset of specific symbols, you can pass in a list of symbols--this works for all symbology types. We support a maximum of 2,000 symbols in a single request. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="XNAS.ITCH", schema="trades", symbols=["AAPL", "MSFT", "NVDA"], start="2024-05-21T15:00", end="2024-05-21T15:01", ) ``` Likewise, this works for live subscriptions as well: ```python import databento as db client = db.Live("$YOUR_API_KEY") client.subscribe( dataset="XNAS.ITCH", schema="ohlcv-1s", symbols=["AAPL", "MSFT", "NVDA"], ) ``` If you need more than 2,000 symbols in a single request, you can split your symbol list and dispatch multiple requests, but in such situations we usually encourage you to fetch all symbols and discard unwanted symbols on client side, even if it may cost more. There's a practical reason for this recommendation and why we limit to the number of symbols you can specify in this manner. While it may seem unintuitive, selecting specific symbols is usually a much slower operation than requesting all symbols, because our servers have to validate your symbol list, resolve any parent symbols into their child instruments, and inspect each event if it qualifies your selection. ## Getting all child instruments using parent symbology For futures and options data, there's often a need to get all symbols with the same underlying or all symbols that share the same product root. We allow you to reference such a product or cluster of symbols with the same underlying using a **parent symbol**. To specify a parent symbol for futures, you can pass in `"parent"` as the input symbology argument, `stype_in`. You must append the `.FUT` suffix to the symbol: > **See also** > > For more on parent symbology, see the [Symbology](https://databento.com/docs/conventions/symbology.md#parent) guide. ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="GLBX.MDP3", schema="trades", stype_in="parent", symbols=["ES.FUT", "NQ.FUT"], start="2024-05-21T15:00", end="2024-05-21T15:01", ) df = data.to_df() print(f"{len(df):,d} trade(s) for {len(df['instrument_id'].unique()):,d} instrument(s)") ``` ```plaintext 1,101 trade(s) for 4 instrument(s) ``` We can also use parent symbology for options with the `.OPT` suffix: ```python import databento as db client = db.Historical("$YOUR_API_KEY") data = client.timeseries.get_range( dataset="OPRA.PILLAR", schema="trades", stype_in="parent", symbols=["QQQ.OPT", "VIX.OPT"], start="2024-05-21T15:00", end="2024-05-21T15:01", ) df = data.to_df() print(f"{len(df):,d} trade(s) for {len(df['instrument_id'].unique()):,d} instrument(s)") ``` ```plaintext 1,021 trade(s) for 146 instrument(s) ``` Common use cases of parent symbology include fetching [option chains](https://databento.com/docs/examples/options/equity-options-introduction.md#using-parent-symbology-to-fetch-an-option-chain), all outrights and expirations of a futures product, or all maturities of a fixed income product. > **See also** > > See the [0DTE symbols](https://databento.com/docs/examples/options/zero-dte-options.md) example on how to filter instruments based on `expiration`. # Request a large number of symbols > **Info** > > If you're new to Databento, see the [Quickstart guide](https://databento.com/docs/quickstart.md) for a walkthrough on setting up our live and historical APIs. > Databento supports several ways for you to get multiple symbols in a single request. If you're interested in a large universe of symbols, we recommend you use any of these supported approaches and filter the symbols on client side as opposed to requesting one symbol at a time. This minimizes the number of requests, which will usually give you better performance. ## Getting all symbols with `Symbols::All` The simplest way to do this is to get all symbols of a given dataset. Passing in `Symbols::All` as the `symbols` argument will give you every listed symbol. ```rust use std::collections::HashSet; use databento::{ HistoricalClient, Symbols, dbn::{Dataset, Schema, TradeMsg}, historical::timeseries::GetRangeParams, }; use time::macros::datetime; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let mut data = client .timeseries() .get_range( &GetRangeParams::builder() .dataset(Dataset::XnasItch) .schema(Schema::Trades) .symbols(Symbols::All) .date_time_range(datetime!(2024-05-21 15:00 UTC)..datetime!(2024-05-21 15:01 UTC)) .build(), ) .await?; let trades = data.decode_records::().await?; let instruments = trades .iter() .map(|trade| trade.hd.instrument_id) .collect::>(); println!( "{} trades for {} instruments", trades.len(), instruments.len() ); Ok(()) } ``` ```plaintext 31405 trades for 2854 instruments ``` This works for live subscriptions as well, which gives you a firehose subscription to all symbols: ```rust use databento::{ LiveClient, Symbols, dbn::{Dataset, Schema}, live::Subscription, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = LiveClient::builder() .key_from_env()? .dataset(Dataset::XnasItch) .build() .await?; client .subscribe( Subscription::builder() .symbols(Symbols::All) .schema(Schema::Ohlcv1S) .build(), ) .await?; Ok(()) } ``` Our live data gateways are capable of delivering every book event on most venues--like a direct feed, except over internet. You'll usually run into bandwidth limitations long before our gateway runs into other practical limits. If your application requires real-time data of all symbols at the granularity of the `Trades` schema or more, we recommend you to look into our supported options for [dedicated connectivity](https://databento.com/docs/architecture/dedicated-connectivity-guide.md). ## Getting multiple, specific symbols If you're only interested in a subset of specific symbols, you can pass in a list of symbols--this works for all symbology types. We support a maximum of 2,000 symbols in a single request. ```rust use databento::{ HistoricalClient, dbn::{Dataset, Schema}, historical::timeseries::GetRangeParams, }; use time::macros::datetime; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let _data = client .timeseries() .get_range( &GetRangeParams::builder() .dataset(Dataset::XnasItch) .schema(Schema::Trades) .symbols(["AAPL", "MSFT", "NVDA"]) .date_time_range(datetime!(2024-05-21 15:00 UTC)..datetime!(2024-05-21 15:01 UTC)) .build(), ) .await?; Ok(()) } ``` Likewise, this works for live subscriptions as well: ```rust use databento::{ LiveClient, dbn::{Dataset, Schema}, live::Subscription, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = LiveClient::builder() .key_from_env()? .dataset(Dataset::XnasItch) .build() .await?; client .subscribe( Subscription::builder() .symbols(["AAPL", "MSFT", "NVDA"]) .schema(Schema::Ohlcv1S) .build(), ) .await?; Ok(()) } ``` If you need more than 2,000 symbols in a single request, you can split your symbol list and dispatch multiple requests, but in such situations we usually encourage you to fetch all symbols and discard unwanted symbols on client side, even if it may cost more. There's a practical reason for this recommendation and why we limit to the number of symbols you can specify in this manner. While it may seem unintuitive, selecting specific symbols is usually a much slower operation than requesting all symbols, because our servers have to validate your symbol list, resolve any parent symbols into their child instruments, and inspect each event if it qualifies your selection. ## Getting all child instruments using parent symbology For futures and options data, there's often a need to get all symbols with the same underlying or all symbols that share the same product root. We allow you to reference such a product or cluster of symbols with the same underlying using a **parent symbol**. To specify a parent symbol for futures, you can pass in `SType::Parent` as the input symbology argument, `stype_in`. You must append the `.FUT` suffix to the symbol: > **See also** > > For more on parent symbology, see the [Symbology](https://databento.com/docs/conventions/symbology.md#parent) guide. ```rust use std::collections::HashSet; use databento::{ HistoricalClient, dbn::{Dataset, SType, Schema, TradeMsg}, historical::timeseries::GetRangeParams, }; use time::macros::datetime; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let mut data = client .timeseries() .get_range( &GetRangeParams::builder() .dataset(Dataset::GlbxMdp3) .schema(Schema::Trades) .stype_in(SType::Parent) .symbols(["ES.FUT", "NQ.FUT"]) .date_time_range(datetime!(2024-05-21 15:00 UTC)..datetime!(2024-05-21 15:01 UTC)) .build(), ) .await?; let trades = data.decode_records::().await?; let instruments = trades .iter() .map(|trade| trade.hd.instrument_id) .collect::>(); println!( "{} trades for {} instruments", trades.len(), instruments.len() ); Ok(()) } ``` ```plaintext 1101 trades for 4 instruments ``` We can also use parent symbology for options with the `.OPT` suffix: ```rust use std::collections::HashSet; use databento::{ HistoricalClient, dbn::{Dataset, SType, Schema, TradeMsg}, historical::timeseries::GetRangeParams, }; use time::macros::datetime; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let mut data = client .timeseries() .get_range( &GetRangeParams::builder() .dataset(Dataset::OpraPillar) .schema(Schema::Trades) .stype_in(SType::Parent) .symbols(["QQQ.OPT", "VIX.OPT"]) .date_time_range(datetime!(2024-05-21 15:00 UTC)..datetime!(2024-05-21 15:01 UTC)) .build(), ) .await?; let trades = data.decode_records::().await?; let instruments = trades .iter() .map(|trade| trade.hd.instrument_id) .collect::>(); println!( "{} trades for {} instruments", trades.len(), instruments.len() ); Ok(()) } ``` ```plaintext 1021 trades for 146 instruments ``` Common use cases of parent symbology include fetching [option chains](https://databento.com/docs/examples/options/equity-options-introduction.md#using-parent-symbology-to-fetch-an-option-chain), all outrights and expirations of a futures product, or all maturities of a fixed income product. ## Symbology | Title | Subtitle | Supported languages | | [Continuous contracts](https://databento.com/docs/examples/symbology/continuous.md) | Get symbols by lead month and handle rollovers using continuous symbology. | Python, Rust, Futures, Historical | | [Parent symbology](https://databento.com/docs/examples/symbology/parent-symbology.md) | Get all symbols with the same product root or option chains using parent symbology. | Python, C++, Rust, Historical | | [Symbology mapping for live data](https://databento.com/docs/examples/symbology/live-symbol-mapping.md) | Convert between instrument IDs and raw symbols. | Python, C++, Rust, Live | | [Dataset symbols](https://databento.com/docs/examples/symbology/all-dataset-symbols.md) | Get all symbols available in a dataset. | Python, Historical, Live | # Continuous contracts > **Info** > Our continuous contract symbology is a notation that maps to an actual, tradable instrument on any given date. > The continuous contract prices provided are the original, unadjusted prices. Unlike some vendor implementations that > back-adjust prices to remove jumps during rollovers, our approach maintains the original properties of the data. When dealing with far-reaching historical data on expiring products (like the [E-mini S&P 500 Futures](https://databento.com/portal/datasets/GLBX.MDP3/Futures/ES)), users have to be concerned with expiration cycles and what is defined as the [front month](https://www.cftc.gov/LearnAndProtect/AdvisoriesAndArticles/CFTCGlossary/index.htm#frontmonth) or first expiry, second expiry, etc. When requesting (E-mini S&P 500 Futures) data, we can specify the particular product we want. Take for example, `ESH5`, the March 2025 expiring E-Mini S&P 500 futures contract. Since this instrument has a March 2025 expiry, it will not exist after March 2025. Since March 2025 is the front month (i.e. the nearest expiration date for the contract) only from December 2024 to March 2025, the instrument's data dating before December 2024 could be considered irrelevant. Continuous contracts symbology provides a way to navigate this complexity. You can easily access recent contracts: front month, second expiry, and so on. ## Symbology To access a specific contract in a continuous contract product, query for that contract using this symbol structure. `[ticker].v.[expiry]` ## Expiry In place of the exchange's proprietary code for specifying the month of expiration (e.g. `H5` for `ESH5` in CME data), we use an index to refer to which expiry in the current cycle we want. | Index | Expiry | | 0 | front month | | 1 | second expiry | | 2 | third expiry | | n | nth expiry | ## Algorithm To request the front month for the `ES` product based on volume, use `ES.v.0`. In this example, the v indicates the calendar roll rule. The zero corresponds to the front month. Learn more about continuous contract symbology in our [continuous symbology](https://databento.com/docs/conventions/symbology.md#continuous) guide. ## Example ```python import databento as db # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2025-03-16" end = "2025-03-23" # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request OHLCV-1d data for the continuous contract data = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=f"{product}.v.0", stype_in="continuous", start=start, end=end, ) # Convert to DataFrame df = data.to_df() print(df) ``` The data received is now related to the volume-based front month for each day of the requested time range. If we are interested in the second month then we can just use `ES.v.1` instead. The results below are from the week of expiration for `ES`. As you can see, the contract that maps to the volume-based front month, as shown by the `instrument_id` field, changes mid-week as the volume shifts to the new contract. ```plaintext rtype publisher_id instrument_id open high low close volume symbol ts_event 2025-03-16 00:00:00+00:00 35 1 5002 5626.75 5632.00 5602.50 5608.50 30078 ES.v.0 2025-03-17 00:00:00+00:00 35 1 5002 5608.75 5707.50 5599.00 5679.50 723022 ES.v.0 2025-03-18 00:00:00+00:00 35 1 5002 5679.75 5683.25 5600.00 5621.25 412431 ES.v.0 2025-03-19 00:00:00+00:00 35 1 4916 5672.50 5770.50 5657.50 5747.75 1331926 ES.v.0 2025-03-20 00:00:00+00:00 35 1 4916 5747.50 5765.25 5682.50 5714.75 1503542 ES.v.0 2025-03-21 00:00:00+00:00 35 1 4916 5714.25 5723.75 5651.25 5720.00 1454908 ES.v.0 ``` # Continuous contracts > **Info** > Our continuous contract symbology is a notation that maps to an actual, tradable instrument on any given date. > The continuous contract prices provided are the original, unadjusted prices. > Unlike some vendor implementations that back-adjust prices to remove jumps during rollovers, our approach maintains the original properties of the data. When dealing with far-reaching historical data on expiring products (like the [E-mini S&P 500 Futures](https://databento.com/portal/datasets/GLBX.MDP3/Futures/ES)), users have to be concerned with expiration cycles and what is defined as the [front month](https://www.cftc.gov/LearnAndProtect/AdvisoriesAndArticles/CFTCGlossary/index.htm#frontmonth) or first expiry, second expiry, etc. When requesting (E-mini S&P 500 Futures) data, we can specify the particular product we want. Take for example, `ESH0`, the March 2020 expiring E-Mini S&P 500 futures contract. Since this product has a March 2020 expiry, the product may not exist after March 2020. Since March 2020 is the front month (i.e. the nearest expiration date for the contract) only from December 2019 to March 2020, the product's data dating before 2019 could be considered irrelevant. Continuous contracts symbology provides a way to navigate this complexity. You can easily access recent contracts: front month, second expiry, and so on. ## Symbology To access a specific contract in a continuous contract product, query for that contract using this symbol structure. `[ticker].c.[expiry]` ## Ticker Use the product's ticker symbol. For a list of tickers, for instance those under CME, see [here]($PLATFORM_HOSTNAME/datasets/GLBX.MDP3). ## Expiry In place of the exchange's proprietary code for specifying the month of expiration (e.g. `H0` for `ESH0` in CME data), we use an index to refer to which expiry in the current cycle we want. | Index | Expiry | | 0 | front month | | 1 | second expiry | | 2 | third expiry | | n | nth expiry | ## Algorithm To request the front month for the `ES` product, use `ES.c.0`. In this example, the c indicates the calendar roll rule. The zero corresponds to the front month. Learn more about continuous contract symbology in our [Symbology](https://databento.com/docs/conventions/symbology.md) guide. ## Example ```rust use databento::{ HistoricalClient, dbn::{Dataset, SType, Schema}, historical::timeseries::GetRangeParams, }; use time::macros::date; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let _data = client .timeseries() .get_range( &GetRangeParams::builder() .date_time_range(date!(2022 - 06 - 01)) .dataset(Dataset::GlbxMdp3) .schema(Schema::Ohlcv1D) .stype_in(SType::Continuous) .symbols("ES.c.0") .build(), ) .await?; Ok(()) } ``` The data received is now related to the front month for each day of the whole requested time range. If we are interested in the second month then we can just use `ES.c.1` instead. # Parent symbology In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve a large number of instrument definitions. We will do this using the [smart symbology](https://databento.com/docs/conventions/symbology.md) feature to request all active E-mini S&P 500 futures contracts on a specified date. ## Definition Schema This example uses the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties such as expiration dates. ## Example ```cpp #include #include #include #include #include #include #include #include #include #include namespace db = databento; int main() { db::TsSymbolMap symbol_mappings; auto extract_symbol_mappings = [&symbol_mappings](db::Metadata&& metadata) { symbol_mappings = metadata.CreateSymbolMap(); }; std::vector definitions; auto print_definitions = [&definitions](const db::Record& record) { definitions.emplace_back(record.Get()); return db::KeepGoing::Continue; }; auto client = db::Historical::Builder().SetKey("$YOUR_API_KEY").Build(); client.TimeseriesGetRange( "GLBX.MDP3", db::DateTimeRange{"2022-08-26"}, {"ES.FUT"}, db::Schema::Definition, db::SType::Parent, db::SType::InstrumentId, {}, extract_symbol_mappings, print_definitions); std::sort( definitions.begin(), definitions.end(), [](const auto& lhs, const auto& rhs) { return std::tuple{lhs.expiration, std::string_view{lhs.RawSymbol()}} < std::tuple{rhs.expiration, std::string_view{rhs.RawSymbol()}}; }); for (const auto& definition : definitions) { std::cout << definition.hd.instrument_id << '\t' << std::setw(9) << symbol_mappings.At(definition) << '\t' << db::ToIso8601(definition.expiration) << '\n'; } return 0; } ``` ## Result ```plaintext 97042 ESU2 2022-09-16T13:30:00.000000000Z 431108 ESU2-ESH3 2022-09-16T13:30:00.000000000Z 26998 ESU2-ESM3 2022-09-16T13:30:00.000000000Z 16445 ESU2-ESU3 2022-09-16T13:30:00.000000000Z 431796 ESU2-ESZ2 2022-09-16T13:30:00.000000000Z 206323 ESZ2 2022-12-16T14:30:00.000000000Z 344396 ESZ2-ESH3 2022-12-16T14:30:00.000000000Z 19355 ESZ2-ESM3 2022-12-16T14:30:00.000000000Z 36396 ESZ2-ESU3 2022-12-16T14:30:00.000000000Z 4931 ESZ2-ESZ3 2022-12-16T14:30:00.000000000Z 206299 ESH3 2023-03-17T13:30:00.000000000Z 431569 ESH3-ESM3 2023-03-17T13:30:00.000000000Z 20604 ESH3-ESU3 2023-03-17T13:30:00.000000000Z 16567 ESH3-ESZ3 2023-03-17T13:30:00.000000000Z 95414 ESM3 2023-06-16T13:30:00.000000000Z 2018 ESM3-ESH4 2023-06-16T13:30:00.000000000Z 28177 ESM3-ESU3 2023-06-16T13:30:00.000000000Z 37976 ESM3-ESZ3 2023-06-16T13:30:00.000000000Z 3445 ESU3 2023-09-15T13:30:00.000000000Z 18909 ESU3-ESH4 2023-09-15T13:30:00.000000000Z 35947 ESU3-ESZ3 2023-09-15T13:30:00.000000000Z 314863 ESZ3 2023-12-15T14:30:00.000000000Z 33040 ESZ3-ESH4 2023-12-15T14:30:00.000000000Z 2826 ESZ3-ESM4 2023-12-15T14:30:00.000000000Z 17077 ESH4 2024-03-15T13:30:00.000000000Z 17684 ESH4-ESM4 2024-03-15T13:30:00.000000000Z 5602 ESM4 2024-06-21T13:30:00.000000000Z 4155 ESM4-ESU4 2024-06-21T13:30:00.000000000Z 118 ESU4 2024-09-20T13:30:00.000000000Z 183748 ESZ4 2024-12-20T14:30:00.000000000Z 294973 ESZ5 2025-12-19T14:30:00.000000000Z 10252 ESZ6 2026-12-18T14:30:00.000000000Z ``` # Parent symbology ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve a large number of instrument definitions using [parent symbology](https://databento.com/docs/conventions/symbology.md#parent). When requesting data with a parent symbol (`ES.FUT`), all contracts that trade under the parent name will be returned. This includes outright futures (`ESM5`), as well as calendar spreads (`ESM5-ESU5`). > **See also** > > If you are only interested in the front month contract for a futures product, see our [continuous symbology](https://databento.com/docs/examples/symbology/continuous.md) example. We'll use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties such as `expiration`, `raw_symbol`, and `instrument_class`. `instrument_class` will indicate the type of instrument, such as future or future spread. A full list of variants can be found in the [instrument class documentation](https://databento.com/docs/schemas/instrument-definitions.md#instrument-class). ## Example ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Request definition data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2025-05-19", symbols="ES.FUT", stype_in="parent", schema="definition", ) # Convert to DataFrame df = data.to_df() # Filter out spreads and sort by expiration df = df[df["instrument_class"] == db.InstrumentClass.FUTURE] df = df.set_index("expiration").sort_index() print(df[["instrument_id", "raw_symbol"]]) ``` ## Result ```plaintext instrument_id raw_symbol expiration 2025-06-20 13:30:00+00:00 4916 ESM5 2025-09-19 13:30:00+00:00 14160 ESU5 2025-12-19 14:30:00+00:00 294973 ESZ5 2026-03-20 13:30:00+00:00 42140878 ESH6 2026-06-18 13:30:00+00:00 42140864 ESM6 ... 2026-09-18 13:30:00+00:00 42140870 ESU6 2026-12-18 14:30:00+00:00 10252 ESZ6 2027-03-19 13:30:00+00:00 42140860 ESH7 2027-06-17 13:30:00+00:00 42140856 ESM7 2027-09-17 13:30:00+00:00 42140874 ESU7 2027-12-17 14:30:00+00:00 17740 ESZ7 2028-03-17 13:30:00+00:00 42140879 ESH8 ``` # Parent symbology In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve a large number of instrument definitions. We will do this using the [smart symbology](https://databento.com/docs/conventions/symbology.md) feature to request all active E-mini S&P 500 futures contracts on a specified date. ## Definition Schema This example uses the [definition schema](https://databento.com/docs/schemas/whats-a-schema.md), which contains instrument definitions and properties such as expiration dates. ## Example ```rust use databento::{ HistoricalClient, dbn::{InstrumentDefMsg, SType, Schema, TsSymbolMap, decode::DbnMetadata}, historical::timeseries::GetRangeParams, }; use time::macros::date; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; // First, we request the definition data let mut decoder = client .timeseries() .get_range( &GetRangeParams::builder() .dataset("GLBX.MDP3") .date_time_range(date!(2022 - 08 - 26)) .symbols("ES.FUT") .stype_in(SType::Parent) .schema(Schema::Definition) .build(), ) .await?; let symbol_mappings = decoder.metadata().symbol_map()?; // Then, we store all the definitions in a Vec let definitions = decoder.decode_records::().await?; // Finally, we sort by contract expiration and print the results print_by_expiration(definitions, &symbol_mappings); Ok(()) } fn print_by_expiration(mut definitions: Vec, symbol_mappings: &TsSymbolMap) { definitions.sort_by_key(|def| (def.expiration, def.raw_symbol().unwrap().to_owned())); println!("expiration instrument_id raw_symbol"); for def in definitions { let expiration = def.expiration().unwrap(); println!( "{expiration} {:6} {:9}", def.hd.instrument_id, symbol_mappings[&(def.ts_recv().unwrap().date(), def.hd.instrument_id)], ); } } ``` ## Result ```plaintext expiration instrument_id raw_symbol 2022-09-16 13:30:00.0 +00:00:00 97042 ESU2 2022-09-16 13:30:00.0 +00:00:00 431108 ESU2-ESH3 2022-09-16 13:30:00.0 +00:00:00 26998 ESU2-ESM3 2022-09-16 13:30:00.0 +00:00:00 16445 ESU2-ESU3 2022-09-16 13:30:00.0 +00:00:00 431796 ESU2-ESZ2 2022-12-16 14:30:00.0 +00:00:00 206323 ESZ2 2022-12-16 14:30:00.0 +00:00:00 344396 ESZ2-ESH3 2022-12-16 14:30:00.0 +00:00:00 19355 ESZ2-ESM3 2022-12-16 14:30:00.0 +00:00:00 36396 ESZ2-ESU3 2022-12-16 14:30:00.0 +00:00:00 4931 ESZ2-ESZ3 2023-03-17 13:30:00.0 +00:00:00 206299 ESH3 2023-03-17 13:30:00.0 +00:00:00 431569 ESH3-ESM3 2023-03-17 13:30:00.0 +00:00:00 20604 ESH3-ESU3 2023-03-17 13:30:00.0 +00:00:00 16567 ESH3-ESZ3 2023-06-16 13:30:00.0 +00:00:00 95414 ESM3 2023-06-16 13:30:00.0 +00:00:00 2018 ESM3-ESH4 2023-06-16 13:30:00.0 +00:00:00 28177 ESM3-ESU3 2023-06-16 13:30:00.0 +00:00:00 37976 ESM3-ESZ3 2023-09-15 13:30:00.0 +00:00:00 3445 ESU3 2023-09-15 13:30:00.0 +00:00:00 18909 ESU3-ESH4 2023-09-15 13:30:00.0 +00:00:00 35947 ESU3-ESZ3 2023-12-15 14:30:00.0 +00:00:00 314863 ESZ3 2023-12-15 14:30:00.0 +00:00:00 33040 ESZ3-ESH4 2023-12-15 14:30:00.0 +00:00:00 2826 ESZ3-ESM4 2024-03-15 13:30:00.0 +00:00:00 17077 ESH4 2024-03-15 13:30:00.0 +00:00:00 17684 ESH4-ESM4 2024-06-21 13:30:00.0 +00:00:00 5602 ESM4 2024-06-21 13:30:00.0 +00:00:00 4155 ESM4-ESU4 2024-09-20 13:30:00.0 +00:00:00 118 ESU4 2024-12-20 14:30:00.0 +00:00:00 183748 ESZ4 2025-12-19 14:30:00.0 +00:00:00 294973 ESZ5 2026-12-18 14:30:00.0 +00:00:00 10252 ESZ6 ``` # Symbology mapping for live data ## Overview In this example we will use the [LiveThreaded client](https://databento.com/docs/api-live/client/live.md) to create symbology mappings using the `SymbolMappingMsg`, which are sent at the start of every session and when a subscribed symbol's mapping changes. These messages allow us to map between the instrument ID on every record and more human-readable STypes like raw and continuous contract symbologies. [PitSymbolMap](https://databento.com/docs/api-live/helpers/pit-symbol-map.md) streamlines keeping track of these mappings. > **See also** > > [Symbology](https://databento.com/docs/api-live/basics/symbology.md) for further details. ## Example ```cpp #include #include // dataset #include // LiveBuilder #include // SymbolMappingMsg #include #include #include namespace db = databento; int main() { auto client = db::LiveBlocking::Builder() .SetKey("$YOUR_API_KEY") .SetDataset(db::Dataset::GlbxMdp3) .BuildBlocking(); std::vector raw_symbols{"ES.c.0", "ES.c.1", "ES.c.2", "ES.c.3"}; client.Subscribe(raw_symbols, db::Schema::Ohlcv1S, db::SType::Continuous); client.Start(); std::size_t mappings_received{}; while (mappings_received < raw_symbols.size()) { auto record = client.NextRecord(); if (auto* sym_mapping = record.GetIf()) { std::cout << sym_mapping->STypeInSymbol() << " (" << sym_mapping->STypeOutSymbol() << ") has an instrument ID of " << sym_mapping->hd.instrument_id << '\n'; ++mappings_received; } } return 0; } ``` ## Result ```plaintext ES.c.0 (ESM5) has an instrument ID of 4916 ES.c.1 (ESU5) has an instrument ID of 14160 ES.c.2 (ESZ5) has an instrument ID of 294973 ES.c.3 (ESH6) has an instrument ID of 42140878 ``` # Symbology mapping for live data ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to create symbology mappings using the `SymbolMappingMsg`. This message is sent at the start of every session and when a subscribed symbol mapping is updated. These messages allow us to map between the instrument ID on every record and more human-readable STypes like raw and continuous contract symbologies. ## Example ```python import databento as db # First, create a live client and connect live_client = db.Live(key="$YOUR_API_KEY") # Next, we will subscribe to a few symbols live_client.subscribe( dataset="GLBX.MDP3", schema="ohlcv-1s", symbols=["ES.c.0", "ES.c.1", "ES.c.2", "ES.c.3"], stype_in="continuous", ) # Finally, we will iterate over the incoming records. # The SymbolMappingMsgs will be sent at the start. for record in live_client: if isinstance(record, db.SymbolMappingMsg): instrument_id = record.instrument_id # The stype_in_symbol contains the continuous symbology we requested continuous_symbol = record.stype_in_symbol # The stype_out_symbol contains the raw symbol for smart symbology raw_symbol = record.stype_out_symbol print( f"{continuous_symbol} ({raw_symbol}) has an instrument ID of {instrument_id}", ) # Close the connection # We will still process all received data live_client.stop() ``` ## Result ```plaintext ES.c.0 (ESM4) has an instrument ID of 5602 ES.c.1 (ESU4) has an instrument ID of 118 ES.c.2 (ESZ4) has an instrument ID of 183748 ES.c.3 (ESH5) has an instrument ID of 5002 ``` > **See also** > > [Symbology](https://databento.com/docs/api-live/basics/symbology.md) for further details. # Symbology mapping for live data ## Overview In this example we will use the [Live client](https://databento.com/docs/api-live/client/live.md) to create symbology mappings using the `SymbolMappingMsg`, which are sent at the start of every session and when a subscribed symbol's mapping changes. These messages allow us to map between the instrument ID on every record and more human-readable STypes like raw and continuous contract symbologies. [PitSymbolMap](https://databento.com/docs/api-live/helpers/pit-symbol-map.md) streamlines keeping track of these mappings. > **See also** > > [Symbology](https://databento.com/docs/api-live/basics/symbology.md) for further details. ## Example ```rust use databento::{ LiveClient, dbn::{Dataset, SType, Schema, SymbolMappingMsg}, live::Subscription, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = LiveClient::builder() .key_from_env()? .dataset(Dataset::GlbxMdp3) .build() .await?; let symbols = vec!["ES.c.0", "ES.c.1", "ES.c.2", "ES.c.3"]; client .subscribe( Subscription::builder() .schema(Schema::Ohlcv1S) .symbols(symbols.clone()) .stype_in(SType::Continuous) .build(), ) .await?; client.start().await?; let mut count = 0; while let Some(record) = client.next_record().await? { if let Some(symbol_mapping) = record.get::() { let instrument_id = symbol_mapping.hd.instrument_id; let symbol = symbol_mapping.stype_in_symbol()?; println!("{symbol} has an instrument ID of {instrument_id}"); count += 1; if count >= symbols.len() { break; } } } client.close().await?; Ok(()) } ``` ## Result ```plaintext ES.c.0 has an instrument ID of 42140878 ES.c.1 has an instrument ID of 42140864 ES.c.2 has an instrument ID of 42140870 ES.c.3 has an instrument ID of 10252 ``` # Dataset symbols ## Overview In this example we will show how to get all symbols for a dataset. We will use the [Historical client](https://databento.com/docs/api-historical/client.md) to get all symbols available on a specific date. We will also use the [Live client](https://databento.com/docs/api-live/client/live.md) to get all symbols active for the current session. ## Definition schema We will use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). We will create a map of `raw_symbol` to `InstrumentDefMsgs`. An `InstrumentDefMsg` contains fields such as: - `raw_symbol` - The instrument name (symbol) provided by the publisher - `instrument_class` - The classification of the instrument - `expiration` - The expiration timestamp for options/futures contracts - `min_price_increment` - The minimum constant trade tick for an instrument ## Historical example First, we'll take a look at how to do this with the Historical client. ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") dataset = "EQUS.SUMMARY" # Download definition schema for all symbols data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", start="2025-02-06", end="2025-02-07", schema="definition", ) # Create a map of raw_symbol -> InstrumentDefMsgs symbol_map = {msg.raw_symbol: msg for msg in data} # Get all symbols symbols = sorted(symbol_map.keys()) # Print out the symbol count and the first 5 symbols print(f"Total symbol count for {dataset} = {len(symbols)}\nFirst 5 symbols...{symbols[:5]}") ``` ```plaintext Total symbol count for EQUS.SUMMARY = 11198 First 5 symbols...['A', 'AA', 'AAA', 'AAAU', 'AACBU'] ``` ## Live example Next, we'll take a look at how to do this with the Live client. > **Info** > > You will need a live license for the dataset to run this example. See our [plans and live data page]($PLATFORM_HOSTNAME/docs/portal/live-data) for more information. ```python import databento as db # Create a live client live_client = db.Live("$YOUR_API_KEY") dataset = "GLBX.MDP3" # Subscribe to the definition schema for all symbols for this session live_client.subscribe( dataset=dataset, schema="definition", symbols="ALL_SYMBOLS", start=0, ) # Create a map of raw_symbol -> InstrumentDefMsgs symbol_map = {} def append_symbol_map(msg): if isinstance(msg, db.InstrumentDefMsg): symbol_map[msg.raw_symbol] = msg # Add the callback and start the stream live_client.add_callback(append_symbol_map) live_client.start() # Listen to the stream for 10 seconds to get all messages live_client.block_for_close(timeout=10) # Get all symbols symbols = sorted(symbol_map.keys()) # Print out the symbol count and the first 5 symbols print(f"Total symbol count for {dataset} = {len(symbols)}\nFirst 5 symbols...{symbols[:5]}") ``` ```plaintext Total symbol count for GLBX.MDP3 = 784038 First 5 symbols...['00CH5', '00CJ5', '00CK5', '00CM5', '00CN5'] ``` ## Instrument definitions | Title | Subtitle | Supported languages | | [Find most liquid instruments](https://databento.com/docs/examples/instrument-definitions/liquid-universe.md) | Find most active instruments using instrument definitions and OHLCV data. | Python, Historical | | [Handling tick sizes](https://databento.com/docs/examples/instrument-definitions/tick-sizes.md) | Convert prices and spread size to tick units. | Python, Historical | | [Find ICE intercommodity spreads](https://databento.com/docs/examples/instrument-definitions/ice-crack-spreads.md) | Find ICE intercommodity futures spreads from leg definitions. | Python, Historical | | [Displaying fractional prices](https://databento.com/docs/examples/instrument-definitions/fractional-prices.md) | Render prices in the venue's own fractional convention, such as 108'185. | Python, Historical | | [Computing tick value and contract notional](https://databento.com/docs/examples/instrument-definitions/contract-notional.md) | Derive the value of one tick and the notional value of a contract. | Python, Historical | # Finding liquid instruments ## What is a liquid instrument? A liquid instrument is an instrument that is readily traded. Identifying liquid instruments is important for a few reasons: - **Reducing transaction costs.** A liquid instrument generally has tight bid-ask spreads and adequate size on the bid and ask. This reduces the amount of slippage incurred from using aggressive order types. - **Signal generation.** Consistent trading activity allows for signal generation in order flow strategies. - **Price efficiency.** Increased participation results in more efficient price discovery. This results in more reliable price valuations for risk management. Check out the [Databento Microstructure Guide](https://databento.com/microstructure/liquidity) for more information about liquidity. ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to find liquid futures instruments. We'll use the following schemas: - The [statistics schema](https://databento.com/docs/schemas/statistics.md), which contains exchange-published statistics such as cleared volume and open interest. - The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md), which contains instrument definitions and properties such as `raw_symbol` and `asset`. The `asset` field refers to the parent product for an instrument. - The [BBO schema](https://databento.com/docs/schemas/bbo.md), which contains the best bid and offer, subsampled at 1-second or 1-minute intervals. We'll request statistics and definition data for all symbols. Next, we'll filter for the top 25 instruments by volume. After we filter, we'll get BBO data for these instruments to find the median bid and ask size over a full day. ## Example ```python import databento as db # First, create a Historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" start_date = "2025-03-07" top_instruments_count = 25 # First, download definition data for all symbols def_data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="definition", start=start_date, ) # Convert to DataFrame. Filter for outright futures def_df = def_data.to_df() def_df = def_df[def_df["instrument_class"] == db.InstrumentClass.FUTURE] def_df = def_df[["raw_symbol", "instrument_id", "asset", "min_price_increment"]] # Next, download statistics data for all symbols stats_data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="statistics", start=start_date, ) # Convert to DataFrame stats_df = stats_data.to_df() # Get cleared volume records volume_df = stats_df[stats_df["stat_type"] == db.StatType.CLEARED_VOLUME] volume_df = volume_df.drop_duplicates("instrument_id", keep="last") volume_df = volume_df.rename(columns={"quantity": "volume"}) volume_df = volume_df[["instrument_id", "volume"]] # Get open interest records open_interest_df = stats_df[stats_df["stat_type"] == db.StatType.OPEN_INTEREST] open_interest_df = open_interest_df.drop_duplicates("instrument_id", keep="last") open_interest_df = open_interest_df.rename(columns={"quantity": "open_interest"}) open_interest_df = open_interest_df[["instrument_id", "open_interest"]] # Merge volume and open interest data stats_df = volume_df.merge(open_interest_df, on="instrument_id", how="inner") # Merge definition data with statistics data stats_df = stats_df.merge(def_df, on="instrument_id", how="inner") # Sort by volume, keeping one instrument per product stats_df = stats_df.sort_values("volume", ascending=False) stats_df = stats_df.drop_duplicates("asset") # Get instrument IDs for highest volume instruments top_instruments = stats_df["instrument_id"].to_list()[:top_instruments_count] # Download BBO-1s data for highest volume instruments bbo_data = client.timeseries.get_range( dataset=dataset, symbols=top_instruments, stype_in="instrument_id", schema="bbo-1s", start=start_date, ) # Convert to DataFrame bbo_df = bbo_data.to_df() # Merge DataFrames df = bbo_df.merge(stats_df, on="instrument_id", how="inner") df["spread_ticks"] = (df["ask_px_00"] - df["bid_px_00"]) / df["min_price_increment"] # Calculate aggregated values and sort by volume df = ( df.groupby(by="instrument_id") .agg( product=("asset", "first"), symbol=("raw_symbol", "first"), volume=("volume", "first"), open_interest=("open_interest", "first"), median_bid_size=("bid_sz_00", lambda x: int(x.median())), median_ask_size=("ask_sz_00", lambda x: int(x.median())), median_tick_spread=("spread_ticks", lambda x: int(x.median().round())), ) .sort_values("volume", ascending=False) ) print(df) ``` ## Result ```plaintext product symbol volume open_interest median_bid_size median_ask_size median_tick_spread instrument_id 42004113 ZN ZNM5 3259518 4675273 903 946 1 5002 ES ESH5 2553819 2115168 9 9 1 42003617 MNQ MNQH5 2419630 154823 2 2 2 42325990 ZF ZFM5 2039292 6111849 483 492 1 42005347 MES MESH5 1931181 210864 12 12 1 42325992 ZT ZTM5 1320314 3854308 443 455 1 42002878 TN TNM5 962958 2206335 278 285 1 254274 SR3 SR3Z5 894673 979670 2864 2898 1 42288528 NQ NQH5 763150 273526 1 1 4 42004255 ZB ZBM5 708447 1770658 228 236 1 42001682 UB UBM5 474088 1757409 99 102 1 42272 6E 6EH5 392404 601400 15 14 1 625061 CL CLJ5 341632 239110 6 6 1 680969 ZC ZCK5 313858 756635 83 74 1 42001620 RTY RTYH5 258273 454862 2 3 2 57969 6J 6JH5 242466 278473 23 21 1 42011026 MYM MYMH5 216632 23321 3 3 2 892 NG NGJ5 202475 208149 3 3 2 457556 ZS ZSK5 192141 382968 16 15 1 42002868 YM YMH5 185559 70792 3 3 2 29307 6B 6BH5 178634 192142 39 40 1 19604 GC GCJ5 166364 329381 2 2 2 713217 ZQ ZQJ5 158838 497521 41383 11147 1 45908 6C 6CH5 137522 303329 27 28 1 52126 6A 6AH5 131315 180048 29 31 1 ``` # Handling tick sizes ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to retrieve instrument definitions and top-of-book data for two futures contracts during non-US equity cash session hours. We will then calculate the mean spread and normalize the values to a discrete number of price increments or "ticks" using the `min_price_increment` field from the definition schema. Finally, we will plot the changes in mean spread. The futures we will use for this example are [British Pound Futures (6B)](https://databento.com/portal/datasets/GLBX.MDP3/Futures/6B) and [Euro FX Futures (6E)](https://databento.com/portal/datasets/GLBX.MDP3/Futures/6E). We will use [continuous contract symbology](https://databento.com/docs/conventions/symbology.md#continuous) to request the second highest contracts by open interest, which should have a more dynamic mean spread. ## Definition schema The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) contains instrument definitions and properties. In this example, we are interested in the `min_price_increment` field, which will allow us to normalize a spread to a discrete number of increments. ## Mean spread The mean spread is the average difference between the bid and ask price of an instrument. The mean spread is a metric which helps traders understand the liquidity and transaction costs associated with a particular trade. Illiquid and volatile instruments tend to have a larger mean spread. Such instruments are more likely to experience price slippage, where the actual price of a trade moves or "slips" away from the expected price. ## Dependencies For plotting, this example will use the [matplotlib](https://pypi.org/project/matplotlib/) package. This dependency can be installed with the following: ```bash $ pip install matplotlib ``` ## Example ```python import databento as db import pandas as pd from matplotlib import pyplot as plt # Get symbols for 6B and 6E futures with the second highest open interest symbols = ["6B.n.1", "6E.n.1"] # Define a start and end time for this example dataset = "GLBX.MDP3" start = pd.Timestamp("2024-04-07T21:00:00", tz="US/Eastern") end = pd.Timestamp("2024-04-08T04:00:00", tz="US/Eastern") # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, retrieve the instrument definitions for our symbols definitions_data = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=symbols, stype_in="continuous", start=start.date(), ) # And extract the min_price_increment field definitions_df = definitions_data.to_df() tick_sizes = definitions_df[["raw_symbol", "instrument_id", "min_price_increment"]].set_index( "instrument_id", ) # Print the min_price_increment for each symbol print(tick_sizes) # Then, request top-of-book updates for our symbols overnight mbp_data = client.timeseries.get_range( dataset=dataset, schema="MBP-1", symbols=symbols, stype_in="continuous", start=start, end=end, ) # And convert to a DataFrame and join with our tick size mbp_df = mbp_data.to_df(tz="US/Eastern") mbp_df = mbp_df.join(tick_sizes, on="instrument_id") mbp_df = mbp_df[["raw_symbol", "ask_px_00", "bid_px_00", "min_price_increment"]] # Now, calculate the spread in number of ticks mbp_df["spread"] = (mbp_df["ask_px_00"] - mbp_df["bid_px_00"]) / mbp_df["min_price_increment"] # Finally, plot the mean spread for each symbol for symbol, data in mbp_df.groupby("raw_symbol"): # Average this data using a 5-minute rolling window mean_spread = ( data["spread"] .ewm( times=data.index, halflife=pd.Timedelta(minutes=5), ) .mean() .reset_index() ) plt.plot( "ts_recv", "spread", data=mean_spread, label=symbol, ) plt.legend() plt.title(f"Overnight Mean Spread - {start.date()}") plt.ylabel("Mean Spread (ticks)") plt.xlabel("Time (UTC)") plt.show() ``` ## Result ```plaintext raw_symbol min_price_increment instrument_id 200251 6BU4 0.00010 201330 6EU4 0.00005 ``` ![Handling tick sizes](https://databento.com/docs/tick-sizes-0.svg.md) # Finding ICE intercommodity spreads ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to find ICE intercommodity futures spreads, where the legs do not all belong to the same asset. Crack spreads are a specific type of intercommodity spread. ICE's [Strategy Code Reference Manual](https://www.ice.com/publicdocs/technology/ICE_Strategy_Code_Reference_Manual.pdf) describes `CRACK` as an exchange-defined spread between two products where one leg is a refined product and the other is an unrefined product. ICE also lists concrete crack spread contracts such as [RBOB Gasoline/Brent Crack Spread](https://www.ice.com/products/2634472/RBOB-Gasoline-Brent-Crack-Spread). We'll request the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) for all symbols in `IFEU.IMPACT`, filter to instruments with `instrument_class` equal to `FUTURE_SPREAD`, and use the leg definition fields to inspect each spread's legs. For each leg, we'll join `leg_instrument_id` back to the definitions to recover the leg instrument's `asset`. The example prints the `raw_symbol` of every futures spread where at least one leg has the target asset `BRN`, and the set of leg assets contains more than one asset. ## Example ```python import databento as db # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "IFEU.IMPACT" target_asset = "BRN" date = "2026-05-26" # Download definition data for all symbols data = client.timeseries.get_range( dataset=dataset, symbols="ALL_SYMBOLS", schema="definition", start=date, ) # Convert to a DataFrame df = data.to_df() # Map each instrument_id to its asset. This lets us look up the asset for each # leg_instrument_id in a spread definition. latest_defs = df.drop_duplicates("instrument_id", keep="last") asset_by_instrument_id = latest_defs.set_index("instrument_id")["asset"] # In the definition schema, each spread has one row for each leg. The primary # fields identify the spread, while fields beginning with leg_ identify the leg. spread_legs = df[ (df["instrument_class"] == db.InstrumentClass.FUTURE_SPREAD) & (df["leg_count"] > 0) ].copy() spread_legs = spread_legs.drop_duplicates( ["instrument_id", "leg_index"], keep="last", ) spread_legs["leg_asset"] = spread_legs["leg_instrument_id"].map( asset_by_instrument_id, ) spread_legs = spread_legs.dropna(subset=["leg_asset"]) def matches_asset_and_is_intercommodity(legs): leg_assets = set(legs["leg_asset"]) return target_asset in leg_assets and len(leg_assets) > 1 matches = spread_legs.groupby("instrument_id").filter( matches_asset_and_is_intercommodity, ) raw_symbols = matches.drop_duplicates("instrument_id")["raw_symbol"].sort_values() print(raw_symbols) ``` ## Result ```plaintext ts_recv 2026-05-26 00:00:00+00:00 ADM FMF0027-BRN FMF0027 2026-05-26 00:00:00+00:00 ADM FMG0027-BRN FMG0027 2026-05-26 00:00:00+00:00 ADM FMH0027-BRN FMH0027 2026-05-26 00:00:00+00:00 ADM FMJ0027-BRN FMJ0027 2026-05-26 00:00:00+00:00 ADM FMK0027-BRN FMK0027 ... 2026-05-26 00:00:00+00:00 WBS FYF0028.Z0028-BRN FYF0028.Z0028 2026-05-26 00:00:00+00:00 WBS FYF0029.Z0029-BRN FYF0029.Z0029 2026-05-26 00:00:00+00:00 WBS FYF0030.Z0030-BRN FYF0030.Z0030 2026-05-26 00:00:00+00:00 WBS FYF0031.Z0031-BRN FYF0031.Z0031 2026-05-26 00:00:00+00:00 WBS FYF0032.Z0032-BRN FYF0032.Z0032 Name: raw_symbol, Length: 436, dtype: str ``` # Displaying fractional prices ## Overview CME quotes its grain and treasury products in fractions rather than decimals. A 10-year treasury note trading at 108.578125 is quoted as `108'185`, i.e. 108 37⁄64, and corn trading at 498.25 is quoted as `498'2`, i.e. 498 2⁄8. In this example we will use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to retrieve instrument definitions and daily bars for several of these products, then render each close price the way the exchange does. ## Definition schema The [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) describes the convention with three fields, which CME documents under [fractional pricing](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457092591/Fractional+Pricing). | Field | Description | | `main_fraction` | The denominator of the main fraction. 255 when unused. | | `sub_fraction` | The denominator of a fraction of one unit of the main fraction. 255 when unused. | | `price_display_format` | The number of digits printed to the right of the tick mark. | `sub_fraction` subdivides one unit of the main fraction, so the product of the two fields give the finest fraction the venue displays. For example, a `main_fraction` of 32 with a `sub_fraction` of 2 displays halves of a 32nd, or 64ths. When `sub_fraction` is unset, the main fraction is the finest unit. | Product | `main_fraction` | `sub_fraction` | `price_display_format` | Decimal price | Quoted | Interpretation | | Corn (`ZCZ6`) | 8 | unset | 1 | 498.25 | `498'2` | 498 2⁄8 | | 30-year bond (`ZBZ6`) | 32 | unset | 2 | 109.5625 | `109'18` | 109 18⁄32 | | 10-year note (`ZNZ6`) | 32 | 2 | 3 | 108.578125 | `108'185` | 108 18⁄32 plus half a 32nd, or 108 37⁄64 | | 2-year note (`ZTZ6`) | 32 | 8 | 3 | 102.9453125 | `102'302` | 102 30⁄32 plus a quarter of a 32nd, or 102 242⁄256 | `price_display_format` is the total number of digits printed after the tick mark ('), including one digit for the subdivision when `sub_fraction` is set. The last digit is a *decimal* rendering of the subdivision rather than its numerator, which is why half of a 32nd prints as `5` in `108'185`. That digit is truncated rather than rounded, so a price of 102.9609375, which is 102 and 30 6⁄8 32nds, renders as `102'307` and not `102'308`. The finest displayed fraction can be finer than the instrument's tick. Corn futures have a `min_price_increment` of 0.25, or two eighths, so only even digits appear in their final position. Products quoted in decimals, such as E-mini S&P 500 futures, leave all three fields unset. > **Info** > > These three fields carry venue-specific meanings. > This example applies to GLBX.MDP3, and other datasets may populate them differently, so check the [venue documentation](https://databento.com/docs/datasets.md) before reusing this logic elsewhere. ## Example ```python import databento as db # The instruments to display. ESU6 is included to show a decimal-quoted product symbols = ["ZCZ6", "ZBZ6", "ZNZ6", "ZTZ6", "ESU6"] dataset = "GLBX.MDP3" date = "2026-08-19" # Value when unset or null UNSET = 255 def display_price(price, main_fraction, sub_fraction, price_display_format): """Render a price in the venue's own quoting convention, such as 108'185.""" if main_fraction == UNSET or price_display_format == UNSET: return f"{price:g}" whole = int(price) # The number of main fractions, such as 32nds, in the remainder. Fractional # prices land on a power-of-two grid, so this arithmetic is exact ticks = (price - whole) * main_fraction main = int(ticks) if sub_fraction == UNSET: return f"{whole}'{main:0{price_display_format}d}" # The last digit is the single-digit decimal of the leftover fraction of one main # fraction, so a half prints as 5 and three quarters as 7 digits = max(price_display_format - 1, 0) leftover = int((ticks - main) * 10) return f"{whole}'{main:0{digits}d}{leftover}" # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, retrieve the instrument definitions definitions_data = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=symbols, start=date, ) # And extract the three fields that describe the display convention definitions_df = definitions_data.to_df() definitions_df = definitions_df.loc[ definitions_df["instrument_class"] == db.InstrumentClass.FUTURE, :, ] conventions = definitions_df.drop_duplicates("raw_symbol").set_index("instrument_id")[ ["raw_symbol", "main_fraction", "sub_fraction", "price_display_format"] ] # Then, request daily bars ohlcv_data = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbols, start=date, ) # And join the bars with the display convention of each instrument df = ohlcv_data.to_df().join(conventions, on="instrument_id").sort_values("raw_symbol") # Finally, render each closing price in the venue's convention df["decimal_close"] = df["close"] df["display_close"] = df.apply( lambda row: display_price( row["close"], row["main_fraction"], row["sub_fraction"], row["price_display_format"], ), axis=1, ) print( df[ [ "raw_symbol", "decimal_close", "main_fraction", "sub_fraction", "price_display_format", "display_close", ] ], ) ``` ## Result ```plaintext raw_symbol decimal_close main_fraction sub_fraction price_display_format display_close ts_event 2026-08-19 00:00:00+00:00 ESU6 7740.000000 255 255 255 7740 2026-08-19 00:00:00+00:00 ZBZ6 109.562500 32 255 2 109'18 2026-08-19 00:00:00+00:00 ZCZ6 498.250000 8 255 1 498'2 2026-08-19 00:00:00+00:00 ZNZ6 108.578125 32 2 3 108'185 2026-08-19 00:00:00+00:00 ZTZ6 102.945312 32 8 3 102'302 ``` # Computing tick value and contract notional ## Overview The value of a tick and the notional value of a contract can be derived from the contract and tick size in the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md). E-mini S&P 500 futures are quoted in index points, crude oil in dollars per barrel, lean hogs in cents per pound, and corn in cents per bushel. In this example we will use the [Historical client](https://databento.com/docs/api-historical/client/historical.md) to derive this information from the definition schema and convert daily closes into contract values. ## Definition schema | Field | Description | | `unit_of_measure` | The unit the contract is measured in, such as `BBL` for barrels or `LBS` for pounds. | | `unit_of_measure_qty` | The contract size, in the `unit_of_measure`. | | `min_price_increment` | The minimum tick size, in price units. | | `min_price_increment_amount` | The value of one tick, with the display factor applied. | | `display_factor` | The multiplier from the venue's display price to the conventional price. | > **Warning** > > CME lists tag 1146, which carries `min_price_increment_amount`, as under development and permits a null value. > Treat the tick value and notional below as a first pass and confirm them against the contract specification for instruments you trade. Three steps take you from these fields to a tick value and a contract value. ``` scale = (min_price_increment_amount / display_factor) / (min_price_increment * unit_of_measure_qty) tick_value = min_price_increment * unit_of_measure_qty * scale notional = close * unit_of_measure_qty * scale ``` `scale` is the multiplier that turns `close * unit_of_measure_qty` into the currency named by `currency`. A value of 1 means the price is already in that currency's major unit per unit of measure, as for E-mini S&P 500 futures at 7740.00 index points with a contract size of 50, giving a notional of 387,000 USD. A value of 0.01 means the price is in hundredths, as for lean hogs at 81.50 cents per pound with a contract size of 40,000 pounds, giving 32,600 USD. The formula for `scale` reconciles CME's published tick value against the contract terms. When the two agree it gives a power of ten. > **Info** > > A `scale` of 0.01 reads as cents when `unit_of_measure` is a physical unit, and as percent of par when it is a currency, as it is for treasuries. ## Fractionally quoted products CME does not publish a usable `min_price_increment_amount` for the products it quotes in fractions. Corn reports 1.0 against a real tick value of 12.50 USD, and the 10-year note reports 0.001 against 15.625 USD. These products have [fractional pricing](https://databento.com/docs/examples/instrument-definitions/fractional-prices.md), and they are quoted in hundredths: grains are cents per bushel and treasuries are percent of par, so a `scale` of 0.01 applies to both. Taking that value gives the correct tick value and notional for both. All fractionally-priced products in CME are listed on CBOT (`exchange=XCBT`), but there are many non-fractionally priced future assets on CBOT. ## Products with a contradictory tick value For some products, CME publishes a `min_price_increment_amount` that contradicts the contract size it accompanies. Silver futures (`SI.FUT`) report 5.00 USD, the tick value of the 1,000-ounce contract (`SIL.FUT`), against a `unit_of_measure_qty` of 5,000 ounces. `min_price_increment` and `unit_of_measure_qty` are correct here. They give a tick value of 25.00 and a notional of 339,275 USD at a close of 67.855. > **Info** > > A `scale` reached this way is an inference. > Exact reconciliation shows that the amount and the contract terms agree, not that either is right. > Check tick values against the venue's contract specifications. ## Validating the result `price_scale` returns `None` when the tick amount is missing, or too far from the contract terms to reconcile. The example covers outright futures, and options and spreads need separate handling. A tick size can change during the day. Other datasets populate the field differently, so check the [venue documentation](https://databento.com/docs/datasets.md) before reusing this logic. ## Example ```python import math import databento as db # A mix of products quoted in the currency's major unit and in cents symbols = [ "ESU6", "CLV6", "GCZ6", "6EU6", "HEV6", "ZLZ6", "MZCZ6", "ZCZ6", "ZNZ6", "SIZ6", ] dataset = "GLBX.MDP3" date = "2026-08-19" # Value when unset or null UNSET = 255 # First, create a historical client client = db.Historical(key="$YOUR_API_KEY") # Next, retrieve the instrument definitions definitions = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=symbols, start=date, ).to_df() # And extract the fields that describe the contract size and the tick definitions = definitions.loc[definitions["instrument_class"] == db.InstrumentClass.FUTURE] contracts = definitions.drop_duplicates("raw_symbol").set_index("instrument_id")[ [ "raw_symbol", "unit_of_measure", "unit_of_measure_qty", "main_fraction", "min_price_increment", "min_price_increment_amount", "display_factor", ] ] # Then, request daily bars bars = client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbols, start=date, ).to_df() # And join the bars with the contract terms of each instrument df = bars.join(contracts, on="instrument_id").sort_values("raw_symbol") # Prices arrive in the currency's major unit or in hundredths of it CONVENTIONS = (1.0, 0.01) def nearest_convention(scale): """Return the convention nearest scale, or None when it is over ten times off.""" convention = min(CONVENTIONS, key=lambda c: abs(math.log10(scale / c))) if abs(math.log10(scale / convention)) >= 1: return None return convention def price_scale( main_fraction, min_price_increment, min_price_increment_amount, display_factor, unit_of_measure_qty, ): """Return the multiplier from price * unit_of_measure_qty to the currency, or None.""" # Fractionally quoted products are priced in hundredths if main_fraction != UNSET: return 0.01 # A scale needs a positive tick, contract size, and tick amount terms = (min_price_increment, unit_of_measure_qty, min_price_increment_amount) if not all(term > 0.0 for term in terms): return None tick_value = min_price_increment_amount / display_factor contract_tick = min_price_increment * unit_of_measure_qty # One tick moves the contract by one tick value scale = tick_value / contract_tick exponent = round(math.log10(scale)) if math.isclose(scale, 10.0**exponent, rel_tol=1e-9): return 10.0**exponent # Otherwise the amount contradicts the contract size, as it does for silver return nearest_convention(scale) # Now, derive the price scale of each contract df["scale"] = df.apply( lambda row: price_scale( row["main_fraction"], row["min_price_increment"], row["min_price_increment_amount"], row["display_factor"], row["unit_of_measure_qty"], ), axis=1, ) # Finally, convert one tick and each close price into currency df["tick_value"] = df["min_price_increment"] * df["unit_of_measure_qty"] * df["scale"] df["notional"] = df["close"] * df["unit_of_measure_qty"] * df["scale"] print( df[ [ "raw_symbol", "unit_of_measure", "unit_of_measure_qty", "close", "tick_value", "scale", "notional", ] ], ) ``` ## Result ```plaintext raw_symbol unit_of_measure unit_of_measure_qty close tick_value scale notional ts_event 2026-08-19 00:00:00+00:00 6EU6 EUR 125000.0 1.168450 6.250 1.00 146056.250 2026-08-19 00:00:00+00:00 CLV6 BBL 1000.0 84.300000 10.000 1.00 84300.000 2026-08-19 00:00:00+00:00 ESU6 IPNT 50.0 7740.000000 12.500 1.00 387000.000 2026-08-19 00:00:00+00:00 GCZ6 TRYOZ 100.0 4569.200000 10.000 1.00 456920.000 2026-08-19 00:00:00+00:00 HEV6 LBS 40000.0 81.500000 10.000 0.01 32600.000 2026-08-19 00:00:00+00:00 MZCZ6 BU 500.0 498.000000 2.500 0.01 2490.000 2026-08-19 00:00:00+00:00 SIZ6 TRYOZ 5000.0 67.855000 25.000 1.00 339275.000 2026-08-19 00:00:00+00:00 ZCZ6 BU 5000.0 498.250000 12.500 0.01 24912.500 2026-08-19 00:00:00+00:00 ZLZ6 LBS 60000.0 69.860000 6.000 0.01 41916.000 2026-08-19 00:00:00+00:00 ZNZ6 USD 100000.0 108.578125 15.625 0.01 108578.125 ``` ## Order book | Title | Subtitle | Image | Supported languages | | [Types of order book events](https://databento.com/docs/examples/order-book/order-actions.md) | Learn how Databento normalizes MBO data from an order's life cycle (e.g., **A**dd, **F**ill). | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, C++, Rust, Historical | | [State management of resting orders](https://databento.com/docs/examples/order-book/order-tracking.md) | Manage state of all orders to calculate BBO. Precursor to order book construction. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, C++, Rust, Historical | | [Limit order book construction](https://databento.com/docs/examples/order-book/limit-order-book.md) | A complete order book implementation. Use MBO data to build the limit order book. | ![example1](https://databento.com/docs/limit-order-book.svg.md) | Python, C++, Rust, Historical | | [Microprice and book imbalance](https://databento.com/docs/examples/order-book/microprice.md) | Common predictors using order book data. | ![example2](https://databento.com/docs/microprice-thumbnail.svg.md) | Python, Historical | | [Queue position of an order](https://databento.com/docs/examples/order-book/queue-position.md) | Use MBO data to get an accurate position of any order within a price level. | ![Image placeholder](https://databento.com/docs/queue-position-thumbnail.svg.md) | Python, Historical | # Types of order book events ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve individual order events and process all events for a single order. This will allow us to track the life cycle of an order, such as when it was added to the order book and when it was removed. ## MBO schemas We'll use the [MBO schema](https://databento.com/docs/schemas/mbo.md) for this example. This is the most granular schema which contains every individual order event keyed by its order ID. ## Example ```cpp #include #include #include #include #include namespace db = databento; int main() { auto client = db::Historical::Builder().SetKey("$YOUR_API_KEY").Build(); // We'll inspect the history of a particular order constexpr auto kOrderId = 6410543150678; auto handle_record = [](const db::Record& record) { const auto& mbo = record.Get(); if (mbo.order_id == kOrderId) { constexpr auto kSep = " "; std::cout << db::ToIso8601(mbo.ts_recv) << kSep << std::setprecision(2) << db::pretty::Px{mbo.price} << kSep << std::setw(2) << std::setfill(' ') << mbo.size << kSep; switch (mbo.action) { case db::Action::Add: std::cout << "[A]dded"; break; case db::Action::Cancel: std::cout << "[C]anceled"; break; case db::Action::Modify: std::cout << "[M]odified"; break; case db::Action::Clear: std::cout << "clea[R]ed"; break; case db::Action::Trade: std::cout << "[T]raded"; break; case db::Action::Fill: std::cout << "[F]illed"; break; case db::Action::None: std::cout << "[N]one"; break; } std::cout << '\n'; } return db::KeepGoing::Continue; }; // First, we request a minute of MBO data client.TimeseriesGetRange("GLBX.MDP3", db::DateTimeRange{"2022-08-26T10:59"}, {"ES.v.0"}, db::Schema::Mbo, db::SType::Continuous, db::SType::InstrumentId, {}, {}, handle_record); return 0; } ``` ## Result ```plaintext 2022-08-26T10:59:37.217088184Z 4184.75 14 [A]dded 2022-08-26T10:59:50.544746948Z 4184.75 7 [F]illed 2022-08-26T10:59:50.544805832Z 4184.75 7 [M]odified 2022-08-26T10:59:50.544805832Z 4184.75 1 [F]illed 2022-08-26T10:59:50.544805832Z 4184.75 6 [M]odified 2022-08-26T10:59:50.544880884Z 4184.75 1 [F]illed 2022-08-26T10:59:50.544880884Z 4184.75 5 [M]odified 2022-08-26T10:59:50.544927788Z 4184.75 1 [F]illed 2022-08-26T10:59:50.544927788Z 4184.75 4 [M]odified 2022-08-26T10:59:50.544927788Z 4184.75 1 [F]illed 2022-08-26T10:59:50.544927788Z 4184.75 3 [M]odified 2022-08-26T10:59:50.545075499Z 4184.75 3 [F]illed 2022-08-26T10:59:50.545093643Z 4184.75 3 [C]anceled ``` # Types of order book events ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve individual order events and process all events for a single order. This will allow us to track the life cycle of an order, such as when it was added to the order book and when it was removed. ## MBO schemas We'll use the [MBO schema](https://databento.com/docs/schemas/mbo.md) for this example. This is the most granular schema which contains every individual order event keyed by its order ID. ## Example ```python import databento as db # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request a minute of MBO data data = client.timeseries.get_range( dataset="GLBX.MDP3", start="2022-08-26T10:59:00", end="2022-08-26T11:00:00", symbols="ES.v.0", stype_in="continuous", schema="mbo", ) df = data.to_df() # Now, we will inspect the history of a particular order ORDER_ID = 6410543150678 order_df = df.loc[df["order_id"] == ORDER_ID] # Finally, we will process each event for this order print(f"events for order_id: {ORDER_ID}") for index, row in order_df.iterrows(): event = (row["price"], row["size"]) action = row["action"] if action == "A": # Add action print(index, *event, "[A]dded", sep="\t") elif action == "C": # Cancel action print(index, *event, "[C]anceled", sep="\t") elif action == "M": # Modify action print(index, *event, "[M]odified", sep="\t") elif action == "R": # Book clear action print(index, *event, "clea[R]ed", sep="\t") elif action == "T": # Trade action print(index, *event, "[T]raded", sep="\t") elif action == "F": # Fill action print(index, *event, "[F]illed", sep="\t") elif action == "N": # None action print(index, *event, "[N]one", sep="\t") ``` ## Result ```plaintext events for order_id: 6410543150678 2022-08-26 10:59:37.217088184+00:00 4184.75 14 [A]dded 2022-08-26 10:59:50.544746948+00:00 4184.75 7 [F]illed 2022-08-26 10:59:50.544805832+00:00 4184.75 7 [M]odified 2022-08-26 10:59:50.544805832+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544805832+00:00 4184.75 6 [M]odified 2022-08-26 10:59:50.544880884+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544880884+00:00 4184.75 5 [M]odified 2022-08-26 10:59:50.544927788+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544927788+00:00 4184.75 4 [M]odified 2022-08-26 10:59:50.544927788+00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544927788+00:00 4184.75 3 [M]odified 2022-08-26 10:59:50.545075499+00:00 4184.75 3 [F]illed 2022-08-26 10:59:50.545093643+00:00 4184.75 3 [C]anceled ``` # Types of order book events ## Overview This example shows how to use the [Historical client](https://databento.com/docs/api-historical/client.md) to retrieve individual order events and process all events for a single order. This will allow us to track the life cycle of an order, such as when it was added to the order book and when it was removed. ## MBO schemas We'll use the [MBO schema](https://databento.com/docs/schemas/whats-a-schema.md) for this example. This is the most granular schema which contains every individual order event keyed by its order ID. ## Example ```rust use databento::{ HistoricalClient, dbn::{Action, MboMsg, SType, Schema, pretty::Px}, historical::timeseries::GetRangeParams, }; use time::macros::datetime; #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; // First, we request a minute of MBO data let mut decoder = client .timeseries() .get_range( &GetRangeParams::builder() .dataset("GLBX.MDP3") .date_time_range( datetime!(2022-08-26 10:59:00 UTC)..datetime!(2022-08-26 11:00:00 UTC), ) .symbols("ES.v.0") .stype_in(SType::Continuous) .schema(Schema::Mbo) .build(), ) .await?; // Now, we will inspect the history of a particular order const ORDER_ID: u64 = 6410543150678; println!("events for order_id: {ORDER_ID}"); while let Some(record) = decoder.decode_record::().await? { if record.order_id != ORDER_ID { continue; } let ts_recv = record.ts_recv().unwrap(); let price = Px(record.price); let size = record.size; match record.action()? { Action::Add => { println!("{ts_recv} {price:.2} {size:2} [A]dded") } Action::Cancel => { println!("{ts_recv} {price:.2} {size:2} [C]anceled") } Action::Modify => { println!("{ts_recv} {price:.2} {size:2} [M]odified") } Action::Clear => { println!("{ts_recv} {price:.2} {size:2} clea[R]ed") } Action::Trade => { println!("{ts_recv} {price:.2} {size:2} [T]raded") } Action::Fill => { println!("{ts_recv} {price:.2} {size:2} [F]illed") } Action::None => { println!("{ts_recv} {price:.2} {size:2} [N]one") } }; } Ok(()) } ``` ## Result ```plaintext events for order_id: 6410543150678 2022-08-26 10:59:37.217088184 +00:00:00 4184.75 14 [A]dded 2022-08-26 10:59:50.544746948 +00:00:00 4184.75 7 [F]illed 2022-08-26 10:59:50.544805832 +00:00:00 4184.75 7 [M]odified 2022-08-26 10:59:50.544805832 +00:00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544805832 +00:00:00 4184.75 6 [M]odified 2022-08-26 10:59:50.544880884 +00:00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544880884 +00:00:00 4184.75 5 [M]odified 2022-08-26 10:59:50.544927788 +00:00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544927788 +00:00:00 4184.75 4 [M]odified 2022-08-26 10:59:50.544927788 +00:00:00 4184.75 1 [F]illed 2022-08-26 10:59:50.544927788 +00:00:00 4184.75 3 [M]odified 2022-08-26 10:59:50.545075499 +00:00:00 4184.75 3 [F]illed 2022-08-26 10:59:50.545093643 +00:00:00 4184.75 3 [C]anceled ``` # State management of resting orders ## Overview This example shows how to track orders and update them according to the different MBO actions. Order tracking is simpler to implement than a full [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md): it keeps track of all resting orders, but not the book levels. The book levels such as the top of book (BBO) can be constructed from a map of resting orders, but it requires inefficiently iterating over all the orders. > **Info** > > This example works for a single instrument. > To use it with multiple instruments, will need to extend it to keep separate books for each instrument. ## Order events | Name | Value | Action | | Add | `A` | Insert a new order into the book. | | Modify | `M` | Change an order's price and/or size. | | Cancel | `C` | Fully or partially cancel an order from the book. | | Clear | `R` | Remove all resting orders for the instrument. | | Trade | `T` | An aggressing order traded. Does not affect the book. | | Fill | `F` | A resting order was filled. Does not affect the book. | | None | `N` | No action: does not affect the book, but may carry `flags` or other information. | Trade and Fill actions do not affect the book because all fills will be accompanied by cancel actions that do update the book. ## Handling FlagSet::IsLast A single event from a publisher can be normalized into multiple records. Databento sets the flag `kLast` to denote the last record for an instrument in a packet. The book state should only be examined _after_ a record with `kLast` set. The `IsLast()` method is used to check the value of the bit flag. ## Example ```cpp #include #include #include #include // kUndefPrice #include // UnixNanos #include // Action, Side #include // HistoricalBuilder #include // Px #include #include #include #include #include namespace db = databento; struct Order { Order() = default; Order(db::Side side, int64_t price, uint32_t size, db::UnixNanos ts_event) : side{side}, price{price}, size{size}, ts_event{ts_event} {} db::Side side{db::Side::None}; int64_t price{db::kUndefPrice}; uint32_t size{}; db::UnixNanos ts_event{}; }; struct PriceLevel { PriceLevel() = default; explicit PriceLevel(const Order& order) : price{order.price}, size{order.size}, count{1} {} int64_t price{db::kUndefPrice}; uint32_t size{}; uint32_t count{}; }; class Book { public: std::pair Bbo() const { PriceLevel best_bid; PriceLevel best_ask; for (const auto& order_pair : orders_) { const auto& order = order_pair.second; PriceLevel* side_best = &best_bid; std::function is_improvement = std::greater{}; if (order.side == db::Side::Ask) { side_best = &best_ask; is_improvement = std::less{}; } if (side_best->price == db::kUndefPrice || is_improvement(order.price, side_best->price)) { *side_best = PriceLevel(order); } else if (side_best->price == order.price) { side_best->size += order.size; } } return {best_bid, best_ask}; } void Apply(db::UnixNanos ts_event, db::Action action, db::Side side, uint64_t order_id, int64_t price, uint32_t size, db::FlagSet flags) { switch (action) { // Trade, Fill, or None: no change case db::Action::Trade: case db::Action::Fill: case db::Action::None: break; case db::Action::Clear: // Clear book: remove all resting orders orders_.clear(); break; case db::Action::Add: { // Add: insert a new order if (flags.IsTob()) { // For top-of-book publishers, remove previous order associated with // this side auto order_it = std::find_if( orders_.begin(), orders_.end(), [side](auto pair) { return pair.second.side == side; }); if (order_it != orders_.end()) { orders_.erase(order_it->first); } // kUndefPrice indicates the price level was removed. There's no new // level to add if (price == db::kUndefPrice) { break; } } orders_[order_id] = Order{side, price, size, ts_event}; break; } case db::Action::Modify: { // Modify: change the price and/or size of a // resting order auto& existing_order = orders_.at(order_id); // The order loses its priority if the price changes or the size // increases if (existing_order.price != price || existing_order.size < size) { existing_order.ts_event = ts_event; } existing_order.size = size; existing_order.price = price; break; } case db::Action::Cancel: { // Cancel: partially or fully cancel some size // from a resting order auto order_it = orders_.find(order_id); if (order_it == orders_.end()) { break; } assert(order_it->second.size >= size); order_it->second.size -= size; // If the full size is cancelled, remove the order from the book if (order_it->second.size == 0) { orders_.erase(order_it); } break; } } } private: std::unordered_map orders_; }; int main() { auto client = db::Historical::Builder().SetKey("$YOUR_API_KEY").Build(); // First, we request MBO data starting from the beginning of pre-market // trading hours Book book; bool book_is_ready = false; client.TimeseriesGetRange( "XNAS.ITCH", {"2022-08-26T08:00:00", "2022-08-26T14:30:00"}, {"MSFT"}, db::Schema::Mbo, [&book, &book_is_ready](const db::Record& rec) { const auto& mbo = rec.Get(); book.Apply(mbo.hd.ts_event, mbo.action, mbo.side, mbo.order_id, mbo.price, mbo.size, mbo.flags); book_is_ready = mbo.flags.IsLast(); return db::KeepGoing::Continue; }); if (book_is_ready) { auto bbo = book.Bbo(); std::cout << "Best ask " << std::setprecision(2) << db::pretty::Px{bbo.second.price} << " x " << bbo.second.size << '\n'; std::cout << "Best bid " << std::setprecision(2) << db::pretty::Px{bbo.first.price} << " x " << bbo.first.size << '\n'; } return 0; } ``` ## Result ```plaintext Best ask 276.07 x 62 Best bid 276.01 x 19 ``` # State management of resting orders ## Overview This example shows how to track orders and update them according to the different MBO actions. Order tracking is simpler to implement than a full [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md): it keeps track of all resting orders, but not the book levels. The book levels such as the top of book (BBO) can be constructed from a map of resting orders, but it requires inefficiently iterating over all the orders. > **Info** > > This example works for a single instrument. > To use it with multiple instruments, you will need to extend it to keep separate books for each instrument. ## Order events | Name | Value | Action | | Add | `A` | Insert a new order into the book. | | Modify | `M` | Change an order's price and/or size. | | Cancel | `C` | Fully or partially cancel an order from the book. | | Clear | `R` | Remove all resting orders for the instrument. | | Trade | `T` | An aggressing order traded. Does not affect the book. | | Fill | `F` | A resting order was filled. Does not affect the book. | | None | `N` | No action: does not affect the book, but may carry `flags` or other information. | Trade and Fill actions do not affect the book because all fills will be accompanied by cancel actions that do update the book. ## Handling F_LAST A single event from a publisher can be normalized into multiple records. Databento sets the flag `F_LAST` to denote the last record for an instrument in an event. The book state should only be examined _after_ a record with `F_LAST` set. ## Example ```python from dataclasses import dataclass, field import databento as db OrderId = int UnixTimestamp = int @dataclass(slots=True) class Order: side: str price: int size: int ts_event: UnixTimestamp @dataclass(slots=True) class PriceLevel: price: int | None = None size: int = 0 count: int = 0 @dataclass(slots=True) class Book: orders: dict[OrderId, Order] = field(default_factory=dict) def bbo(self) -> tuple[PriceLevel, PriceLevel]: best_ask = PriceLevel() best_bid = PriceLevel() for order in self.orders.values(): if order.side == "A": if best_ask.price is None or best_ask.price > order.price: best_ask = PriceLevel( price=order.price, size=order.size, count=1, ) elif best_ask.price == order.price: best_ask.size += order.size best_ask.count += 1 elif order.side == "B": if best_bid.price is None or best_bid.price < order.price: best_bid = PriceLevel( price=order.price, size=order.size, count=1, ) elif best_bid.price == order.price: best_bid.size += order.size best_bid.count += 1 return best_bid, best_ask def apply( self, ts_event: UnixTimestamp, action: str, side: str, order_id: int, price: int, size: int, flags: db.RecordFlags, ) -> None: # Trade, Fill, or None: no change if action in ("T", "F", "N"): return # Clear book: remove all resting orders if action == "R": self.orders.clear() # Add: insert a new order elif action == "A": # For top-of-book publishers, remove previous order associated with this side if flags & db.RecordFlags.F_TOB: self.orders = {i: o for i, o in self.orders.items() if o.side != side} # UNDEF_PRICE indicates the price level was removed. There's no new level to add if price == db.UNDEF_PRICE: return self.orders[order_id] = Order(side, price, size, ts_event) # Cancel: partially or fully cancel some size from a resting order elif action == "C": existing_order = self.orders[order_id] assert existing_order.size >= size existing_order.size -= size # If the full size is cancelled, remove the order from the book if existing_order.size == 0: self.orders.pop(order_id) # Modify: change the price and/or size of a resting order elif action == "M": existing_order = self.orders[order_id] # The order loses its priority if the price changes or the size increases if existing_order.price != price or existing_order.size < size: existing_order.ts_event = ts_event existing_order.size = size existing_order.price = price # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request MBO data starting from the beginning of pre-market trading hours data = client.timeseries.get_range( dataset="XNAS.ITCH", start="2022-08-26T08:00:00", end="2022-08-26T14:30:00", symbols="MSFT", schema="mbo", ) # Then, we replay the data, updating the book with each record book = Book() book_is_ready = False for mbo in data: book.apply(mbo.ts_event, mbo.action, mbo.side, mbo.order_id, mbo.price, mbo.size, mbo.flags) book_is_ready = mbo.flags & db.RecordFlags.F_LAST # Finally we inspect the final BBO if book_is_ready: best_bid, best_ask = book.bbo() print(f"Best ask {float(best_ask.price) / db.FIXED_PRICE_SCALE} × {best_ask.size}") print(f"Best bid {float(best_bid.price) / db.FIXED_PRICE_SCALE} × {best_bid.size}") ``` ## Result ```plaintext Best ask 276.07 × 62 Best bid 276.01 × 19 ``` # State management of resting orders ## Overview This example shows how to track orders and update them according to the different MBO actions. Order tracking is simpler to implement than a full [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md): it keeps track of all resting orders, but not the book levels. The book levels such as the top of book (BBO) can be constructed from a map of resting orders, but it requires inefficiently iterating over all the orders. > **Info** > > This example works for a single instrument. > To use it with multiple instruments, you will need to extend it to keep separate books for each instrument. ## Order events | Name | Value | Action | | Add | `A` | Insert a new order into the book. | | Modify | `M` | Change an order's price and/or size. | | Cancel | `C` | Fully or partially cancel an order from the book. | | Clear | `R` | Remove all resting orders for the instrument. | | Trade | `T` | An aggressing order traded. Does not affect the book. | | Fill | `F` | A resting order was filled. Does not affect the book. | Trade and Fill actions do not affect the book because all fills will be accompanied by cancel actions that do update the book. ## Handling flags::LAST A single event from a publisher can be normalized into multiple records. Databento sets the bit flag `LAST` to denote the last record for an instrument in a packet. The book state should only be examined _after_ a record with `LAST` set, which can be checked with the `is_last()` method. ## Example ```rust use std::collections::HashMap; use databento::{ HistoricalClient, dbn::{Action, Dataset::XnasItch, MboMsg, RecordHeader, Schema, Side, UNDEF_PRICE, pretty::Px}, historical::timeseries::GetRangeParams, }; use time::macros::datetime; #[derive(Default)] struct Book { orders: HashMap, } struct Order { side: Side, price: i64, size: u32, ts_event: u64, } struct PriceLevel { price: i64, size: u32, count: u32, } impl Book { fn bbo(&self) -> (PriceLevel, PriceLevel) { self.orders.values().fold( <(PriceLevel, PriceLevel)>::default(), | | (mut best_bid, mut best_ask), order | { | if order.side == Side::Ask { if best_ask.price == UNDEF_PRICE || best_ask.price > order.price { best_ask = PriceLevel::from(order); } else if best_ask.price == order.price { best_ask.size += order.size; best_ask.count += 1; } } else if best_bid.price == UNDEF_PRICE || best_bid.price < order.price { best_bid = PriceLevel::from(order); } else if best_bid.price == order.price { best_bid.size += order.size; best_bid.count += 1; } (best_bid, best_ask) }, ) } fn apply(&mut self, mbo: MboMsg) -> anyhow::Result<()> { let action = mbo.action()?; let side = mbo.side()?; let MboMsg { hd: RecordHeader { ts_event, .. }, order_id, price, size, flags, .. } = mbo; match action { // Trade, Fill, or None: no change Action::Trade | Action::Fill | Action::None => (), // Clear book: remove all resting orders Action::Clear => self.orders.clear(), Action::Add => { // Add: insert a new order if flags.is_tob() { // For top-of-book publishers, remove previous order associated with this side self.orders.retain(|_, order| order.side != side); // UNDEF_PRICE indicates the price level was removed. There's // no new level to add if price == UNDEF_PRICE { return Ok(()); } } self.orders.insert( order_id, Order { side, price, size, ts_event, }, ); } Action::Cancel => { // Cancel: partially or fully cancel some size from a resting order let existing_order = self.orders.get_mut(&order_id).unwrap(); assert!(existing_order.size >= size); existing_order.size -= size; // If the full size is cancelled, remove the // order from the book if existing_order.size == 0 { self.orders.remove(&order_id); } } Action::Modify => { // Modify: change the price and/or size of a resting order let existing_order = self.orders.get_mut(&order_id).unwrap(); // The order loses its priority if the price changes or the size increases if existing_order.price != price || existing_order.size < size { existing_order.ts_event = ts_event; } existing_order.size = size; existing_order.price = price; } } Ok(()) } } #[tokio::main] async fn main() -> anyhow::Result<()> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; // First, we request MBO data starting from the beginning of pre-market trading hours let mut decoder = client .timeseries() .get_range( &GetRangeParams::builder() .dataset(XnasItch) .date_time_range( datetime!(2022-08-26 08:00:00 UTC)..datetime!(2022-08-26 14:30:00 UTC), ) .symbols("MSFT") .schema(Schema::Mbo) .build(), ) .await?; // Then, we replay the data, updating the book with each record let mut book = Book::default(); let mut book_is_ready = false; while let Some(mbo) = decoder.decode_record::().await? { book.apply(mbo.clone())?; book_is_ready = mbo.flags.is_last(); } if book_is_ready { let (best_bid, best_ask) = book.bbo(); println!("Best ask {:.2} × {}", Px(best_ask.price), best_ask.size); println!("Best bid {:.2} × {}", Px(best_bid.price), best_bid.size); } Ok(()) } impl Default for PriceLevel { fn default() -> Self { Self { price: UNDEF_PRICE, size: 0, count: 0, } } } impl From<&Order> for PriceLevel { fn from(order: &Order) -> Self { Self { price: order.price, size: order.size, count: 1, } } } ``` ## Result ```plaintext Best ask 276.07 × 62 Best bid 276.01 × 19 ``` # Limit order book construction ## Overview This example demonstrates a complete limit order book implementation for Databento's MBO normalization schema. It handles multiple instruments, here `GOOG` and `GOOGL`, across multiple venues that make up Databento Equities Basic: - Investors Exchange (`IEXG`) - NYSE Texas (`XCHI`) - NYSE National (`XCIS`) We use `publisher_id` to disambiguate between venues. While this example uses the Historical API, the same implementation can be used with live data by forwarding every `MBOMsg` to `market.Apply`. > **See also** > > It's recommended to first start with the [Order tracking example](https://databento.com/docs/examples/order-book/order-tracking.md#order-events), which explains each event more in depth. ## Design The `unordered_map` stores the price and side of each order for quick retrieval by ID. Each side of the book is a `map` to balance random access of a price's level with retrieving the lowest and highest price level for determining the BBO. The values in the book map are `vector`s of orders sorted by priority. Investors Exchange and NYSE National only provide top-of-book information meaning their MBO data is [normalized differently](https://databento.com/docs/conventions/common-fields-enums-types.md#top-of-book-datasets). Each **A**dd message indicates the new top-of-book and replaces all levels on that side. Add messages with an `kUndefPrice` price indicate there's no depth for that side of the book. The final special case comes when calculating the number of orders for a level since top-of-book updates don't correspond with individual orders. ## Example ```cpp #include #include #include // dataset, kUndefPrice #include // ToIso8601, UnixNanos #include #include // Action, Side #include #include // HistoricalBuilder #include // Px #include // BidAskPair, MboMsg, Record #include // TsSymbolMap #include #include #include #include #include #include #include #include #include namespace db = databento; struct PriceLevel { int64_t price{db::kUndefPrice}; uint32_t size{0}; uint32_t count{0}; bool IsEmpty() const { return price == db::kUndefPrice; } operator bool() const { return !IsEmpty(); } }; std::ostream& operator<<(std::ostream& stream, const PriceLevel& level) { stream << level.size << " @ " << db::pretty::Px{level.price} << " | " << level.count << " order(s)"; return stream; } class Book { public: std::pair Bbo() const { return {GetBidLevel(), GetAskLevel()}; } PriceLevel GetBidLevel(std::size_t idx = 0) const { if (bids_.size() > idx) { // Reverse iterator to get highest bid prices first auto level_it = bids_.rbegin(); std::advance(level_it, idx); return GetPriceLevel(level_it->first, level_it->second); } return PriceLevel{}; } PriceLevel GetAskLevel(std::size_t idx = 0) const { if (offers_.size() > idx) { auto level_it = offers_.begin(); std::advance(level_it, idx); return GetPriceLevel(level_it->first, level_it->second); } return PriceLevel{}; } PriceLevel GetBidLevelByPx(int64_t px) const { auto level_it = bids_.find(px); if (level_it == bids_.end()) { throw std::invalid_argument{"No bid level at " + db::pretty::PxToString(px)}; } return GetPriceLevel(px, level_it->second); } PriceLevel GetAskLevelByPx(int64_t px) const { auto level_it = offers_.find(px); if (level_it == offers_.end()) { throw std::invalid_argument{"No ask level at " + db::pretty::PxToString(px)}; } return GetPriceLevel(px, level_it->second); } const db::MboMsg& GetOrder(uint64_t order_id) { auto order_it = orders_by_id_.find(order_id); if (order_it == orders_by_id_.end()) { throw std::invalid_argument{"No order with ID " + std::to_string(order_id)}; } auto& level = GetLevel(order_it->second.side, order_it->second.price); return *GetLevelOrder(level, order_id); } uint32_t GetQueuePos(uint64_t order_id) { auto order_it = orders_by_id_.find(order_id); if (order_it == orders_by_id_.end()) { throw std::invalid_argument{"No order with ID " + std::to_string(order_id)}; } const auto& level_it = GetLevel(order_it->second.side, order_it->second.price); uint32_t prior_size = 0; for (const auto& order : level_it) { if (order.order_id == order_id) { break; } prior_size += order.size; } return prior_size; } std::vector GetSnapshot(std::size_t level_count = 1) const { std::vector res; for (size_t i = 0; i < level_count; ++i) { db::BidAskPair ba_pair{db::kUndefPrice, db::kUndefPrice, 0, 0, 0, 0}; auto bid = GetBidLevel(i); if (bid) { ba_pair.bid_px = bid.price; ba_pair.bid_sz = bid.size; ba_pair.bid_ct = bid.count; } auto ask = GetAskLevel(i); if (ask) { ba_pair.ask_px = ask.price; ba_pair.ask_sz = ask.size; ba_pair.ask_ct = ask.count; } res.emplace_back(ba_pair); } return res; } void Apply(const db::MboMsg& mbo) { switch (mbo.action) { case db::Action::Clear: { Clear(); break; } case db::Action::Add: { Add(mbo); break; } case db::Action::Cancel: { Cancel(mbo); break; } case db::Action::Modify: { Modify(mbo); break; } case db::Action::Trade: case db::Action::Fill: case db::Action::None: { break; } default: { throw std::invalid_argument{std::string{"Unknown action: "} + db::ToString(mbo.action)}; } } } private: using LevelOrders = std::vector; struct PriceAndSide { int64_t price; db::Side side; }; using Orders = std::unordered_map; using SideLevels = std::map; static PriceLevel GetPriceLevel(int64_t price, const LevelOrders& level) { PriceLevel res{price}; for (const auto& order : level) { if (!order.flags.IsTob()) { ++res.count; } res.size += order.size; } return res; } static LevelOrders::iterator GetLevelOrder(LevelOrders& level, uint64_t order_id) { auto order_it = std::find_if(level.begin(), level.end(), [order_id](const db::MboMsg& order) { return order.order_id == order_id; }); if (order_it == level.end()) { throw std::invalid_argument{"No order with ID " + std::to_string(order_id)}; } return order_it; } void Clear() { orders_by_id_.clear(); offers_.clear(); bids_.clear(); } void Add(db::MboMsg mbo) { if (mbo.flags.IsTob()) { SideLevels& levels = GetSideLevels(mbo.side); levels.clear(); // kUndefPrice indicates the side's book should be cleared // and doesn't represent an order that should be added if (mbo.price != db::kUndefPrice) { LevelOrders level = {mbo}; levels.emplace(mbo.price, level); } } else { LevelOrders& level = GetOrInsertLevel(mbo.side, mbo.price); level.emplace_back(mbo); auto res = orders_by_id_.emplace(mbo.order_id, PriceAndSide{mbo.price, mbo.side}); if (!res.second) { throw std::invalid_argument{"Received duplicated order ID " + std::to_string(mbo.order_id)}; } } } void Cancel(db::MboMsg mbo) { LevelOrders& level = GetLevel(mbo.side, mbo.price); auto order_it = GetLevelOrder(level, mbo.order_id); if (order_it->size < mbo.size) { throw std::logic_error{ "Tried to cancel more size than existed for order ID " + std::to_string(mbo.order_id)}; } order_it->size -= mbo.size; if (order_it->size == 0) { orders_by_id_.erase(mbo.order_id); level.erase(order_it); if (level.empty()) { RemoveLevel(mbo.side, mbo.price); } } } void Modify(db::MboMsg mbo) { auto price_side_it = orders_by_id_.find(mbo.order_id); if (price_side_it == orders_by_id_.end()) { // If order not found, treat it as an add Add(mbo); return; } if (price_side_it->second.side != mbo.side) { throw std::logic_error{"Order " + std::to_string(mbo.order_id) + " changed side"}; } auto prev_price = price_side_it->second.price; LevelOrders& prev_level = GetLevel(mbo.side, prev_price); auto level_order_it = GetLevelOrder(prev_level, mbo.order_id); if (prev_price != mbo.price) { price_side_it->second.price = mbo.price; prev_level.erase(level_order_it); if (prev_level.empty()) { RemoveLevel(mbo.side, prev_price); } LevelOrders& level = GetOrInsertLevel(mbo.side, mbo.price); // Changing price loses priority level.emplace_back(mbo); } else if (level_order_it->size < mbo.size) { LevelOrders& level = prev_level; // Increasing size loses priority level.erase(level_order_it); level.emplace_back(mbo); } else { level_order_it->size = mbo.size; } } SideLevels& GetSideLevels(db::Side side) { switch (side) { case db::Side::Ask: { return offers_; } case db::Side::Bid: { return bids_; } case db::Side::None: default: { throw std::invalid_argument{"Invalid side"}; } } } LevelOrders& GetLevel(db::Side side, int64_t price) { SideLevels& levels = GetSideLevels(side); auto level_it = levels.find(price); if (level_it == levels.end()) { throw std::invalid_argument{ std::string{"Received event for unknown level "} + db::ToString(side) + " " + db::pretty::PxToString(price)}; } return level_it->second; } LevelOrders& GetOrInsertLevel(db::Side side, int64_t price) { SideLevels& levels = GetSideLevels(side); return levels[price]; } void RemoveLevel(db::Side side, int64_t price) { SideLevels& levels = GetSideLevels(side); levels.erase(price); } Orders orders_by_id_; SideLevels offers_; SideLevels bids_; }; class Market { public: struct PublisherBook { uint16_t publisher_id; Book book; }; const std::vector& GetBooksByPub(uint32_t instrument_id) { return books_[instrument_id]; } const Book& GetBook(uint32_t instrument_id, uint16_t publisher_id) { const std::vector& books = GetBooksByPub(instrument_id); auto book_it = std::find_if(books.begin(), books.end(), [publisher_id](const PublisherBook& pub_book) { return pub_book.publisher_id == publisher_id; }); if (book_it == books.end()) { throw std::invalid_argument{"No book for publisher ID " + std::to_string(publisher_id)}; } return book_it->book; } std::pair Bbo(uint32_t instrument_id, uint16_t publisher_id) { const auto& book = GetBook(instrument_id, publisher_id); return book.Bbo(); } std::pair AggregatedBbo(uint32_t instrument_id) { PriceLevel agg_bid; PriceLevel agg_ask; for (const auto& pub_book : GetBooksByPub(instrument_id)) { const auto bbo = pub_book.book.Bbo(); const auto& bid = bbo.first; const auto& ask = bbo.second; if (bid) { if (agg_bid.IsEmpty() || bid.price > agg_bid.price) { agg_bid = bid; } else if (bid.price == agg_bid.price) { agg_bid.count += bid.count; agg_bid.size += bid.size; } } if (ask) { if (agg_ask.IsEmpty() || ask.price < agg_ask.price) { agg_ask = ask; } else if (ask.price == agg_ask.price) { agg_ask.count += ask.count; agg_ask.size += ask.size; } } } return {agg_bid, agg_ask}; } void Apply(const db::MboMsg& mbo_msg) { auto& instrument_books = books_[mbo_msg.hd.instrument_id]; auto book_it = std::find_if(instrument_books.begin(), instrument_books.end(), [&mbo_msg](const PublisherBook& pub_book) { return pub_book.publisher_id == mbo_msg.hd.publisher_id; }); if (book_it == instrument_books.end()) { instrument_books.emplace_back(PublisherBook{mbo_msg.hd.publisher_id, {}}); book_it = std::prev(instrument_books.end()); } book_it->book.Apply(mbo_msg); } private: std::unordered_map> books_; }; int main() { // First, create a historical client auto client = db::Historical::Builder().SetKey("$YOUR_API_KEY").Build(); // Next, we'll set up the books and book handlers Market market; // We'll parse symbology from the DBN metadata db::TsSymbolMap symbol_map; auto metadata_callback = [&symbol_map](db::Metadata metadata) { symbol_map = metadata.CreateSymbolMap(); }; // For each book update... auto record_callback = [&market, &symbol_map](const db::Record& record) { if (auto* mbo = record.GetIf()) { // We'll apply market.Apply(*mbo); // If it's the last update in an event, print the state of the aggregated // book if (mbo->flags.IsLast()) { const auto& symbol = symbol_map.At(*mbo); auto bbo = market.AggregatedBbo(mbo->hd.instrument_id); std::cout << symbol << " Aggregated BBO | " << db::ToIso8601(mbo->ts_recv) << "\n " << bbo.second << "\n " << bbo.first << '\n'; } } return db::KeepGoing::Continue; }; auto file_path = "dbeq-basic-20240403.mbo.dbn.zst"; // Check if file exists auto file_store = std::ifstream(file_path).good() // And open it if it does ? db::DbnStore{file_path} // Or we'll request the data starting from the beginning // of pre-market trading hours : client.TimeseriesGetRangeToFile( db::dataset::kDbeqBasic, {"2024-04-03T08:00:00", "2024-04-03T14:00:00"}, {"GOOG", "GOOGL"}, db::Schema::Mbo, file_path); // Finally, we'll replay each book update file_store.Replay(metadata_callback, record_callback); return 0; } ``` ## Results ```plaintext GOOGL Aggregated BBO | 2024-04-03T11:00:00.903881953Z 0 @ kUndefPrice | 0 order(s) 200 @ 152.800000000 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03T11:00:00.903887026Z 200 @ 155.200000000 | 1 order(s) 200 @ 152.800000000 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03T11:00:00.914731635Z 200 @ 155.200000000 | 1 order(s) 200 @ 152.800000000 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03T11:00:00.925739296Z 200 @ 155.200000000 | 1 order(s) 200 @ 152.800000000 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03T11:00:00.925744379Z 200 @ 155.200000000 | 1 order(s) 200 @ 152.800000000 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03T11:03:55.585836619Z 200 @ 155.200000000 | 1 order(s) 200 @ 152.800000000 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03T11:03:55.585841782Z 200 @ 155.400000000 | 1 order(s) 200 @ 152.800000000 | 1 order(s) ... ``` # Limit order book construction ## Overview This example demonstrates a complete limit order book implementation for Databento's MBO normalization schema. It handles multiple instruments, here `GOOG` and `GOOGL`, across multiple venues that make up Databento Equities Basic: - Investors Exchange (`IEXG`) - NYSE Texas (`XCHI`) - NYSE National (`XCIS`) We use `publisher_id` to disambiguate between venues. While this example uses the Historical API, the same implementation can be used with live data by forwarding every `MBOMsg` to `market.apply`. > **See also** > > It's recommended to first start with the [Order tracking example](https://databento.com/docs/examples/order-book/order-tracking.md#order-events), which explains each event more in depth. ## Design Orders are stored in a `dict` for quick retrieval. Each side of the book is a `SortedDict` to support both random access by price to add, modify, or cancel an order, as well as retrieving the lowest or highest price level for determining the BBO. The values in the `SortedDict` represent one level and are lists of orders sorted by priority. Investors Exchange and NYSE National only provide top-of-book information meaning their MBO data is [normalized differently](https://databento.com/docs/conventions/common-fields-enums-types.md#top-of-book-datasets). Each **A**dd message indicates the new top-of-book and replaces all levels on that side. Add messages with an `UNDEF_PRICE` price indicate there's no depth for that side of the book. The final special case comes when calculating the number of orders for a level since top-of-book updates don't correspond with individual orders. ## Dependencies This example takes advantage of the [sortedcontainers](https://grantjenks.com/docs/sortedcontainers/) package for performant lists and dicts. The `SortedDict` hybrid data structures balances the performance of random access to handle order book events with the typical pattern of checking the top of book. ## Example ```python from __future__ import annotations import os from collections import defaultdict from dataclasses import dataclass, field from itertools import takewhile import databento as db from sortedcontainers import SortedDict @dataclass(slots=True) class PriceLevel: price: int size: int = 0 count: int = 0 def __str__(self) -> str: price = self.price / db.FIXED_PRICE_SCALE return f"{self.size:4} @ {price:6.2f} | {self.count:2} order(s)" @dataclass(slots=True) class LevelOrders: price: int orders: list[db.MBOMsg] = field(default_factory=list, compare=False) def __bool__(self) -> bool: return bool(self.orders) @property def level(self) -> PriceLevel: return PriceLevel( price=self.price, count=sum(1 for o in self.orders if not o.flags & db.RecordFlags.F_TOB), size=sum(o.size for o in self.orders), ) @dataclass(slots=True) class Book: orders_by_id: dict[int, db.MBOMsg] = field(default_factory=dict) offers: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) bids: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) def bbo(self) -> tuple[PriceLevel | None, PriceLevel | None]: return self.get_bid_level(), self.get_ask_level() def get_bid_level(self, idx: int = 0) -> PriceLevel | None: if self.bids and len(self.bids) > idx: # Reverse for bids to get highest prices first return self.bids.peekitem(-(idx + 1))[1].level return None def get_ask_level(self, idx: int = 0) -> PriceLevel | None: if self.offers and len(self.offers) > idx: return self.offers.peekitem(idx)[1].level return None def get_bid_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "B").level except KeyError: return None def get_ask_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "A").level except KeyError: return None def get_order(self, id: int) -> db.MBOMsg | None: return self.orders_by_id.get(id) def get_queue_pos(self, id: int) -> int | None: order = self.get_order(id) if not order: return None level = self._get_level(order.price, order.side) return sum(order.size for order in takewhile(lambda o: o.order_id != id, level.orders)) def get_snapshot(self, level_count: int = 1) -> list[db.BidAskPair]: snapshots = [] for level in range(level_count): ba_pair = db.BidAskPair() bid = self.get_bid_level(level) if bid: ba_pair.bid_px = bid.price ba_pair.bid_sz = bid.size ba_pair.bid_ct = bid.count ask = self.get_ask_level(level) if ask: ba_pair.ask_px = ask.price ba_pair.ask_sz = ask.size ba_pair.ask_ct = ask.count snapshots.append(ba_pair) return snapshots def apply(self, mbo: db.MBOMsg) -> None: # Trade, Fill, or None: no change if mbo.action in ("T", "F", "N"): return # Clear book: remove all resting orders if mbo.action == "R": self._clear() return # side=N is only valid with Trade, Fill, and Clear actions assert mbo.side in ("A", "B") # UNDEF_PRICE indicates the book level should be removed if mbo.price == db.UNDEF_PRICE and mbo.flags & db.RecordFlags.F_TOB: self._side_levels(mbo.side).clear() return # Add: insert a new order if mbo.action == "A": self._add(mbo) # Cancel: partially or fully cancel some size from a resting order elif mbo.action == "C": self._cancel(mbo) # Modify: change the price and/or size of a resting order elif mbo.action == "M": self._modify(mbo) else: raise ValueError(f"Unknown action={mbo.action}") def _clear(self) -> None: self.orders_by_id.clear() self.offers.clear() self.bids.clear() def _add(self, mbo: db.MBOMsg) -> None: if mbo.flags & db.RecordFlags.F_TOB: levels = self._side_levels(mbo.side) levels.clear() levels[mbo.price] = LevelOrders(price=mbo.price, orders=[mbo]) else: level = self._get_or_insert_level(mbo.price, mbo.side) assert mbo.order_id not in self.orders_by_id self.orders_by_id[mbo.order_id] = mbo level.orders.append(mbo) def _cancel(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id[mbo.order_id] level = self._get_level(mbo.price, mbo.side) assert order.size >= mbo.size order.size -= mbo.size # If the full size is cancelled, remove the order from the book if order.size == 0: self.orders_by_id.pop(mbo.order_id) level.orders.remove(order) # If the level is now empty, remove it from the book if not level: self._remove_level(mbo.price, mbo.side) def _modify(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id.get(mbo.order_id) if order is None: # If order not found, treat it as an add self._add(mbo) return assert order.side == mbo.side, f"Order {order} changed side to {mbo.side}" level = self._get_level(order.price, order.side) if order.price != mbo.price: # Changing price loses priority level.orders.remove(order) if not level: self._remove_level(order.price, mbo.side) level = self._get_or_insert_level(mbo.price, mbo.side) level.orders.append(mbo) elif order.size < mbo.size: # Increasing size loses priority level.orders.remove(order) level.orders.append(mbo) else: # Update in place level.orders[level.orders.index(order)] = mbo self.orders_by_id[mbo.order_id] = mbo def _side_levels(self, side: str) -> SortedDict: if side == "A": return self.offers if side == "B": return self.bids raise ValueError(f"Invalid {side =}") def _get_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price not in levels: raise KeyError(f"No price level found for {price =} and {side =}") return levels[price] def _get_or_insert_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price in levels: return levels[price] level = LevelOrders(price=price) levels[price] = level return level def _remove_level(self, price: int, side: str) -> None: levels = self._side_levels(side) levels.pop(price) @dataclass(slots=True) class Market: books: defaultdict[int, defaultdict[int, Book]] = field( default_factory=lambda: defaultdict(lambda: defaultdict(Book)), ) def get_books_by_pub(self, instrument_id: int) -> defaultdict[int, Book]: return self.books[instrument_id] def get_book(self, instrument_id: int, publisher_id: int) -> Book: return self.books[instrument_id][publisher_id] def bbo( self, instrument_id: int, publisher_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: return self.books[instrument_id][publisher_id].bbo() def aggregated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: agg_bbo: list[PriceLevel | None] = [None, None] # max for bids, min for asks for idx, reducer in [(0, max), (1, min)]: all_best = [b.bbo()[idx] for b in self.books[instrument_id].values()] all_best = [b for b in all_best if b] if not all_best: continue best_price = reducer(b.price for b in all_best) best = [b for b in all_best if b.price == best_price] agg_bbo[idx] = PriceLevel( price=best_price, size=sum(b.size for b in best), count=sum(b.count for b in best), ) return tuple(agg_bbo) def apply(self, mbo: db.MBOMsg) -> None: book = self.books[mbo.instrument_id][mbo.publisher_id] book.apply(mbo) if __name__ == "__main__": # First, create a historical client client = db.Historical("$YOUR_API_KEY") # Next, we will request MBO data starting from the beginning of pre-market trading hours # or load the file if we've already downloaded it. data_path = "dbeq-basic-20240403.mbo.dbn.zst" if os.path.exists(data_path): data = db.DBNStore.from_file(data_path) else: data = client.timeseries.get_range( dataset="DBEQ.BASIC", start="2024-04-03T08:00:00", end="2024-04-03T14:00:00", symbols=["GOOG", "GOOGL"], schema="mbo", path=data_path, ) # Then we parse the symbology into a more usable format instrument_map = db.common.symbology.InstrumentMap() instrument_map.insert_metadata(data.metadata) # Finally, we iterate over each book update market = Market() for mbo in data: # And apply it market.apply(mbo) # If it's the last update in an event, print the state of the aggregated book if mbo.flags & db.RecordFlags.F_LAST: symbol = instrument_map.resolve(mbo.instrument_id, mbo.pretty_ts_recv.date()) or "" print(f"{symbol} Aggregated BBO | {mbo.pretty_ts_recv}") best_bid, best_offer = market.aggregated_bbo(mbo.instrument_id) print(f" {best_offer}") print(f" {best_bid}") ``` ## Results ```plaintext GOOGL Aggregated BBO | 2024-04-03 11:00:00.903881953+00:00 None 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.903887026+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.914731635+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.925739296+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.925744379+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:03:55.585836619+00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:03:55.585841782+00:00 200 @ 155.40 | 1 order(s) 200 @ 152.80 | 1 order(s) ... ``` # Limit order book construction ## Overview This example demonstrates a complete limit order book implementation for Databento's MBO normalization schema. It handles multiple instruments, here `GOOG` and `GOOGL`, across multiple venues that make up Databento Equities Basic: - Investors Exchange (`IEXG`) - NYSE Texas (`XCHI`) - NYSE National (`XCIS`) We use `publisher_id` to disambiguate between venues. While this example uses the Historical API, the same implementation can be used with live data by forwarding every `MBOMsg` to `market.apply`. > **See also** > > It's recommended to first start with the [Order tracking example](https://databento.com/docs/examples/order-book/order-tracking.md#order-events), which explains each event more in depth. ## Design The `HashMap` stores the price and side of each order for quick retrieval by ID. Each side of the book is a `BTreeMap` to balance random access of a price's level with retrieving the lowest and highest price level for determining the BBO. The values in the book map are `VecDeque`s of orders sorted by priority. Investors Exchange and NYSE National only provide top-of-book information meaning their MBO data is [normalized differently](https://databento.com/docs/conventions/common-fields-enums-types.md#top-of-book-datasets). Each **A**dd message indicates the new top-of-book and replaces all levels on that side. Add messages with an `UNDEF_PRICE` price indicate there's no depth for that side of the book. The final special case comes when calculating the number of orders for a level since top-of-book updates don't correspond with individual orders. ## Example ```rust use std::{ collections::{BTreeMap, HashMap, VecDeque}, fmt::Display, }; use databento::{ HistoricalClient, dbn::{ Action, BidAskPair, Dataset, MboMsg, Publisher, Record, Schema, Side, SymbolIndex, UNDEF_PRICE, decode::{AsyncDbnDecoder, DbnMetadata}, pretty, }, historical::timeseries::GetRangeToFileParams, }; use time::macros::datetime; use tokio::fs; #[tokio::main] async fn main() -> Result<(), databento::Error> { let mut client = HistoricalClient::builder().key_from_env()?.build()?; let mut market = Market::default(); let path = "dbeq-basic-20240403.mbo.dbn.zst"; if !fs::try_exists(path).await? { client .timeseries() .get_range_to_file( &GetRangeToFileParams::builder() .dataset(Dataset::DbeqBasic) .symbols(vec!["GOOG", "GOOGL"]) .date_time_range( datetime!(2024-04-03 08:00:00 UTC)..datetime!(2024-04-03 14:00:00 UTC), ) .schema(Schema::Mbo) .path(path) .build(), ) .await?; }; let mut decoder = AsyncDbnDecoder::from_zstd_file(path).await?; let symbol_map = decoder.metadata().symbol_map()?; while let Some(mbo) = decoder.decode_record::().await? { market.apply(mbo.clone()); // If it's the last update in an event, print the state of the aggregated book if mbo.flags.is_last() { let symbol = symbol_map.get_for_rec(mbo).unwrap(); let (best_bid, best_offer) = market.aggregated_bbo(mbo.hd.instrument_id); println!("{symbol} Aggregated BBO | {}", mbo.ts_recv().unwrap()); if let Some(best_offer) = best_offer { println!(" {best_offer}"); } else { println!(" None"); } if let Some(best_bid) = best_bid { println!(" {best_bid}"); } else { println!(" None"); } } } Ok(()) } #[derive(Debug, Default)] pub struct Market { books: HashMap>, } #[derive(Debug, Default)] pub struct Book { orders_by_id: HashMap, offers: BTreeMap, bids: BTreeMap, } #[derive(Debug, Clone)] pub struct PriceLevel { pub price: i64, pub size: u32, pub count: u32, } type Level = VecDeque; impl Market { pub fn new() -> Self { Self::default() } pub fn books_by_pub(&self, instrument_id: u32) -> Option<&[(Publisher, Book)]> { self.books .get(&instrument_id) .map(|pub_books| pub_books.as_slice()) } pub fn book(&self, instrument_id: u32, publisher: Publisher) -> Option<&Book> { let books = self.books.get(&instrument_id)?; books.iter().find_map(|(book_pub, book)| { if *book_pub == publisher { Some(book) } else { None } }) } pub fn bbo( &self, instrument_id: u32, publisher: Publisher, ) -> (Option, Option) { self.book(instrument_id, publisher) .map(|book| book.bbo()) .unwrap_or_default() } pub fn aggregated_bbo(&self, instrument_id: u32) -> (Option, Option) { let mut agg_bid = None; let mut agg_ask = None; let Some(books_by_pub) = self.books_by_pub(instrument_id) else { return (None, None); }; for (_, book) in books_by_pub.iter() { let (bid, ask) = book.bbo(); if let Some(bid) = bid { match &mut agg_bid { None => agg_bid = Some(bid), Some(ab) if bid.price > ab.price => agg_bid = Some(bid), Some(ab) if bid.price == ab.price => { ab.size += bid.size; ab.count += bid.count; } Some(_) => {} } } if let Some(ask) = ask { match &mut agg_ask { None => agg_ask = Some(ask), Some(aa) if ask.price < aa.price => agg_ask = Some(ask), Some(aa) if ask.price == aa.price => { aa.size += ask.size; aa.count += ask.count; } Some(_) => {} } } } (agg_bid, agg_ask) } pub fn apply(&mut self, mbo: MboMsg) { let publisher = mbo.publisher().unwrap(); let books = self.books.entry(mbo.hd.instrument_id).or_default(); let book = if let Some((_, book)) = books .iter_mut() .find(|(book_pub, _)| *book_pub == publisher) { book } else { books.push((publisher, Book::default())); &mut books.last_mut().unwrap().1 }; book.apply(mbo); } } impl Book { pub fn new() -> Self { Self::default() } pub fn bbo(&self) -> (Option, Option) { (self.bid_level(0), self.ask_level(0)) } pub fn bid_level(&self, idx: usize) -> Option { self.bids .iter() // Reverse to get highest first .rev() .nth(idx) .map(|(price, orders)| PriceLevel::new(*price, orders.iter())) } pub fn ask_level(&self, idx: usize) -> Option { self.offers .iter() .nth(idx) .map(|(price, orders)| PriceLevel::new(*price, orders.iter())) } pub fn bid_level_by_px(&self, px: i64) -> Option { self.bids .get(&px) .map(|orders| PriceLevel::new(px, orders.iter())) } pub fn ask_level_by_px(&self, px: i64) -> Option { self.offers .get(&px) .map(|orders| PriceLevel::new(px, orders.iter())) } pub fn order(&self, order_id: u64) -> Option<&MboMsg> { let (side, price) = self.orders_by_id.get(&order_id)?; let levels = self.side_levels(*side); let level = levels.get(price)?; level.iter().find(|order| order.order_id == order_id) } pub fn queue_pos(&self, order_id: u64) -> Option { let (side, price) = self.orders_by_id.get(&order_id)?; let levels = self.side_levels(*side); let level = levels.get(price)?; Some( level .iter() .take_while(|order| order.order_id != order_id) .fold(0, |acc, order| acc + order.size), ) } pub fn snapshot(&self, level_count: usize) -> Vec { (0..level_count) .map(|i| { let mut ba_pair = BidAskPair::default(); if let Some(bid) = self.bid_level(i) { ba_pair.bid_px = bid.price; ba_pair.bid_sz = bid.size; ba_pair.bid_ct = bid.count; } if let Some(ask) = self.ask_level(i) { ba_pair.ask_px = ask.price; ba_pair.ask_sz = ask.size; ba_pair.ask_ct = ask.count; } ba_pair }) .collect() } pub fn apply(&mut self, mbo: MboMsg) { let action = mbo.action().unwrap(); match action { Action::Modify => self.modify(mbo), Action::Trade | Action::Fill | Action::None => {} Action::Cancel => self.cancel(mbo), Action::Add => self.add(mbo), Action::Clear => self.clear(), } } fn clear(&mut self) { self.orders_by_id.clear(); self.offers.clear(); self.bids.clear(); } fn add(&mut self, mbo: MboMsg) { let price = mbo.price; let side = mbo.side().unwrap(); if mbo.flags.is_tob() { let levels: &mut BTreeMap = self.side_levels_mut(side); levels.clear(); // UNDEF_PRICE indicates the side's book should be cleared // and doesn't represent an order that should be added if mbo.price != UNDEF_PRICE { levels.insert(price, VecDeque::from([mbo])); } } else { assert_ne!(price, UNDEF_PRICE); assert!( self.orders_by_id .insert(mbo.order_id, (side, price)) .is_none() ); let level: &mut Level = self.get_or_insert_level(side, price); level.push_back(mbo); } } fn cancel(&mut self, mbo: MboMsg) { let side = mbo.side().unwrap(); let level = self.level_mut(side, mbo.price); let order_idx = Self::find_order(level, mbo.order_id); let existing_order = level.get_mut(order_idx).unwrap(); assert!(existing_order.size >= mbo.size); existing_order.size -= mbo.size; if existing_order.size == 0 { level.remove(order_idx).unwrap(); if level.is_empty() { self.remove_level(side, mbo.price); } self.orders_by_id.remove(&mbo.order_id).unwrap(); } } fn modify(&mut self, mbo: MboMsg) { let order_id = mbo.order_id; let side = mbo.side().unwrap(); let Some((id_side, id_price)) = self.orders_by_id.get_mut(&order_id) else { // If order not found, treat it as an add return self.add(mbo); }; let prev_side = *id_side; let prev_price = *id_price; // Update orders by ID *id_side = side; *id_price = mbo.price; // Update level order let level = self.level_mut(prev_side, prev_price); let order_idx = Self::find_order(level, order_id); let existing_order = level.get_mut(order_idx).unwrap(); let should_keep_priority = prev_price == mbo.price && existing_order.size >= mbo.size; existing_order.size = mbo.size; if should_keep_priority { return; } if prev_price != mbo.price { let prev_level = level; Self::remove_order(prev_level, order_id); if prev_level.is_empty() { self.remove_level(prev_side, prev_price); } let level = self.get_or_insert_level(side, mbo.price); level.push_back(mbo); } else { Self::remove_order(level, order_id); level.push_back(mbo); } } fn get_or_insert_level(&mut self, side: Side, price: i64) -> &mut Level { let levels = self.side_levels_mut(side); levels.entry(price).or_default() } fn level_mut(&mut self, side: Side, price: i64) -> &mut Level { let levels = self.side_levels_mut(side); levels.get_mut(&price).unwrap() } fn remove_level(&mut self, side: Side, price: i64) { self.side_levels_mut(side).remove(&price).unwrap(); } fn find_order(level: &VecDeque, order_id: u64) -> usize { level.iter().position(|o| o.order_id == order_id).unwrap() } fn remove_order(level: &mut VecDeque, order_id: u64) { let index = Self::find_order(level, order_id); level.remove(index).unwrap(); } fn side_levels_mut(&mut self, side: Side) -> &mut BTreeMap { match side { Side::Ask => &mut self.offers, Side::Bid => &mut self.bids, Side::None => panic!("Invalid side None"), } } fn side_levels(&self, side: Side) -> &BTreeMap { match side { Side::Ask => &self.offers, Side::Bid => &self.bids, Side::None => panic!("Invalid side None"), } } } impl PriceLevel { fn new<'a>(price: i64, orders: impl Iterator) -> Self { orders.fold( PriceLevel { price, size: 0, count: 0, }, | | mut level, order | { | if !order.flags.is_tob() { level.count += 1; } level.size += order.size; level }, ) } } impl Display for PriceLevel { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { write!( f, "{:4} @ {:6.2} | {:2} order(s)", self.size, pretty::Px(self.price), self.count ) } } ``` ## Results ```plaintext GOOGL Aggregated BBO | 2024-04-03 11:00:00.903881953 +00:00:00 None 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.903887026 +00:00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.914731635 +00:00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.925739296 +00:00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:00:00.925744379 +00:00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:03:55.585836619 +00:00:00 200 @ 155.20 | 1 order(s) 200 @ 152.80 | 1 order(s) GOOGL Aggregated BBO | 2024-04-03 11:03:55.585841782 +00:00:00 200 @ 155.40 | 1 order(s) 200 @ 152.80 | 1 order(s) ... ``` # Microprice and book imbalance In this example, we'll explore different ways to estimate the fair price of a financial instrument using market data. Specifically, we'll examine microprice and book imbalance to better understand order book dynamics. ## Overview We'll use the [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md) ("L2" data) which shows all trades, as well as all updates to the top 10 levels of the order book. We'll plot various ways you can use microprice or book imbalance when analyzing price action for an instrument. The standard formula for microprice, which only uses the top-of-book, is shown below: \[\text{Microprice} = \frac{s_b p_a + s_a p_b}{s_b + s_a}\] where - \(p_b\) and \(p_a\) are the best bid and ask prices - \(s_b\) and \(s_a\) are the sizes at the best bid and ask price levels To calculate book imbalance over N levels, we will use the following formula: \[Imb_N = \frac{\sum_{i=1}^{N} s_{b,i} - \sum_{i=1}^{N} s_{a,i}}{\sum_{i=1}^{N} s_{b,i} + \sum_{i=1}^{N} s_{a,i}}\] where - \(s_{b,i}\) is the size at the i-th best bid price level - \(s_{a,i}\) is the size at the i-th best ask price level - \(N\) is the number of levels to aggregate (e.g., N=5 for top 5 levels) - \(Imb_N\) ranges from -1 to 1 ## Example ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd def calc_microprice( df_mbp10: pd.DataFrame, ) -> pd.DataFrame: """ Calculate top-of-book microprice. """ numerator = (df_mbp10["bid_sz_00"] * df_mbp10["ask_px_00"]) + ( df_mbp10["ask_sz_00"] * df_mbp10["bid_px_00"] ) denominator = df_mbp10["bid_sz_00"] + df_mbp10["ask_sz_00"] df_mbp10["Microprice"] = numerator / denominator return df_mbp10 def calc_book_imbalance( df_mbp10: pd.DataFrame, n: int, ) -> pd.DataFrame: """ Calculate book imbalance over N levels of the order book. If a positive value, there is more size on the bid than the offer. If a negative value, there is more size on the offer than the bid. If the size is equal, or there is no size on either the bid or the ask, the value will be 0. """ total_bid_size = df_mbp10[[f"bid_sz_{i:02}" for i in range(n)]].sum(axis=1).astype("int64") total_ask_size = df_mbp10[[f"ask_sz_{i:02}" for i in range(n)]].sum(axis=1).astype("int64") s = ((total_bid_size - total_ask_size) / (total_bid_size + total_ask_size)).fillna(0) df_mbp10[f"Book imbalance ({n})"] = s return df_mbp10 def plot_microprice( df_mbp10: pd.DataFrame, symbol: str, ) -> None: """ Plot microprice against midprice and trades. """ df_trades = df_mbp10[df_mbp10["action"] == db.Action.TRADE] df_mbp10["Midprice"] = (df_mbp10["bid_px_00"] + df_mbp10["ask_px_00"]) / 2 df_mbp10 = calc_microprice(df_mbp10) y_tick_range = np.arange( min(df_mbp10["bid_px_00"].min(), df_trades["price"].min()), max(df_mbp10["ask_px_00"].max() + tick_size, df_trades["price"].max() + tick_size), tick_size, ) # Plot df_mbp10.plot( y=["Midprice", "Microprice"], title=f"{symbol} Midprice vs. Microprice", xlabel="Time (UTC)", ylabel="Price", drawstyle="steps-post", ) # Plot trades plt.scatter( df_trades.index, df_trades["price"], color="C2", marker=".", s=100, label="Trades", ) plt.yticks(y_tick_range) plt.legend() plt.show() def plot_book_imbalance( df_mbp10: pd.DataFrame, symbol: str, n: int = 10, ) -> None: """ Plot book imbalance over N levels of the book, along with trades. """ df_trades = df_mbp10[df_mbp10["action"] == db.Action.TRADE] df_mbp10 = calc_book_imbalance(df_mbp10, n) # Plot ax1 = df_mbp10[f"Book imbalance ({n})"].plot( drawstyle="steps-post", title=f"{symbol} Book Imbalance ({n} {'level' if n == 1 else 'levels'})", xlabel="Time (UTC)", ylabel="Imbalance", ) ax1.set_ylim(-1, 1) # Plot trades df_trades["is_buy"] = np.where(df_trades["side"] == "B", 1, -1) ax2 = ax1.twinx() ax2.scatter( df_trades.index, df_trades["is_buy"], color="C2", marker=".", s=150, label="Trades", ) lines1, labels1 = ax1.get_legend_handles_labels() lines2, labels2 = ax2.get_legend_handles_labels() ax2.legend(lines1 + lines2, labels1 + labels2) ax2.grid(False) ax2.set_yticks([-1, 1]) ax2.set_yticklabels(["Sell", "Buy"]) ax2.set_ylabel("Trade Side") plt.show() if __name__ == "__main__": # Set parameters dataset = "XNAS.ITCH" symbol = "TSLA" tick_size = 0.01 start = "2025-09-10T14:00:00" end = "2025-09-10T14:00:05" # Create historical client client = db.Historical("$YOUR_API_KEY") # Download MBP-10 data and convert to a DataFrame mbp10_df = client.timeseries.get_range( dataset=dataset, symbols=symbol, start=start, end=end, schema="mbp-10", ).to_df() # Plot top-of-book microprice plot_microprice(mbp10_df, symbol) # Plot book imbalance with 10 levels of the orderbook plot_book_imbalance(mbp10_df, symbol, 10) ``` ## Results First, we'll take a look at the microprice and compare it to the midprice. Microprice will take into account the size on the bid and offer. ![Microprice](https://databento.com/docs/microprice-0.svg.md) Since microprice is sensitive to changes in the orderbook, it may offer some predictive power for future price movements. Below, we'll take a look at book imbalance, which is a calculated indicator between -1 and 1. Book imbalance can look at multiple levels of the order book. Positive values mean there is more cumulative size on the bid than the offer, negative values mean there is more cumulative size on the offer, and a zero value means the bid and offer have the same total size. ![Book imbalance](https://databento.com/docs/microprice-1.svg.md) # Queue position of an order > **See also** > > It's recommended to first start with the [Order tracking](https://databento.com/docs/examples/order-book/order-tracking.md) and [Limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) examples. ## Overview In this example, we will use a complete limit order book to track the queue position of an order. This example assumes a [First-In-First-Out](https://cmegroupclientsite.atlassian.net/wiki/spaces/EPICSANDBOX/pages/457218479/Supported+Matching+Algorithms#FIFO) (FIFO) matching algorithm is being utilized. FIFO is the most common algorithm used in modern matching engines. When market participants have active orders in different instruments and contract months, knowing exact queue positions can be important in managing execution risk. The ability to track orders as they move throughout the limit order book can be utilized as a trading signal in automated strategies. ## Design We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) ("L3" data), which assigns a unique ID to every order in the book, to exactly track this queue position. The logic for this calculation is a straightforward extension of our [limit order book](https://databento.com/docs/examples/order-book/limit-order-book.md) example. We will also use the [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md) ("L2" data) to estimate queue position, assuming we did not have access to MBO data. Suppose \(p\left(x\right)\) denotes the probability of a cancel in front when your order is positioned \(0 \leq x \leq 1\) in the queue. - When you first join the queue, \(p\left(1\right)=1\), since all cancels must happen in front of you. - When you are at the front of the queue, \(p\left(0\right)=0\), since all cancels must happen from behind. A naive assumption is that orders cancel uniformly through the queue, i.e. \(p\left(x\right)=x\). In reality, cancels are more likely to happen behind you vs. in front of you, since the orders in front of you have value from their time priority. We can model this cancellation bias with a parameter \(k\), ranging from -1 to 1. \(k=0\) results in an unbiased, uniform distribution. \[p_k\left(x\right)= \begin{cases} x^{1+k} & \text{if } k < 0 \\ x & \text{if } k = 0 \\ 1-\left(1-x\right)^{1-k} & \text{if } k > 0 \end{cases} \] We'll take a look at this uniform cancellation estimation (\(k=0\)), as well as a few weighted estimations that assumes a higher probability of cancels from behind (\(k=-0.8\) and \(k=-0.95\)). ## Example ```python from __future__ import annotations import os import random from collections import defaultdict from dataclasses import dataclass, field from itertools import takewhile import databento as db import matplotlib.pyplot as plt import pandas as pd from sortedcontainers import SortedDict @dataclass(slots=True) class PriceLevel: price: int size: int = 0 count: int = 0 def __str__(self) -> str: price = self.price / db.FIXED_PRICE_SCALE return f"{self.size:4} @ {price:6.2f} | {self.count:2} order(s)" @dataclass(slots=True) class LevelOrders: price: int orders: list[db.MBOMsg] = field(default_factory=list, compare=False) def __bool__(self) -> bool: return bool(self.orders) @property def level(self) -> PriceLevel: return PriceLevel( price=self.price, count=sum(1 for o in self.orders if not o.flags & db.RecordFlags.F_TOB), size=sum(o.size for o in self.orders), ) @dataclass(slots=True) class Book: orders_by_id: dict[int, db.MBOMsg] = field(default_factory=dict) offers: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) bids: SortedDict[int, LevelOrders] = field(default_factory=SortedDict) def bbo(self) -> tuple[PriceLevel | None, PriceLevel | None]: return self.get_bid_level(), self.get_ask_level() def get_bid_level(self, idx: int = 0) -> PriceLevel | None: if self.bids and len(self.bids) > idx: # Reverse for bids to get highest prices first return self.bids.peekitem(-(idx + 1))[1].level return None def get_ask_level(self, idx: int = 0) -> PriceLevel | None: if self.offers and len(self.offers) > idx: return self.offers.peekitem(idx)[1].level return None def get_bid_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "B").level except KeyError: return None def get_ask_level_by_px(self, px: int) -> PriceLevel | None: try: return self._get_level(px, "A").level except KeyError: return None def get_order(self, id: int) -> db.MBOMsg | None: return self.orders_by_id.get(id) def get_queue_pos(self, id: int) -> int | None: order = self.get_order(id) if not order: return None level = self._get_level(order.price, order.side) return sum(order.size for order in takewhile(lambda o: o.order_id != id, level.orders)) def get_snapshot(self, level_count: int = 1) -> list[db.BidAskPair]: snapshots = [] for level in range(level_count): ba_pair = db.BidAskPair() bid = self.get_bid_level(level) if bid: ba_pair.bid_px = bid.price ba_pair.bid_sz = bid.size ba_pair.bid_ct = bid.count ask = self.get_ask_level(level) if ask: ba_pair.ask_px = ask.price ba_pair.ask_sz = ask.size ba_pair.ask_ct = ask.count snapshots.append(ba_pair) return snapshots def apply(self, mbo: db.MBOMsg) -> None: # Trade, Fill, or None: no change if mbo.action in ("T", "F", "N"): return # Clear book: remove all resting orders if mbo.action == "R": self._clear() return # side=N is only valid with Trade, Fill, and Clear actions assert mbo.side in ("A", "B") # UNDEF_PRICE indicates the book level should be removed if mbo.price == db.UNDEF_PRICE and mbo.flags & db.RecordFlags.F_TOB: self._side_levels(mbo.side).clear() return # Add: insert a new order if mbo.action == "A": self._add(mbo) # Cancel: partially or fully cancel some size from a resting order elif mbo.action == "C": self._cancel(mbo) # Modify: change the price and/or size of a resting order elif mbo.action == "M": self._modify(mbo) else: raise ValueError(f"Unknown action={mbo.action}") def _clear(self) -> None: self.orders_by_id.clear() self.offers.clear() self.bids.clear() def _add(self, mbo: db.MBOMsg) -> None: if mbo.flags & db.RecordFlags.F_TOB: levels = self._side_levels(mbo.side) levels.clear() levels[mbo.price] = LevelOrders(price=mbo.price, orders=[mbo]) else: level = self._get_or_insert_level(mbo.price, mbo.side) assert mbo.order_id not in self.orders_by_id self.orders_by_id[mbo.order_id] = mbo level.orders.append(mbo) def _cancel(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id[mbo.order_id] level = self._get_level(mbo.price, mbo.side) assert order.size >= mbo.size order.size -= mbo.size # If the full size is cancelled, remove the order from the book if order.size == 0: self.orders_by_id.pop(mbo.order_id) level.orders.remove(order) # If the level is now empty, remove it from the book if not level: self._remove_level(mbo.price, mbo.side) def _modify(self, mbo: db.MBOMsg) -> None: order = self.orders_by_id.get(mbo.order_id) if order is None: # If order not found, treat it as an add self._add(mbo) return assert order.side == mbo.side, f"Order {order} changed side to {mbo.side}" level = self._get_level(order.price, order.side) if order.price != mbo.price: # Changing price loses priority level.orders.remove(order) if not level: self._remove_level(order.price, mbo.side) level = self._get_or_insert_level(mbo.price, mbo.side) level.orders.append(mbo) elif order.size < mbo.size: # Increasing size loses priority level.orders.remove(order) level.orders.append(mbo) else: # Update in place level.orders[level.orders.index(order)] = mbo self.orders_by_id[mbo.order_id] = mbo def _side_levels(self, side: str) -> SortedDict: if side == "A": return self.offers if side == "B": return self.bids raise ValueError(f"Invalid {side =}") def _get_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price not in levels: raise KeyError(f"No price level found for {price =} and {side =}") return levels[price] def _get_or_insert_level(self, price: int, side: str) -> LevelOrders: levels = self._side_levels(side) if price in levels: return levels[price] level = LevelOrders(price=price) levels[price] = level return level def _remove_level(self, price: int, side: str) -> None: levels = self._side_levels(side) levels.pop(price) @dataclass(slots=True) class Market: books: defaultdict[int, defaultdict[int, Book]] = field( default_factory=lambda: defaultdict(lambda: defaultdict(Book)), ) def get_books_by_pub(self, instrument_id: int) -> defaultdict[int, Book]: return self.books[instrument_id] def get_book(self, instrument_id: int, publisher_id: int) -> Book: return self.books[instrument_id][publisher_id] def bbo( self, instrument_id: int, publisher_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: return self.books[instrument_id][publisher_id].bbo() def aggregated_bbo( self, instrument_id: int, ) -> tuple[PriceLevel | None, PriceLevel | None]: agg_bbo: list[PriceLevel | None] = [None, None] # max for bids, min for asks for idx, reducer in [(0, max), (1, min)]: all_best = [b.bbo()[idx] for b in self.books[instrument_id].values()] all_best = [b for b in all_best if b] if not all_best: continue best_price = reducer(b.price for b in all_best) best = [b for b in all_best if b.price == best_price] agg_bbo[idx] = PriceLevel( price=best_price, size=sum(b.size for b in best), count=sum(b.count for b in best), ) return tuple(agg_bbo) def apply(self, mbo: db.MBOMsg) -> None: book = self.books[mbo.instrument_id][mbo.publisher_id] book.apply(mbo) def apply_bias(x: float, k: float) -> float: """ Apply bias to the random number `x` in the range [0, 1]. -1 < k < 0 results in a bias towards 1. k == 0 results in a uniform distribution 0 < k < 1 results in a bias towards 0 k == -1 always returns 1 k == 1 always returns 0 """ assert 0 <= x <= 1 # Random number between 0 and 1 assert -1 <= k <= 1 # Bias between -1 and 1 if k <= 0: return x ** (1 + k) else: return 1 - (1 - x) ** (1 - k) if __name__ == "__main__": # Create a historical client client = db.Historical("$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" start_date = "2025-02-12" end_date = "2025-02-13" symbol = "ES.v.0" # For this example, we will retroactively pick an order to track order_id = 6414447712675 order_add_time = 1739375799442261697 order_fill_time = 1739375974175277429 order_price = 6064000000000 order_size = 1 # Now we will request MBO data. mbo_data_path = "glbx-mdp3-20250212.mbo.dbn.zst" if os.path.exists(mbo_data_path): mbo_data = db.DBNStore.from_file(mbo_data_path) else: mbo_data = client.timeseries.get_range( dataset=dataset, start=start_date, end=end_date, symbols=symbol, stype_in="continuous", schema="mbo", path=mbo_data_path, ) # Next, we will record all updates at this price while the order was active price_level_msgs: list[dict] = [] book = Book() for mbo_msg in mbo_data: # Process MBO msg book.apply(mbo_msg) # Only record updates when order is active if order_add_time <= mbo_msg.ts_event < order_fill_time: # Wait for end of event if mbo_msg.flags & db.RecordFlags.F_LAST: # Get total size at price and actual queue position price_level_msg = { "ts_event": mbo_msg.ts_event, "Depth at price level": book.get_ask_level_by_px(order_price).size, "Queue position": book.get_queue_pos(order_id), } price_level_msgs.append(price_level_msg) # Construct a DataFrame from MBO updates df = pd.DataFrame.from_records(price_level_msgs) df["Order lifetime (s)"] = (df["ts_event"] - df["ts_event"][0]) / 1e9 df = df.set_index("Order lifetime (s)") # Plot actual queue position using MBO data ax = df[["Depth at price level", "Queue position"]].plot( drawstyle="steps-post", ylabel="Contracts", ) def generate_simulations(df: pd.DataFrame, k: float) -> pd.Series: """ Generate simulations for estimated queue position. """ # Run simulations for estimated queue position sim_list = [df.index.to_list()] # Order is starting at the back of the queue total_size_list = df["Depth at price level"].to_list() start_queue_pos = total_size_list[0] # Run 100 simulations for i in range(100): c_total_size = start_queue_pos # Current size at price level c_queue_pos = start_queue_pos - order_size # Current estimated queue position estimated_queue_pos = [] for total_size in total_size_list: # Decrease in size at price level. A cancel occurred if total_size < c_total_size: cancel_size = c_total_size - total_size others_size = total_size - order_size # Size of rest of level random_number = apply_bias(random.random(), k) if random_number < (c_queue_pos / others_size): c_queue_pos -= cancel_size c_queue_pos = min(max(c_queue_pos, 0), others_size) c_total_size = total_size estimated_queue_pos.append(c_queue_pos) sim_list.append(estimated_queue_pos) col_names = [f"sim_{i}" for i in range(len(sim_list))] df_sim = pd.DataFrame(dict(zip(col_names, sim_list))) df_sim = df_sim.set_index(df.index) return df_sim.quantile(0.50, axis=1) # Next, we will also download MBP-10 data for estimating queue position mbp10_data_path = "glbx-mdp3-20250212.mbp-10.dbn.zst" if os.path.exists(mbp10_data_path): mbp10_data = db.DBNStore.from_file(mbp10_data_path) else: mbp10_data = client.timeseries.get_range( dataset=dataset, start=start_date, end=end_date, symbols=symbol, stype_in="continuous", schema="mbp-10", path=mbp10_data_path, ) # Now we will use the MBP-10 data to record similar information as above mbp10_price_level_msgs: list[dict] = [] for mbp10_msg in mbp10_data: if order_add_time <= mbp10_msg.ts_event < order_fill_time: for level_pair in mbp10_msg.levels: if level_pair.ask_px == order_price: # Get total size at price mbp_price_level_msg = { "ts_event": mbp10_msg.ts_event, "Depth at price level": level_pair.ask_sz, } mbp10_price_level_msgs.append(mbp_price_level_msg) # Construct a DataFrame from MBP-10 updates df_mbp10 = pd.DataFrame.from_records(mbp10_price_level_msgs) df_mbp10["Order lifetime (s)"] = (df_mbp10["ts_event"] - df_mbp10["ts_event"][0]) / 1e9 df_mbp10 = df_mbp10.set_index("Order lifetime (s)") # Run simulations using various weights for k in (0, -0.8, -0.95): df_mbp10[f"Estimated queue position (k={k})"] = generate_simulations(df_mbp10, k) # Add estimated queue position using MBP-10 data to plot df_mbp10.filter(regex="Estimated", axis=1).plot( ax=ax, drawstyle="steps-post", title="Actual vs. estimated queue position", style=["C3", "C5", "C2"], ylim=(0, 85), ) plt.show() ``` ## Results We can show how the exact queue position of the order within its price level decreases over time (orange), even as orders are added and removed from the book (purple). The naive model assuming uniform cancellation (green) does a poor job estimating the position, but the biased models perform better. ![Queue position](https://databento.com/docs/queue-position-0.svg.md) ## Algorithmic trading | Title | Subtitle | Image | Supported languages | | [A high-frequency liquidity-taking strategy](https://databento.com/docs/examples/algo-trading/high-frequency.md) | Implement simulated trading (paper trading) using live data. | ![example1](https://databento.com/docs/results.png.md) | Python, Live | | [Build prediction models with machine learning](https://databento.com/docs/examples/algo-trading/machine-learning.md) | Construct alpha signals using Databento and regression models from sklearn. | ![example2](https://databento.com/docs/machine-learning-thumbnail.svg.md) | Python, Historical | | [Execution slippage and markouts](https://databento.com/docs/examples/algo-trading/execution-slippage.md) | Plot markout curves for transaction cost analysis and measuring execution quality. | ![example3](https://databento.com/docs/execution-slippage-thumbnail.svg.md) | Python, Historical | | [Matching engine latencies](https://databento.com/docs/examples/algo-trading/latency.md) | Measure matching engine and feed latency with nanosecond, PTP timestamps. | ![example4](https://databento.com/docs/latency-thumbnail.svg.md) | Python, Historical | | [Using messaging rates as a proxy for implied volatility](https://databento.com/docs/examples/algo-trading/messaging-volatility.md) | Determine and compare implied volatility (IV) and messaging rates. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Mean reversion and portfolio optimization](https://databento.com/docs/examples/algo-trading/portfolio-optimization.md) | Fetch cross-sectional returns. Build a mean reversion strategy and optimal portfolios. | ![Image placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Pairs trading based on cointegration](https://databento.com/docs/examples/algo-trading/pairs-trading.md) | A cross-venue pairs trading strategy between WTI (CME) and Brent (ICE) crude. | ![example7](https://databento.com/docs/spread.png.md) | Python, Historical | | [Build a real-time stock screener](https://databento.com/docs/examples/algo-trading/live-stock-screener.md) | Build a live stock screener for all U.S. equity tickers. | ![example7](https://databento.com/docs/placeholder.svg.md) | Python, Historical, Live | # A high-frequency liquidity-taking strategy We'll introduce some terminology for this tutorial: - A **feature** is any kind of basic independent variable that we think has some predictive value. This follows machine learning nomenclature; others may refer to this as a **predictor** or **regressor** in a statistical or econometric setting. - A **trading rule** is a hardcoded trading decision. An example of a trading rule is "If there's only 1 order left at the best offer, lift the offer; if there is only 1 order left at the best bid, hit the bid". A trading rule may be a hardcoded trading decision taken when a feature value exceeds a certain threshold. - A strategy that is based on trading rules is a **rule-based** strategy. - A **liquidity-taking** strategy takes liquidity by crossing the spread with aggressive or marketable orders. - A **high-frequency** strategy is characterized by a large number of trades. ## Book skew and trading rule The simplest type of book feature is called the book skew, which is the imbalance between resting bid depth (vb) and resting ask depth (va) at the top of the book. We can formulate this as some sort of difference between vb and va. It's convenient to scale these by their order of magnitude, so we take their log differences instead. \[\text{skew} = \log(v_b)- \log(v_a) = \log \dfrac{v_b}{v_a}\] Notice that we picked this ordering simply because it's useful to formulate features such that positive values imply that we expect prices to go up; this makes it easier to debug your strategy. Intuitively, we expect higher bid depth to imply higher buy demand and hence higher prices. We can introduce a trading rule that buys when this feature exceeds some alpha threshold `k`, and sells when it goes below some threshold `-k`. ## Minimum order quantity While there's some practical advantage to trading larger clips for a liquidity-taking strategy, we'll start with a constant trade size equal to the minimum order quantity so that we minimize slippage and market impact considerations. The minimum order quantity depends on the trading platform or market that you're on. For this toy strategy, we'll use E-mini S&P 500 (ES) futures as an example and trade in clips of 1 contract. \[\text{skew} > k \rightarrow \text{Buy 1 lot}\] \[\text{skew} < -k \rightarrow \text{Sell 1 lot}\] ## Commissions This strategy will be very sensitive to commissions due to its volume, so we'll include commissions on the estimated PnL. These commissions can be found [here](https://www.cmegroup.com/company/clearing-fees.html). ## Position and risk limits This strategy is convenient because you don't have to worry about complex order and position state management. You just let it build up whatever maximum position you want. Because we expect buys and sells to be symmetrically distributed in the long run, you will eventually get out of position. However, you might not have enough margin to build up arbitrarily large positions, so for proof of concept, we'll specify a maximum position of 10 contracts. \[\text{skew} > k\text{ and pos} < 10\text{ lots} \rightarrow \text{Buy 1 lot}\] \[\text{skew} < -k\text{ and pos} > -10\text{ lots} \rightarrow \text{Sell 1 lot}\] ## Implementation These are the parameters we have so far: > **Info** > > This example uses the instrument with raw symbol ESU3. You'll need to substitute this > with your desired symbol. ```python import math import json import sys from dataclasses import dataclass, field from decimal import Decimal import pandas as pd import databento as db API_KEY = "$YOUR_API_KEY" class Config: # Alpha threshold to buy/sell, k ALPHA_THRESHOLD: int = 1.7 # Symbol DATASET = "GLBX.MDP3" SYMBOL = "ESU3" POINT_VALUE = 50 # $50 per index point # Fees VENUE_FEES_PER_SIDE: Decimal = Decimal("0.39") CLEARING_FEES_PER_SIDE: Decimal = Decimal("0.05") FEES_PER_SIDE: Decimal = VENUE_FEES_PER_SIDE + CLEARING_FEES_PER_SIDE # Position limit POSITION_MAX: int = 10 ``` Since we're only simulating liquidity-taking at minimum size, our MBP-1 schema is sufficient. You can learn more about our MBP-1 schema [here](https://databento.com/docs/schemas/mbp-1.md). To keep this example simple, we'll assume zero round-trip latency for any orders placed. This enables us to implement a simple, online calculation of PnL for our real-time trading simulation. ```python @dataclass(slots=True) class Strategy: # Dataset dataset: str # Instrument to trade symbol: str # Current position, in contract units position: int = 0 # Number of long contract sides traded buy_qty: int = 0 # Number of short contract sides traded sell_qty: int = 0 ## Total realized buy price real_total_buy_px: Decimal = 0 ## Total realized sell price real_total_sell_px: Decimal = 0 # Total buy price to liquidate current position theo_total_buy_px: Decimal = 0 # Total sell price to liquidate current position theo_total_sell_px: Decimal = 0 # Total fees paid fees: Decimal = 0 # List to track results results: list = field(default_factory=list) def run(self) -> None: client = db.Live() client.subscribe( dataset=self.dataset, schema="mbp-1", stype_in="raw_symbol", symbols=[self.symbol], # start="2023-08-08T12:00", # Burn-in start time ) for record in client: if not isinstance(record, db.MBP1Msg): continue self.update(record) def update(self, record: db.MBP1Msg) -> None: ask_size = record.levels[0].ask_sz bid_size = record.levels[0].bid_sz ask_price = record.levels[0].ask_px / Decimal("1e9") bid_price = record.levels[0].bid_px / Decimal("1e9") skew = math.log10(bid_size) - math.log10(ask_size) mid_price = (ask_price + bid_price) / 2 # Buy/sell based when skew signal is large if skew > Config.ALPHA_THRESHOLD and self.position < Config.POSITION_MAX: self.position += 1 self.buy_qty += 1 self.real_total_buy_px += ask_price self.fees += Config.FEES_PER_SIDE elif skew < -Config.ALPHA_THRESHOLD and self.position > -Config.POSITION_MAX: self.position -= 1 self.sell_qty += 1 self.real_total_sell_px += bid_price self.fees += Config.FEES_PER_SIDE # Update prices # Fill prices are based on BBO with assumed zero latency # In practice, should be worse because of alpha decay if self.position == 0: self.theo_total_buy_px = 0 self.theo_total_sell_px = 0 elif self.position > 0: self.theo_total_sell_px = bid_price * abs(self.position) elif self.position < 0: self.theo_total_buy_px = ask_price * abs(self.position) # Compute PnL theo_pnl = ( Config.POINT_VALUE * ( self.real_total_sell_px + self.theo_total_sell_px - self.real_total_buy_px - self.theo_total_buy_px ) - self.fees ) result = { "ts_strategy": str(pd.Timestamp(record.ts_recv, tz="UTC")), "bid": f"{bid_price:0.2f}", "ask": f"{ask_price:0.2f}", "skew" : f"{skew:0.3f}", "position": self.position, "trade_ct": self.buy_qty + self.sell_qty, "fees": f"{self.fees:0.2f}", "pnl": f"{theo_pnl:0.2f}", } print(json.dumps(result, indent=4)) self.results.append(result) if __name__ == "__main__": strategy = Strategy(dataset=Config.DATASET, symbol=Config.SYMBOL) while True: try: strategy.run() except KeyboardInterrupt: df = pd.DataFrame(strategy.results) df.to_csv("strategy_log.csv", index=False) sys.exit() ``` ## Results ![Results](https://databento.com/docs/results.png.md) ## Further improvements This implementation uses our simple synchronous client. For production applications, we recommend using our [asynchronous client](https://databento.com/docs/api-live/client/aiter.md?lang=python) or [callback model](https://databento.com/docs/api-live/client/add-callback.md?lang=python). A problem encountered with the book skew is that extreme values may be influenced by spoofing. One possibility is to modify the trading rule and introduce an upper limit as follows: \[\text{skew} > k \text{ and abs(skew)} < L \text{ and pos} < 10 \text{ lots} \rightarrow \text{Buy 1 lot}\] \[\text{skew} < -k \text{ and abs(skew)} < L \text{ and pos} > -10 \text{ lots} \rightarrow \text{Sell 1 lot}\] Finally, recall that we assumed zero delay in order placement and fill. It's also important to incorporate a delay when extending this example. # Build prediction models with machine learning ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to build high-frequency trading signals and use them to train a machine learning model. ## MBP-10 schema We'll demonstrate this example using the [MBP-10 schema](https://databento.com/docs/schemas/mbp-10.md). The MBP-10 schema contains the top ten levels of aggregated book depth. This updates on every order book event. ## Dependencies This example will use the [scikit-learn](https://pypi.org/project/scikit-learn/) package for creating a machine learning model. The [matplotlib](https://pypi.org/project/matplotlib/) package will be used for charting. These dependencies can be installed with the following: ```bash $ pip install scikit-learn matplotlib ``` ## Example ```python import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd from sklearn.linear_model import LinearRegression # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Set parameters dataset = "GLBX.MDP3" product = "ES" start = "2023-12-06T14:30:00" end = "2023-12-06T20:33:00" # Request MBP-10 for the front-month ES contract and convert to DataFrame df = client.timeseries.get_range( dataset="GLBX.MDP3", schema="mbp-10", symbols=f"{product}.v.0", stype_in="continuous", start=start, end=end, ).to_df() # Filter out trades only df = df[df["action"] == db.Action.TRADE] # Calculate midprice returns with a forward markout of 500 trades df["mid"] = (df["bid_px_00"] + df["ask_px_00"]) / 2 df["ret_500t"] = df["mid"].shift(-500) - df["mid"] df = df.dropna() # Calculate depth imbalance on top level ('skew') df["skew"] = np.log(df.bid_sz_00) - np.log(df.ask_sz_00) # Calculate order imbalance on top ten levels ('imbalance') bid_count = df[list(df.filter(regex="bid_ct_0[0-9]"))].sum(axis=1) ask_count = df[list(df.filter(regex="ask_ct_0[0-9]"))].sum(axis=1) df["imbalance"] = np.log(bid_count) - np.log(ask_count) # Split in-sample and out-of-sample split = int(0.66 * len(df)) split -= split % 100 df_in = df.iloc[:split] df_out = df.iloc[split:] # Evaluate signal correlation corr = df_in[["skew", "imbalance", "ret_500t"]].corr() print(corr.where(np.triu(np.ones(corr.shape)).astype(bool))) reg = LinearRegression(fit_intercept=False, positive=True) # Create a model using skew only reg.fit(df_in[["skew"]], df_in["ret_500t"]) pred_skew = reg.predict(df_out[["skew"]]) # Create a model using imbalance only reg.fit(df_in[["imbalance"]], df_in["ret_500t"]) pred_imbalance = reg.predict(df_out[["imbalance"]]) # Create a model using both skew and imbalance reg.fit(df_in[["skew", "imbalance"]], df_in["ret_500t"]) pred_combined = reg.predict(df_out[["skew", "imbalance"]]) # Define a function to calculate profit and loss def get_cumulative_markout_pnl(pred): df_pnl = pd.DataFrame({"pred": pred, "ret_500t": df_out["ret_500t"].values}) df_pnl.loc[df_pnl["pred"] < 0, "ret_500t"] *= -1 df_pnl = df_pnl.sort_values(by="pred") return df_pnl["ret_500t"].cumsum().values # Collect results into a DataFrame results = pd.DataFrame( { "skew": get_cumulative_markout_pnl(pred_skew), "imbalance": get_cumulative_markout_pnl(pred_imbalance), "combined": get_cumulative_markout_pnl(pred_combined), }, index=np.linspace(0, 100, num=len(df_out)), ) # Plot the results results.plot( title="Forecasting with book skew vs. imbalance", xlabel="Predictor value (percentile)", ylabel="Cumulative return", ).legend() plt.show() ``` ## Result ```plaintext skew imbalance ret_500t skew 1.0 0.47513 0.109004 imbalance NaN 1.00000 0.064666 ret_500t NaN NaN 1.000000 ``` ![Forecasting with book skew vs. imbalance](https://databento.com/docs/machine-learning-0.svg.md) # Execution slippage and markouts ## What is a markout? A markout measures the price change within some arbitrary time interval before or after an event, such as a fill or a lit trading signal. Markouts are often used for transaction cost analysis (TCA) and determining execution quality, but they may also be used for alpha research. For more information on markouts, check out the [Databento Microstructure Guide](https://databento.com/microstructure/markout). ## Overview In this example we'll use the [Historical client](https://databento.com/docs/api-historical/client.md) to create markout curves. This example will use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md). This schema provides every trade and order book event that updates the top of book. In this case we'll compute the markouts by midprice change, but it's also possible to compute it on trade-to-mid or trade-to-trade basis. Markouts are usually computed at different time intervals and then averaged over multiple events to a create a single curve. ## Markout events We'll calculate markout curves for the aggressive and passive side of every trade in NVDA for the following quantities: - less than 100 shares - 100 or more shares - 200 or more shares We'll use these curves to help understand adverse selection on passive limit orders and execution slippage on aggressive market orders to help determine order execution strategy. ## Example ```python from pathlib import Path import databento as db import matplotlib.pyplot as plt import numpy as np import pandas as pd NUM_TIME_SAMPLES = 1000 WINDOW_LIMITS_MICROS = 120 * 1e6 def calc_markouts( midprice: pd.Series, events: pd.DataFrame, offsets: pd.TimedeltaIndex, aggressor_markout: bool = True, ) -> pd.DataFrame: # Create an empty array to store results results = np.empty(shape=(len(events), len(offsets), 1)) # Calculate markouts for each event for i, (ts_event, event) in enumerate(events.iterrows()): markouts = midprice.reindex(offsets + ts_event, method="ffill") - event["price"] # All "event" records here are aggressive trades, so `side` is the aggressor side # Passive seller if not aggressor_markout and event["side"] == "B": markouts *= -1 # Aggressive seller elif aggressor_markout and event["side"] == "A": markouts *= -1 markouts_in_mils = markouts * 1e4 results[i] = markouts_in_mils.values.reshape(-1, 1) # Transpose the array and calculate the mean return pd.DataFrame( data=np.nanmean(results, 0), index=offsets.total_seconds() * 1e6, ) def plot_markouts( events: pd.DataFrame, midprice: pd.Series, lt_thresholds: tuple[int] = (100,), gt_thresholds: tuple[int] = (100, 200), aggressor_markout: bool = True, ) -> None: # Create log spaced intervals from -2 minutes -> +2 minutes samples = np.geomspace(1e-3, WINDOW_LIMITS_MICROS, num=NUM_TIME_SAMPLES) offsets = pd.to_timedelta( np.append(-np.flip(samples), np.append(0, samples)), unit="microseconds", ) for threshold in lt_thresholds: # Filter trade events below the size threshold events_mask = events[events["size"] < threshold] # Calculate markouts markouts = calc_markouts(midprice, events_mask, offsets, aggressor_markout) plt.plot( markouts, drawstyle="steps-post", label=f"order size < {threshold:d} (n={len(events_mask):,d})", ) for threshold in gt_thresholds: # Filter trade events above the size threshold events_mask = events[events["size"] >= threshold] # Calculate markouts markouts = calc_markouts(midprice, events_mask, offsets, aggressor_markout) plt.plot( markouts, drawstyle="steps-post", label=f"order size >= {threshold:d} (n={len(events_mask):,d})", ) # Annotations plt.title(f"Markouts - {'Aggressive' if aggressor_markout else 'Passive'} orders") plt.xlabel("Time since event (μs)") plt.ylabel("PnL/share (mils)") plt.xscale("symlog") plt.legend() plt.show() if __name__ == "__main__": # Set parameters dataset = "XNAS.ITCH" symbol = "NVDA" start = "2024-05-21" # Create historical client client = db.Historical("$YOUR_API_KEY") # Download data or read an existing file mbp_data_file = Path(f"xnas-itch-{start.replace('-', '')}-mbp-1.dbn.zst") if not mbp_data_file.exists(): mbp_data = client.timeseries.get_range( dataset=dataset, symbols=symbol, schema="mbp-1", start=start, path=mbp_data_file, ) else: mbp_data = db.DBNStore.from_file(mbp_data_file) # Convert to DataFrame df = mbp_data.to_df() df = df.dropna().set_index("ts_event") # We will use the midprice to calculate markouts midprice_s = df[["bid_px_00", "ask_px_00"]].mean(axis=1).groupby("ts_event").last() # Filter for trades (exclude non-cross and cross trades) trades = df[(df["action"] == "T") & (df["side"] != "N")] # Plot markouts for limit orders plot_markouts(trades, midprice_s, aggressor_markout=False) trades["cost"] = trades["price"] * trades["size"] agg_trades = trades.groupby("ts_event").agg( { "size": "sum", "side": "first", "cost": "sum", }, ) agg_trades["price"] = agg_trades["cost"] / agg_trades["size"] # Plot markouts for market orders plot_markouts(agg_trades, midprice_s, aggressor_markout=True) ``` ## Result ![NVDA aggressive markouts](https://databento.com/docs/execution-slippage-1.svg.md) Observe that crossing the spread with smaller orders exhibits less market impact, which agrees with our general intuition. ![NVDA passive markouts](https://databento.com/docs/execution-slippage-0.svg.md) We see the markouts are initially positive after the fill, as passive orders collect the spread and don't incur instantaneous slippage. However, the markouts are negative after roughly 1 second, showing the strong effects of adverse selection on passive orders of all sizes. # Matching engine latencies ## Overview In this example we will use the [Historical client](https://databento.com/docs/api-historical/client.md) to process instrument definition and MBO data to calculate matching engine and feed latency. The *matching engine latency* is the time between a market event and when the message was sent by the exchange. The *feed latency* is the time between that message being sent by the exchange and when Databento's servers received it. > **See also** > > For calculating latency from the [Live client](https://databento.com/docs/api-live/client.md) see [this example](https://databento.com/docs/examples/basics-live/live-latency.md). ## Definition We will use the [definition schema](https://databento.com/docs/schemas/instrument-definitions.md) in this example to assign the asset to each instrument, allowing us to examine these measurements for different assets. ## MBO We will use the [MBO schema](https://databento.com/docs/schemas/mbo.md) in this example to measure the matching engine and feed latency, but any schema which contains the `ts_recv` and `ts_in_delta` timestamps can be used for this purpose. ## Timestamps Databento provides several nanosecond resolution timestamps for every record. For these calculations, we will use: `ts_event`, `ts_recv`, and `ts_in_delta`. You can review our [timestamping guide](https://databento.com/docs/architecture/timestamping-guide.md) to learn more about where each of these are captured, but they are briefly described as: | Timestamp | Description | | `ts_event` | The matching-engine-received timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `ts_recv` | The capture-server-received timestamp expressed as the number of nanoseconds since the UNIX epoch. | | `ts_in_delta` | The matching-engine-sending timestamp expressed as the number of nanoseconds before `ts_recv`. | ## Example We will use a few parent symbols for this analysis: `ES.FUT`, `SR3.FUT`, `6E.FUT`, `BTC.FUT`, `ZF.FUT`, `GC.FUT`, and `CL.FUT`. These assets belong to different channels in CME Globex's matching engine which will allow us to compare the matching engine latency between channels. You can read more about Databento's parent symbology [here](https://databento.com/docs/conventions/symbology.md#parent). ```python import databento as db import matplotlib.pyplot as plt import numpy as np # Set parameters dataset = "GLBX.MDP3" products = [ "ES.FUT", "SR3.FUT", "6E.FUT", "ZF.FUT", "GC.FUT", "CL.FUT", ] # Create a historical client client = db.Historical(key="$YOUR_API_KEY") # Request definition data definitions = client.timeseries.get_range( dataset=dataset, schema="definition", symbols=products, stype_in="parent", start="2024-01-21T00:00:00-6", end="2024-01-22T17:00:00-6", ) # Request MBO data to calculate latency mbo = client.timeseries.get_range( dataset=dataset, schema="mbo", symbols=products, stype_in="parent", start="2024-01-26T05:00:00-6", end="2024-01-26T17:00:00-6", # 12 hours of mbo ) # Create a DataFrame for the mbo data mbo_df = mbo.to_df(pretty_ts=False, map_symbols=False) # Filter out any records with a bad ts_recv timestamp mbo_df[mbo_df["flags"] & db.RecordFlags.F_BAD_TS_RECV == 0] # Check for bad timestamps if (mbo_df.index == db.UNDEF_TIMESTAMP).any(): raise ValueError("Data contains one or more undefined ts_recv timestamps") if (mbo_df["ts_in_delta"] == db.UNDEF_TIMESTAMP).any(): raise ValueError("Data contains one or more undefined ts_in_delta timestamps") # Calculate latency mbo_df["latency_matching_us"] = (mbo_df.index - mbo_df["ts_in_delta"] - mbo_df["ts_event"]) / 1e3 mbo_df["latency_send_to_recv_us"] = mbo_df["ts_in_delta"] / 1e3 mbo_df = mbo_df[["instrument_id", "latency_matching_us", "latency_send_to_recv_us"]] # Get the asset and instrument class for every symbol def_df = definitions.to_df(pretty_ts=False, map_symbols=False) def_df = def_df[["instrument_id", "asset", "instrument_class"]].set_index("instrument_id") # Join the two DataFrames latency_df = mbo_df.join(def_df, on="instrument_id") # Plot send to receive latency across all symbols data = latency_df["latency_send_to_recv_us"] sorted_data = np.sort(data) cum_probability = np.linspace(0, 1, num=len(sorted_data)) plt.plot(sorted_data, cum_probability) plt.title("Send to Receive Latency") plt.grid(axis="x") plt.xscale("log") plt.ylabel("Cumulative Probability") plt.xlabel("Latency (microseconds)") plt.show() # Plot matching engine latency across all symbols data = latency_df["latency_matching_us"] sorted_data = np.sort(data) cum_probability = np.linspace(0, 1, num=len(sorted_data)) plt.plot(sorted_data, cum_probability) plt.title("Matching Engine Latency") plt.grid(axis="x") plt.xscale("log") plt.ylabel("Cumulative Probability") plt.xlabel("Latency (microseconds)") plt.show() # Plot matching engine latency for each asset for asset, group in latency_df.groupby("asset")["latency_matching_us"]: data = np.percentile(group, range(0, 101)) plt.plot( data.take([50, 90, 95, 99]), [asset] * 4, marker="|", linestyle="", markersize=20, ) plt.scatter( data, [asset] * len(data), marker=".", alpha=[1 - abs(i - 50) / 50 for i, _ in enumerate(data)], ) plt.title("Matching Engine Latency") plt.grid(axis="x") plt.xscale("log") plt.ylabel("Channel") plt.xlabel("Latency (microseconds)") plt.show() ``` ## Results ![Feed latency](https://databento.com/docs/latency-0.svg.md) ![Matching engine latency](https://databento.com/docs/latency-1.svg.md) ![Matching engine channel latency](https://databento.com/docs/latency-2.svg.md) # Using messaging rates as a proxy for implied volatility > **Info** > > The Databento team is currently working on this article. > # Mean reversion and portfolio optimization > **Info** > > The Databento team is currently working on this article. > # Pairs trading based on cointegration **Pairs trading** is a form of statistical arbitrage that takes advantage of the co-movement in prices of two instruments. It seeks to profit from the convergence of the two prices. The simplest variation of this strategy involves going long and short simultaneously on a pair of **cointegrated** instruments. But more generally, you can construct a spread from any linear combination of the two instruments with the approach in this tutorial--even going long simultaneously on two instruments. ## Setup and dependencies We'll make use of the following external libraries: - [matplotlib](https://matplotlib.org/) for plotting. - [numpy](https://numpy.org/) for data manipulation. - [pandas](https://pandas.pydata.org/) for data manipulation. - [scikit-learn](https://scikit-learn.org/) to fit the linear regression between the two price series. - [statsmodels](https://www.statsmodels.org) for the Engle-Granger two-step cointegration test. ``` import databento as db import matplotlib.pyplot as plt import numpy import pandas as pd from statsmodels.tsa.stattools import coint from sklearn.linear_model import LinearRegression ``` ## CME WTI (CL) vs. ICE Brent crude oil (BRN) We'll demonstrate the pairs trading strategy on the two most liquid crude oil products traded on CME and ICE, the [WTI crude oil futures]($PLATFORM_HOSTNAME/datasets/GLBX.MDP3/Futures/CL) (CL) [and Brent crude oil futures]($PLATFORM_HOSTNAME/datasets/IFEU.IMPACT/Futures/BRN) (BRN) contracts respectively. This also showcases the value of Databento's supplemental receive timestamps, `ts_recv`, which are used to construct our OHLCV aggregates. Most academic literature avoids cross-venue arbitrage like this because timestamps from two different exchanges are based on the different reference clocks that cannot be synced easily. `ts_recv` mitigates this problem by providing a common reference time with sub-microsecond accuracy between geographically-distant venues. ## Configuration Exchange and clearing fees can be found on [CME](https://www.cmegroup.com/company/clearing-fees.html) and [ICE](https://www.ice.com/publicdocs/IFEU_Exchange_Clearing_Fees.pdf)'s websites. Tick sizes and notional tick values can be obtained from [instrument definitions](https://databento.com/docs/schemas/instrument-definitions.md). The [Computing tick value and contract notional](https://databento.com/docs/examples/instrument-definitions/contract-notional.md) example shows the calculation. ```python # Backtest configuration start = "2024-04-01" end = "2024-05-15" # Instrument-specific configuration symbol_x = "CLM4" symbol_y = "BRN FMN0024!" tick_size_x = 0.01 tick_size_y = 0.01 tick_value_x = 10.0 tick_value_y = 10.0 fees_per_side_x = 0.70 # NYMEX corporate member fees fees_per_side_y = 0.88 # Global configuration commissions_per_side = 0.08 trading_days_count = 252 # Number of trading days in one year # Model and monetization parameters lookback = 100 entry_threshold = 1.5 exit_threshold = 0.5 p_threshold = 0.05 ``` ## Data extraction We'll define a function to download the data needed in this example. ```python def download_data( dataset: str, schema: str, symbol: str, start: str, end: str | None = None, ) -> pd.DataFrame: client = db.Historical(key="$YOUR_API_KEY") data = client.timeseries.get_range( dataset=dataset, schema=schema, symbols=symbol, start=start, end=end, ) return data.to_df() ``` We'll fetch 1-minute [OHLCV aggregates](https://databento.com/docs/schemas/ohlcv.md) for ease of demonstration, but you may specify `schema="mbp-1"` to use [MBP-1](https://databento.com/docs/schemas/mbp-1.md) instead: ```python df_ohlcv_x = download_data("GLBX.MDP3", "ohlcv-1m", symbol_x, start, end) df_ohlcv_y = download_data("IFEU.IMPACT", "ohlcv-1m", symbol_y, start, end) df_ohlcv_x = df_ohlcv_x["close"].to_frame(name="x") df_ohlcv_y = df_ohlcv_y["close"].to_frame(name="y") df = df_ohlcv_x.join(df_ohlcv_y, how="outer").ffill() ``` Since OHLCV aggregates only print when there's a trade in the interval, you'll need to use the `.ffill()` method to synchronize the two time series in cases where one leg doesn't print. > **Tip** > > Although CL and BRN are liquid futures contracts, there's the possibility > that both sides don't print in a given minute interval. A more complete > implementation should resample during every minute of the trading session. ## Engle-Granger cointegration test The basis of this strategy is to check that the residuals of a linear regression between the prices of the two instruments exhibits **stationarity**, also called the **Engle-Granger two-step cointegration test**. This implies that the two instruments' price series are cointegrated, and that the spread constructed based on this linear association of the two instruments' prices is stationary. In turn, this implies that the spread is **mean-reverting**. ## Cointegration vs. correlation Cointegration is preferred over correlation when constructing a pairs trade, since correlation doesn't apply to non-stationary time series, and financial instrument prices are typically non-stationary and trending. High correlation between such series can occur even without a meaningful relationship. For instance, two price series which are trending in the same direction but diverging in the long run may exhibit a spurious, positive correlation. On the other hand, cointegrated prices series may exhibit positive, negative, or even zero correlation over any period due to short-term fluctuations, but tend to track closely over the long run. ## Implementation The strategy is simple: First, over the lookback period (last 100 records), we run the cointegration test. If the two series are found to be cointegrated, then the coefficient of linear regression between the two series provides the ratio of the two instruments needed to construct the spread, also called the **hedge ratio**. We'll also compute the sample mean and standard deviation over this period. Then, over the forward window (also 100 records), we compute the spread (residual) and normalize it by taking its z-score. ```python # Initialization df["cointegrated"] = 0 df["residual"] = 0.0 df["zscore"] = 0.0 is_cointegrated = False lr = LinearRegression() for i in range(lookback, len(df), lookback): x = df["x"].iloc[i - lookback:i].values.reshape(-1, 1) y = df["y"].iloc[i - lookback:i].values.reshape(-1, 1) if is_cointegrated: # Compute and normalize signal on a forward window x_new = df["x"].iloc[i:(i + lookback)].values.reshape(-1, 1) y_new = df["y"].iloc[i:(i + lookback)].values.reshape(-1, 1) spread_back = y - lr.coef_ * x spread_forward = y_new - lr.coef_ * x_new zscore = (spread_forward - spread_back.mean()) / spread_back.std() df.iloc[i:(i + lookback), df.columns.get_loc("cointegrated")] = 1 df.iloc[i:(i + lookback), df.columns.get_loc("residual")] = spread_forward df.iloc[i:(i + lookback), df.columns.get_loc("zscore")] = zscore _, p, _ = coint(x, y) is_cointegrated = p < p_threshold lr.fit(x, y) ``` If the normalized residual is negative, it means the instrument associated with y is underpriced, i.e. we'll buy y and sell x to monetize the prediction that their prices will converge in the future. Conversely, if the normalized residual is positive, we'll sell y and buy x. Since the two products have a similar notional value and futures have large minimum order size, we'll trade a static hedge ratio of 1. If you're modifying this strategy to trade stocks or ETFs or products that have significant differences in volatility or notional value, you should consider using the dynamic hedge ratio `lr.coef_` to scale your position. > **Tip** > > The hedge ratio `lr.coef_` may be negative. This regularly occurs, for instance, if > one leg of the pair is an inverse ETF. ```python # Standardized residual is negative => y is underpriced => sell x, buy y # Standardized residual is positive => y is overpriced => buy x, sell y df["position_y"] = np.select([df["zscore"] < -entry_threshold, df["zscore"] > entry_threshold], [1, -1]) df["position_x"] = -df["position_y"] ``` It's common to specify a score-based or time-based exit condition. For example, we can exit the position when the spread converges to within +/-0.5 standard deviations. This has the effect of longer holding periods and lower transaction costs. ```python # Exit positions when z-score crosses +/- exit_threshold hold_y_long = np.where(df["zscore"] <= -exit_threshold, 1, 0) hold_y_short = np.where(df["zscore"] >= exit_threshold, 1, 0) # Carry forward positions until exit condition is met df["position_y"] = df["position_y"].mask((df["position_y"].shift() == -1) & (hold_y_short == 1), -1) df["position_y"] = df["position_y"].mask((df["position_y"].shift() == 1) & (hold_y_long == 1), 1) df["position_x"] = -df["position_y"] ``` Here's a visualization of the strategy. The outer dashed lines show the entry conditions while the inner dashed lines show the exit conditions: ![Strategy](https://databento.com/docs/spread.png.md) ## Estimating slippage Exact instantaneous slippage from crossing the spread can be calculated using MBP-1 data and execution timestamps from the backtest. In the interest of time, you can gauge the viability of the strategy by applying a constant spread cost per contract side. This is typically assumed to be half a tick or one tick for liquid futures, but we can also estimate it quickly from a small slice of MBP-1 data: ```python # Estimate slippage df_mbp1_x = download_data("GLBX.MDP3", "mbp-1", symbol_x, start) df_mbp1_y = download_data("IFEU.IMPACT", "mbp-1", symbol_y, start) avg_spread_x = (df_mbp1_x["ask_px_00"] - df_mbp1_x["bid_px_00"]).mean() avg_spread_y = (df_mbp1_y["ask_px_00"] - df_mbp1_y["bid_px_00"]).mean() ``` ## Backtest We can now compute the gross PnL and transaction costs: ```python # Compute backtest results df["pnl_x"] = df["position_x"].shift() * df["x"].diff() / tick_size_x * tick_value_x df["pnl_y"] = df["position_y"].shift() * df["y"].diff() / tick_size_y * tick_value_y df["volume_x"] = df["position_x"].diff().abs() df["volume_y"] = df["position_y"].diff().abs() df["tcost"] = ( df["volume_x"] * fees_per_side_x + df["volume_y"] * fees_per_side_y + # Exchange fees (df["volume_x"] + df["volume_y"]) * commissions_per_side + # Commissions (df["volume_x"] * avg_spread_x / tick_size_x / 2 * tick_value_x) + # Slippage (df["volume_y"] * avg_spread_y / tick_size_y / 2 * tick_value_y) # Slippage ) df["gross_pnl"] = df["pnl_x"] + df["pnl_y"] df["net_pnl"] = df["gross_pnl"] - df["tcost"] ``` ## Results ```python # Plot and print results daily_pnl = df["net_pnl"].groupby(df.index.date).sum() sr = daily_pnl.mean() / daily_pnl.std() * np.sqrt(trading_days_count) print(f"Sharpe ratio: {sr:.2f}") df[["gross_pnl", "net_pnl", "tcost"]].cumsum().plot(xlabel="Date", ylabel="PnL (USD)", title="Trading results") plt.legend(["Gross PnL", "Net PnL", "t-cost"]) plt.show() ``` ```python Sharpe ratio: 8.57 ``` ![Results](https://databento.com/docs/pairs-trading-0.svg.md) > **Info** > > The full code example can be found on our GitHub [here](https://gist.github.com/databento-bot/23ea8f2ef0de6f1290c1c6d2c87b60c3). ## Practical considerations and improvements This tutorial is kept intentionally short using a simple vectorized implementation. Ideally, you should use [MBP-1](https://databento.com/docs/schemas/mbp-1.md) data with an event-driven backtest loop that incorporates the delays between your system and the exchange gateways. This can be done using the [market replay](https://databento.com/docs/api-historical/helpers/dbn-store-replay.md) method. This strategy is very sensitive to slippage and the exact expiration month that's traded; non-lead month contracts will have wider spreads which can make the strategy infeasible. We assumed exchange fees and commissions that are realistic for a medium-sized trading firm or a team within a larger firm that's already a CME member firm. In practice, achieving similar rates requires a [NYMEX 106.J corporate membership](https://www.cmegroup.com/company/membership/corporate/nymex.html) and thousands of contract sides executed per day. Scaling up to larger trade sizes will significantly expand the strategy's state machine, since you'll have to handle partial executions and different sizes on each leg of the spread to achieve optimal hedge ratio. MBP-1 data can also inform trade sizing. We introduced a few free model and monetization parameters, namely `LOOKBACK`, `ENTRY_THRESHOLD`, `EXIT_THRESHOLD`. These can be optimized using any hyperparameter tuning method (e.g. grid search) and cross-validated. There are also a few common variations of this strategy to consider: - **Basket trading** by constructing combinations of more than two instruments, using another procedure such as the Johansen test. - Calibrating an Ornstein-Uhlenbeck process, which gives a parametric model for time-based exit. - Using **total least squares** instead of ordinary least squares to determine the hedge ratio. - Scanning a wide universe systematically for cointegrated pairs, to discover unintuitive relationships. However, this runs the risk of poor generalization if too many pairs are tested. Dimensionality reduction methods like PCA can improve the search and generalization error. ## Further reading Alexander, C. (2008). Market Risk Analysis: Practical Financial Econometrics (Vol. 2). Wiley. Tsay, R. S. (2010). Analysis of Financial Time Series (3rd edition). Wiley. Engle, R. F., & Granger, C. W. J. (1987). Co-integration and error correction: Representation, estimation, and testing. Econometrica, 55(2), 251-276. # Build a real-time stock screener In this example, we will build a real-time scanner for approximately 9,000 tickers in the U.S. equity markets. We'll use the [OHLCV schema](https://databento.com/docs/schemas/ohlcv.md) from the [Databento US Equities Summary](https://databento.com/docs/datasets/equs-summary.md) (`EQUS.SUMMARY`) dataset. This dataset provides end-of-day summary statistics for all RegNMS symbols via the Nasdaq NLS+ feed. These statistics include consolidated OHLCV data across all U.S. equity venues. We'll also use the [MBP-1 schema](https://databento.com/docs/schemas/mbp-1.md) from the [Databento US Equities Mini](https://databento.com/docs/datasets/equs-mini.md) (`EQUS.MINI`) dataset. This is a derived top-of-book dataset that provides aggregated BBO and trade information from a variety of proprietary market data feeds. > **Info** > > A [Databento US Equities]($PLATFORM_HOSTNAME/equities) Standard subscription is required to run this example. ```python import databento as db import pandas as pd class PriceMovementScanner: """Scanner for detecting large price movements in all US equities.""" def __init__( self, pct_threshold: float = 0.03, # Default threshold for alert ) -> None: """Initialize scanner with a configurable threshold.""" self.pct_threshold = pct_threshold self.start = pd.Timestamp.now(tz="US/Eastern").normalize() self.premarket_start = self.start.replace(hour=4) # Start of today's pre-market session self.symbol_directory: dict[int, str] = {} self.last_day_lookup: dict[str, float] = self.get_last_day_lookup() self.is_signal_lit: dict[str, bool] = {symbol: False for symbol in self.last_day_lookup} def get_last_day_lookup(self) -> dict[str, float]: """Get yesterday's closing prices for all symbols.""" client = db.Historical() # Get OHLCV-1d data from the previous session data = client.timeseries.get_range( dataset="EQUS.SUMMARY", schema="ohlcv-1d", symbols="ALL_SYMBOLS", start=(self.start - pd.offsets.BusinessDay(1)).date(), ) # Request symbology: This is required for ALL_SYMBOLS requests # which don't automatically map instrument ID to raw ticker symbol symbology_json = data.request_symbology(client) data.insert_symbology_json(symbology_json) df = data.to_df() return df.set_index("symbol")["close"].to_dict() def scan(self, event: db.DBNRecord) -> None: """ Scan for large price movements in market data events. """ if isinstance(event, db.SymbolMappingMsg): self.symbol_directory[event.instrument_id] = event.stype_out_symbol return if not isinstance(event, db.MBP1Msg): return symbol = self.symbol_directory[event.instrument_id] # Skip if alert already triggered for the symbol if self.is_signal_lit[symbol]: return bid = event.levels[0].pretty_bid_px ask = event.levels[0].pretty_ask_px # Skip if one side of the book is empty if pd.isna(bid) or pd.isna(ask): return # Calculate change since the previous close mid = (bid + ask) / 2 previous_close = self.last_day_lookup.get(symbol) # New listing, no data from previous session if previous_close is None: return pct_change = (mid - previous_close) / previous_close # Trigger alert if the threshold is exceeded and no previous alert if abs(pct_change) > self.pct_threshold: ts = event.pretty_ts_event.tz_convert("US/Eastern") print( f"[{ts.isoformat()}] {symbol} moved by {pct_change:.2%} " f"(current: {mid:.4f}, previous: {previous_close:.4f})", ) self.is_signal_lit[symbol] = True if __name__ == "__main__": # Instantiate scanner class scanner = PriceMovementScanner() # Create a live client live = db.Live(key="$YOUR_API_KEY") # Subscribe MBP-1 schema for ALL_SYMBOLS # Start subscription at pre-market open live.subscribe( dataset="EQUS.MINI", schema="mbp-1", symbols="ALL_SYMBOLS", start=scanner.premarket_start, ) # Add callback and start live.add_callback(scanner.scan) live.start() # Run indefinitely live.block_for_close() ``` ```plaintext [2025-05-06T04:00:00.023079908-04:00] CORN moved by 4.19% (current: 19.1400, previous: 18.3700) [2025-05-06T04:00:00.046518714-04:00] CPER moved by 4.04% (current: 30.4200, previous: 29.2400) [2025-05-06T04:00:00.047119403-04:00] FETH moved by -8.56% (current: 16.6050, previous: 18.1600) [2025-05-06T04:00:00.069275113-04:00] XXRP moved by -3.80% (current: 29.3300, previous: 30.4900) [2025-05-06T04:00:00.118536539-04:00] ETHT moved by 24.40% (current: 5.7100, previous: 4.5900) ... [2025-05-06T09:33:43.338614253-04:00] XMVM moved by 19.04% (current: 62.5450, previous: 52.5400) [2025-05-06T09:33:46.814411294-04:00] BELFB moved by -5.09% (current: 65.4500, previous: 68.9600) [2025-05-06T09:33:53.530064671-04:00] CATO moved by -3.25% (current: 2.2350, previous: 2.3100) [2025-05-06T09:33:53.711869173-04:00] WRB moved by -11.42% (current: 64.2150, previous: 72.4900) [2025-05-06T09:33:55.223052174-04:00] QUS moved by -3.72% (current: 149.8250, previous: 155.6100) ``` ## Corporate actions | Title | Subtitle | Supported languages | | [Dividends](https://databento.com/docs/examples/corporate-actions/dividends.md) | Obtain regular and special dividend information. | Python, Historical | | [New listings](https://databento.com/docs/examples/corporate-actions/new-listings.md) | Explore new listings on US and global markets. | Python, Historical | | [Splits and reverse splits](https://databento.com/docs/examples/corporate-actions/splits-and-reverse-splits.md) | Handle forward and reverse stock split capital events. | Python, Historical | | [Mergers and demergers](https://databento.com/docs/examples/corporate-actions/mergers-and-demergers.md) | Analyze company merger and demerger events. | Python, Historical | # Dividends > **Info** > > The Databento team is currently working on this article. ## Overview Dividends are a form of capital distribution paid to shareholders. They can be issued in various forms, most commonly as cash payments or additional shares of stock. Companies may pay dividends on a regular basis, such as quarterly, or as special one-off dividends that are typically issued due to exceptional earnings or significant events. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting dividends This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all Costco dividends from the beginning of 2019 through 2020. Costco is a good example of a company that pays a regular quarterly dividend. Additionally, we can observe the inclusion of a special dividend during this period. We can query for dividends specifically by filtering for the `DIV` (Dividend) event type. Once we obtain the records, we can filter the relevant columns for analysis. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions df_raw = ref_client.corporate_actions.get_range( symbols=["COST"], start="2019", end="2021", index="ex_date", events=["DIV"], countries=["US"], ) # Filter relevant columns columns = [ "record_date", "payment_date", "payment_type", "marker", "frequency", "gross_dividend", "operating_mic", "isin", "symbol", "issuer_name", ] df = df_raw[columns] df.head(10) ``` ## Result ```plaintext record_date payment_date payment_type marker frequency gross_dividend operating_mic isin symbol issuer_name ex_date 2019-02-07 2019-02-08 2019-02-22 C INT QTR 0.57 XNAS US22160K1051 COST Costco Wholesale Corp 2019-05-09 2019-05-10 2019-05-24 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2019-08-29 2019-08-30 2019-09-13 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2019-10-31 2019-11-01 2019-11-15 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2020-02-06 2020-02-07 2020-02-21 C INT QTR 0.65 XNAS US22160K1051 COST Costco Wholesale Corp 2020-04-30 2020-05-01 2020-05-15 C INT QTR 0.70 XNAS US22160K1051 COST Costco Wholesale Corp 2020-07-30 2020-07-31 2020-08-14 C INT QTR 0.70 XNAS US22160K1051 COST Costco Wholesale Corp 2020-10-29 2020-10-30 2020-11-13 C INT QTR 0.70 XNAS US22160K1051 COST Costco Wholesale Corp 2020-12-01 2020-12-02 2020-12-11 C SPL UN 10.00 XNAS US22160K1051 COST Costco Wholesale Corp ``` ## Special dividends Special dividends are one-time or irregular dividend payments made by a company, typically in addition to regular dividend distributions. Unlike regular dividends, which are usually paid on a fixed schedule (e.g., quarterly or annually), special dividends are declared in response to exceptional circumstances, such as significant financial gains, asset sales, or other extraordinary events. For example, consider the special dividend paid by Costco on December 11, 2020: - `marker` was `SPL` (Special) - `frequency` was `UN` (Unspecified) - The dividend was paid less than a month after the previous quarterly dividend ## Fields The table below provides detailed descriptions for each **dividend-specific** field found in the [dynamic info JSON fields](https://databento.com/docs/schemas/corporate-actions.md#dynamic-fields). | Field | Description | | `marker` | The dividend marker. See the [MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker) enum lookup table. | | `frequency` | The dividend frequency. See the [FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq) enum lookup table. | | `declaration_date` | The date the dividend was announced by the issuer. | | `period_end_date` | The end of the financial period for which the dividend is being paid. | | `foreign_ex_date` | The date on which the US dollar rate to `currency` value was taken. | | `ex_date2` | This is the ex date value for stock component of a dividend with a `payment_type` of (**B**)oth stock and cash. The main `ex_date` field will be for the cash payment portion of the same record. | | `payment_date2` | This is the payment date value for stock component of a dividend with a `payment_type` of (**B**)oth stock and cash. The main `payment_date` field will be for the cash payment portion of the same record. | | `tba_flag` | The company intends to pay a dividend but the details are yet to be announced. | | `nil_dividend` | A real dividend event but of zero payrate. | | `div_rescind` | A dividend event but of zero payrate. | | `decl_currency` | The declared currency - ISO 4217. | | `decl_gross_amount` | The declared gross amount. | | `approx_flag` | If dividend is temporarily an approximate value. | | `tax_rate` | The domestic withholding tax rate. | | `coupon` | In countries where coupons are to be submitted to obtain a benefit this field is useful. | | `coupon_id` | The coupon ID. | | `div_rate` | The rate of dividend as a percentage of par value. | | `cur_pair` | Identifies the different currencies involved for the FX rate to be applied - ISO 4217. | | `dep_fees` | Depositary Fees - applies only to `DR` dividend. | | `dep_fees_currency` | Depositary Fees currency - ISO 4217. | | `rescind_cash_div` | If cash dividend cancelled. | | `rescind_stock_div` | If stock dividend cancelled. | | `usd_rate_to_currency` | The US dollar exchange rate for dividend currency. | | `div_postponed` | Indicating postponement of dividend payment. | ## Dividend types The `marker` field specifies the type of dividend. See [MARKER](https://databento.com/docs/conventions/reference-data-enums.md#marker) enum lookup table for possible values. The table below provides detailed descriptions for each dividend type: | Value | Description | | `ANL` | **Annual**: Refers to a dividend that is paid once per year. | | `ARR` | **Arrears**: Denotes a dividend payment made to cover missed or deferred dividends from previous periods. | | `CG` | **Capital Gain**: Indicates a dividend that is categorized as a gain realized from the appreciation of an asset. | | `CGL` | **Capital Gain Long Term**: Refers to a capital gain from an asset held for more than a year before being sold, often benefiting from preferential tax treatment. | | `CGS` | **Capital Gain Short Term**: Refers to a capital gain from an asset held for less than a year before being sold, typically subject to higher tax rates. | | `FNL` | **Final**: Indicates that this dividend is the last payment related to a particular series or event. | | `INS` | **Installment**: Refers to dividends paid in multiple payments over a period rather than a single lump sum. | | `INT` | **Interim**: A dividend paid between regular annual dividends, often reflecting earnings for a part of the year. | | `ISC` | **Interest on Capital**: Represents payments made as interest on the capital invested in a company or financial product. | | `MEM` | **Memorial**: Indicates a dividend paid in memory of an individual, often reflecting a special purpose or tribute. | | `QTR` | **Quarterly**: Refers to dividends paid every three months. | | `REG` | **Regular**: Denotes dividends that are paid at regular intervals, typically on a fixed schedule. | | `SMA` | **Semi-annual**: Refers to dividends paid twice a year. | | `SPL` | **Special**: Indicates a one-time or irregular dividend payment, often separate from regular dividends. | | `SUP` | **Supplementary**: Refers to additional dividends paid in addition to regular payments, usually reflecting extra earnings or adjustments. | | `UN` | **Unspecified**: Indicates that the dividend type is not defined or categorized under the specified options. | | `VAR` | **Variable**: Refers to dividends that can vary in amount or frequency, based on company performance or other factors. | ## Dividend frequency The `frequency` field specifies the frequency of dividend payments or occurrences. See [FREQ](https://databento.com/docs/conventions/reference-data-enums.md#freq) enum lookup table for possible values. The table below provides detailed descriptions for each dividend frequency: | Value | Description | | `28D` | **28 Days**: Dividend occurs every 28 days. | | `35DAY` | **Every 35 Days**: Dividend occurs every 35 days. | | `BIM` | **Bi-Monthly**: Dividend occurs every two months. | | `DLY` | **Daily**: Dividend occurs every day. | | `FNL` | **Final**: The last dividend in a series. | | `INT` | **Interim**: Dividend occurring between regular annual payments. | | `INTONMAT` | **Arrears**: Interest paid upon the maturity of a financial product. | | `INTONTRIG` | **Interest on Trigger**: Interest paid when a specific trigger condition is met. | | `IRG` | **Irregular**: Dividend occurs at irregular intervals. | | `ITM` | **Interest on Maturity**: Interest paid upon the maturity of a financial product. | | `MNT` | **Monthly**: Dividend occurs every month. | | `QTR` | **Quarterly**: Dividend occurs every three months. | | `SMA` | **Semi-Annual**: Dividend occurs twice a year. | | `TRM` | **Trimesterly**: Dividend occurs every four months (three times a year). | | `UN` | **Unspecified**: Dividend frequency is not defined or categorized. | | `WKL` | **Weekly**: Dividend occurs every week. | # New listings > **Info** > > The Databento team is currently working on this article. ## Overview New listings refer to the addition of securities to a stock exchange, allowing them to be publicly traded. This can include initial public offerings (IPOs) of companies entering the public market for the first time, as well as existing companies that choose to list additional securities or switch their listings to a different exchange. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting new listings This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all new US market listings for 31st July 2024. We can query for new listings specifically by filtering for the `NLIST` (New Listing) event type. Once we obtain the records, we can filter the relevant columns for analysis. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions df_raw = ref_client.corporate_actions.get_range( symbols=["ALL_SYMBOLS"], start="2024-07-31", events=["NLIST"], countries=["US"], ) # Drop rows where there is not yet a confirmed symbol or ISIN df = df_raw.dropna(subset=["symbol", "isin"]) # Filter relevant columns columns = [ "event", "operating_mic", "isin", "symbol", "issuer_name", "security_type", ] df = df[columns] df.head(20) ``` ## Result ```plaintext event operating_mic isin symbol issuer_name security_type event_date 2024-07-31 NLIST OTCM US56564V1199 MAQCW Maquia Capital Acquisition Corp WAR 2024-07-31 NLIST ARCX US31423L5030 FLCC Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US31423L6020 FSCC Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US31423L7010 FLCV Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US31423L8000 FLCG Federated Hermes ETF Trust ETF 2024-07-31 NLIST ARCX US26922B4692 BDIV ETF Series Solutions Trust ETF 2024-07-31 NLIST ARCX US26922B4775 SAWG ETF Series Solutions Trust ETF 2024-07-31 NLIST ARCX US26922B4858 SAWS ETF Series Solutions Trust ETF 2024-07-31 NLIST ARCX US3899301085 BTC Grayscale Bitcoin Mini Trust ETC 2024-07-31 NLIST OTCM US56564V2007 MAQCU Maquia Capital Acquisition Corp STP 2024-07-31 NLIST OTCM KYG9879S1003 YSBIF YSB Inc. EQS 2024-07-31 NLIST OTCM US56564V1017 MAQC Maquia Capital Acquisition Corp EQS 2024-07-31 NLIST OTCM TH0010010013 SUOFF Saha-Union Public Company Ltd EQS 2024-07-31 NLIST OTCM CA09643M1059 BSKCF BluSky Carbon Inc. EQS 2024-08-01 NLIST OTCM ID1000198500 BENGF Barito Renewables Energy Tbk PT EQS 2024-08-01 NLIST OTCM FR0014005AL0 ANFPF Antin Infrastructure Partners EQS 2024-08-01 NLIST ARCX US45783Y1459 IAUG Innovator ETFs Trust ETF 2024-08-01 NLIST BATS US45783Y1376 KAUG Innovator ETFs Trust ETF 2024-08-01 NLIST BATS US45783Y1111 ZAUG Innovator ETFs Trust ETF 2024-08-01 NLIST BATS US00888H5625 AUGU AIM ETF Products Trust ETF ``` # Splits and reverse splits > **Info** > > The Databento team is currently working on this article. ## Overview A forward stock split (often just referred to as a stock split), is when an issuer increases the number of outstanding shares by dividing each existing share into multiple shares, thereby reducing the per-share price. A reverse stock split in contrast, involves reducing the number of outstanding shares by consolidating multiple existing shares into a single share, which increases the per-share price. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting US stock splits This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all US stock splits for January 2024. We can query for all forward and reverse stock splits by filtering specifically for the following event types: - `FSPLT` (Forward Split) - `RSPLT` (Reverse Split) ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the stock split events df_raw = ref_client.corporate_actions.get_range( symbols="ALL_SYMBOLS", start="2024-01", end="2024-02", index="ex_date", events=["FSPLT", "RSPLT"], countries=["US"], ) # Filter relevant columns columns = [ "event", "ratio_old", "ratio_new", "operating_mic", "isin", "symbol", "issuer_name", "security_description", ] df = df_raw[columns] df.head(10) ``` ## Result ```plaintext event ratio_old ratio_new operating_mic isin symbol issuer_name security_description ex_date 2024-01-02 RSPLT 20.0 1.0 XNAS US70261F1030 KTTA Pasithea Therapeutics Corp Ordinary Shares 2024-01-02 RSPLT 65.0 1.0 XNAS US89157G8841 TBLT Toughbuilt Industries Inc Ordinary Shares 2024-01-03 RSPLT 500.0 1.0 OTCM US4662993025 JZZI Ideal Group of Companies Inc. Ordinary Shares 2024-01-03 RSPLT 80.0 1.0 XNAS US88032L5066 TENX Tenax Therapeutics Inc Ordinary Shares 2024-01-03 CONSD 10.0 1.0 OTCM HK8028008944 TMLSF Timeless Resources Holdings Limited Ordinary Shares 2024-01-03 RSPLT 2.0 1.0 XNAS US53934A1152 LXEH Lixiang Education Holding Co Ltd ADR 2024-01-03 RSPLT 40.0 1.0 XNAS US71360T1016 PRSO Peraso Inc Ordinary Shares 2024-01-04 CONSD 2.0 1.0 OTCM CA46123W1041 IMTCF Intrepid Metals Corp Ordinary Shares 2024-01-04 RSPLT 10.0 1.0 XNAS US4718711033 JSPR Jasper Therapeutics Inc Ordinary Shares 2024-01-04 RSPLT 16.0 1.0 XNAS US6551871022 ACON Aclarion Inc Ordinary Shares ``` ## NVDA splits This example demonstrates how to retrieve all stock split events for Nvidia Corporation since March 2018. We'll make a request from the corporate actions API for listed Nvidia (`NVDA`) securities for the entire range of dataset history (starting 2018-05-01). All records returned relate to Nvidia Corporation shares listed on Nasdaq (`XNAS`). ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the stock split events df_raw = ref_client.corporate_actions.get_range( symbols=["NVDA"], start="2018-05", index="ex_date", events=["FSPLT", "RSPLT"], countries=["US"], ) # Filter relevant columns columns = [ "event", "ratio_old", "ratio_new", "operating_mic", "isin", "symbol", "issuer_name", "security_description", ] df = df_raw[columns] df.head() ``` ## Result ```plaintext event ratio_old ratio_new operating_mic isin symbol issuer_name security_description ex_date 2021-07-20 FSPLT 1 4 XNAS US67066G1040 NVDA NVIDIA Corp Ordinary Shares 2024-06-10 FSPLT 1 10 XNAS US67066G1040 NVDA NVIDIA Corp Ordinary Shares ``` # Mergers and demergers > **Info** > > The Databento team is currently working on this article. ## Overview Mergers and demergers are significant corporate actions that can fundamentally alter the structure and strategy of companies. A merger involves the combination of two or more companies into a single entity. Conversely, a demerger involves a company splitting into two or more independent entities. The following examples will highlight some different merger and demerger scenarios with a particular emphasis on the continuity of the identifiers used by the corporate actions dataset. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md) dataset guide for further details. ## Requesting mergers and demergers This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all mergers and demergers for January 2024. We can query for all merger and demerger activity by filtering specifically for the following event types: - `MRGR` (Merger) - `DMRGR` (Demerger) - `CONSD` (Consolidation) - `SOFF` (Spin-Off) ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the merger and demerger events df_raw = ref_client.corporate_actions.get_range( symbols="ALL_SYMBOLS", start="2024-01", end="2024-02", events=["MRGR", "DMRGR", "CONSD", "SOFF"], countries=["US"], ) # Filter relevant columns columns = [ "event", "operating_mic", "isin", "symbol", "issuer_name", "security_description", ] df = df_raw[columns] df.head(20) ``` ## Result ```plaintext event operating_mic isin symbol issuer_name security_description event_date 2024-01-03 SOFF XNAS SG9999000020 FLEX Flex Ltd Ordinary Shares 2024-01-03 CONSD OTCM HK8028008944 TMLSF Timeless Resources Holdings Limited Ordinary Shares 2024-01-04 CONSD OTCM CA46123W1041 IMTCF Intrepid Metals Corp Ordinary Shares 2024-01-04 MRGR XNYS US53814L1089 LTHM Livent Corp Ordinary Shares 2024-01-06 CONSD OTCM CA15722J1030 CYLYF Ceylon Graphite Corp Ordinary Shares 2024-01-06 CONSD OTCM CA74359L1058 PMCOF Prospector Metals Corp Ordinary Shares 2024-01-08 CONSD OTCM CA7481191046 QNICF Quebec Nickel Corp Ordinary Shares 2024-01-09 CONSD OTCM NO0003070609 IDXAF IDEX Biometrics ASA Ordinary Shares 2024-01-09 DMRGR OTCM CA51440M1023 LANRF Lancaster resources inc Ordinary Shares 2024-01-11 SOFF XNAS US70387R4039 PAVM PAVmed Inc Ordinary Shares 2024-01-11 CONSD OTCM SE0007692124 IMNTF Vidhance AB Ordinary Shares 2024-01-12 CONSD OTCM CA60928P1071 MEAUF Moneta Gold Inc Ordinary Shares 2024-01-16 DMRGR OTCM CA30330R1082 FUUFF F3 Uranium Corp Ordinary Shares 2024-01-16 SOFF XNYS US60784B1017 MDV Modiv Industrial Inc Ordinary Shares - Class C 2024-01-16 DMRGR OTCM CA30330R1082 FUUFF F3 Uranium Corp Ordinary Shares 2024-01-16 CONSD XASE CA98980M1095 ZOM Zomedica Corp Ordinary Shares 2024-01-16 CONSD OTCM CA00208E2087 AATGF ATI Airtest Technologies, Inc. Ordinary Shares 2024-01-17 CONSD OTCM GB00BLG2TX24 RPNRF Rapid Nutrition PLC Registered Shares 2024-01-18 CONSD OTCM CA53681M1068 GLIOF Lithium Lion Metals Inc Ordinary Shares 2024-01-18 DMRGR OTCM FR0000121220 SDXOF Sodexo Ordinary Shares ``` ## Fiat Chrysler Automobiles and PSA Group merger In January 2021, Fiat Chrysler Automobiles (`FCAU`) and PSA Group (`UG`) completed a merger to form Stellantis (`STLA`), a global automotive giant. This merger combined the strengths and resources of both companies to enhance their competitive edge in the automotive industry. **Original companies** - Fiat Chrysler Automobiles listed on the New York Stock Exchange under the ticker symbol `FCAU`. - PSA Group listed on Euronext Paris under the ticker symbol `UG`. **New company** - Following the merger, Stellantis listed on the New York Stock Exchange under the ticker symbol `STLA`. To analyze the merger to Stellantis (`STLA`), we can request all Fiat Chrysler (`FCAU`) corporate actions for January 2021 and call `.head()` on the data frame. The merger and name change is evident with the record having an `event` of `ISCHG` (Issuer Name Change), an `event_date_label` of `name_change_date` on an `effective_date` of 2021-01-17. Notice the identical `listing_id`, `security_id` and `issuer_id` records for the NYSE listed stock Stellantis (`STLA`) changing symbol to `FCAU`. > **See also** > > [Corporate actions](https://databento.com/docs/datasets/corporate-actions.md#listing-continuity) dataset guide for further details on listing continuity. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions for Fiat Chrysler Automobiles for January 2021 df_raw = ref_client.corporate_actions.get_range( symbols=["FCAU"], start="2021-01", end="2021-02", countries=["US"], ) # Filter relevant columns columns = [ "event", "event_date_label", "issuer_id", "security_id", "listing_id", "operating_mic", "isin", "symbol", "issuer_name", ] df = df_raw[columns] df.head(10) ``` ## Result ```plaintext event event_date_label issuer_id security_id listing_id operating_mic isin symbol issuer_name event_date 2021-01-04 AGM meeting_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-14 DIV ex_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-17 ISCHG name_change_date I-166801 S-4329550 L-16138235 XNYS NL0010877643 FCAU Stellantis N.V 2021-01-18 ICC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-18 BBCC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-18 BBEC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-19 LCC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-19 BBCC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-19 BBEC effective_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V 2021-01-25 ANN notification_date I-166801 S-4329550 L-16138235 XNYS NL00150001Q9 STLA Stellantis N.V ``` ## DowDuPont demerger In 2019, DowDuPont completed its planned separation into three independent, publicly traded companies. This strategic move aimed to unlock value by allowing each company to focus on its core strengths, potentially leading to better operational performance and market valuation. **Original company** - DowDuPont (`DWDP`) formed through the merger of Dow Chemical and DuPont on August 31, 2017. **New companies** - Dow Inc. listed on the New York Stock Exchange under the ticker symbol `DOW`. - DuPont de Nemours, Inc. listed on the New York Stock Exchange under the ticker symbol `DD`. - Corteva, Inc. listed on the New York Stock Exchange under the ticker symbol `CTVA`. To analyze the DowDuPont split, request all DowDuPont (`DWDP`) corporate actions for 2019 and call `.head()` on the data frame. The demerger and name changes are evident with records indicating events such as `ISCHG` (Issuer Name Change) and `SHOCH` (Shares Outstanding Change), showing transitions to Dow Inc. (`DOW`), DuPont (`DD`), and Corteva (`CTVA`). Note the `listing_id` for DuPont of `L-17235812` continued post spin-offs, retaining ticker symbol `DD`. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request corporate actions for DowDuPont for 2019 df = ref_client.corporate_actions.get_range( symbols=["DWDP"], start="2019", countries=["US"], ) # Filter relevant columns columns = [ "event", "event_date_label", "issuer_id", "security_id", "listing_id", "operating_mic", "isin", "symbol", "issuer_name", ] df = df[columns] df.head(10) ``` ## Result ```plaintext event event_date_label issuer_id security_id listing_id operating_mic isin symbol issuer_name event_date 2019-01-31 SHOCH effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-02-27 DIV ex_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-02 SOFF ex_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-03 ANN notification_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-26 SHOCH effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-04-30 SHOCH effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-05-02 DIV ex_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-05-06 ANN notification_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-05-23 AGM meeting_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc 2019-06-01 ICC effective_date I-197078 S-5229608 L-17235812 XNYS US26614N1028 DD DuPont de Nemours Inc ``` ## Adjustment factors | Title | Subtitle | Image | Supported languages | | [Applying adjustment factors](https://databento.com/docs/examples/adjustment-factors/applying-adjustment-factors.md) | Factor in the effects of capital events on securities using adjustment factors. | ![example1](https://databento.com/docs/applying-adjustment-factors-tsla-thumbnail.svg.md) | Python, Historical | | [Handling multiple stock selections](https://databento.com/docs/examples/adjustment-factors/multiple-adjustment-factors.md) | Back-adjust closing prices over a selection of stocks using adjustment factors. | ![example2](https://databento.com/docs/multiple-adjustment-factors-thumbnail.svg.md) | Python, Historical | # Applying adjustment factors ## Overview The following examples demonstrate how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all relevant adjustment factors for corporate actions affecting a specific equity. These factors are then applied to daily close prices (using UTC date boundaries) to create a smoother time series that accounts for the impact of the capital events. > **See also** > > [Adjustment factors](https://databento.com/docs/datasets/adjustment-factors.md) dataset guide for further details and considerations for applying > adjustment factors. ## Procedure The following steps outline the procedure for both examples: - **Obtain data.** Retrieve the adjustment factors for the desired security and time range - **Pre-process data.** - Filter single listings - Filter (**A**)pply status events - Select one `option` per corporate action event - Convert the `ex_date` index to tz-aware UTC timestamps - **Apply adjustment factors.** Iterate over all factors and apply the cumulative product to all records prior to the `ex_date` ## Dividend adjustments In the first example, we will use Microsoft Corporation (MSFT) shares, known for consistently issuing quarterly dividends. Our time range will be from the beginning of June 2018 to the end of June 2024. We will observe the cumulative effect of these dividends on the price series over time and plot both the adjusted and original unadjusted daily close prices. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbol = "MSFT" start = "2018-06" end = "2025-04" # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request adjustment factors data for the relevant time range df_factors = ref_client.adjustment_factors.get_range( symbols=symbol, start=start, end=end, countries=["US"], ) # Filter listing on Nasdaq/NGS (Global Select Market) df_factors = df_factors[df_factors["operating_mic"] == "XNGS"] # Filter factors with (A)pply status df_factors = df_factors[df_factors["status"] == "A"] # Filter option 1 (default) for all factors df_factors = df_factors[(df_factors["option"] == 1)] # Convert the ex_dates index to UTC timestamps df_factors.index = pd.to_datetime(df_factors.index, utc=True) # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request daily bars data to obtain close prices data = hist_client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbol, start=start, end=end, ) df_bars = data.to_df() # Apply the adjustment factors: # Create a new dataframe for the adjusted prices df_adjusted = pd.DataFrame(index=df_bars.index) # Copy the close column from df_bars df_adjusted = df_bars[["close"]].copy() df_adjusted["close_adjusted"] = df_adjusted["close"] # Iterate through the factors and apply cumulative product for ex_date, factor in zip(df_factors.index, df_factors["factor"]): df_adjusted.loc[df_adjusted.index < ex_date, "close_adjusted"] *= factor # Plot the results plt.plot(df_adjusted["close"], label="close") plt.plot(df_adjusted["close_adjusted"], label="adjusted") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Daily Close Price (USD)") plt.title(f"{symbol}") plt.show() ``` ![MSFT adjusted close](https://databento.com/docs/applying-adjustment-factors-msft-0.svg.md) ## Forward split adjustments In the second example, we will use Tesla Inc (`TSLA`) shares, which have experienced significant price growth in recent years. Companies often split their shares to lower the stock price after rapid growth. Our analysis will cover the period from the beginning of June 2018 to the end of June 2024. Tesla underwent two major stock splits in August 2020 and August 2022, resulting in substantial step changes in the close price series. We will apply adjustment factors to account for these splits and create a smoother time series that reflects the true underlying price over time. We'll also plot both the adjusted and original unadjusted daily close prices. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbol = "TSLA" start = "2018-06" end = "2025-04" # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request adjustment factors data for the relevant time range df_factors = ref_client.adjustment_factors.get_range( symbols=symbol, start=start, end=end, countries=["US"], ) # Filter listing on Nasdaq/NGS (Global Select Market) df_factors = df_factors[df_factors["operating_mic"] == "XNGS"] # Filter factors with (A)pply status df_factors = df_factors[df_factors["status"] == "A"] # Filter option 1 (default) for all factors df_factors = df_factors[(df_factors["option"] == 1)] # Convert the ex_dates index to UTC timestamps df_factors.index = pd.to_datetime(df_factors.index, utc=True) # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request daily bars data to obtain close prices data = hist_client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbol, start=start, end=end, ) df_bars = data.to_df() # Apply the adjustment factors: # Create a new dataframe for the adjusted prices df_adjusted = pd.DataFrame(index=df_bars.index) # Copy the close column from df_bars df_adjusted = df_bars[["close"]].copy() df_adjusted["close_adjusted"] = df_adjusted["close"] # Iterate through the factors and apply cumulative product for ex_date, factor in zip(df_factors.index, df_factors["factor"]): df_adjusted.loc[df_adjusted.index < ex_date, "close_adjusted"] *= factor # Plot the results plt.plot(df_adjusted["close"], label="close") plt.plot(df_adjusted["close_adjusted"], label="adjusted") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Daily Close Price (USD)") plt.title(f"{symbol}") plt.show() ``` ![TSLA adjusted close](https://databento.com/docs/applying-adjustment-factors-tsla-0.svg.md) # Handling multiple stock selections ## Overview The following example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all relevant adjustment factors for corporate actions affecting a selection of stocks. These factors are then applied to daily close prices (using UTC date boundaries) to create a smoother time series that accounts for the impact of the capital events. > **See also** > > [Adjustment factors](https://databento.com/docs/datasets/adjustment-factors.md) dataset guide for further details and considerations for applying > adjustment factors. ## Procedure We'll use the same procedure as outlined in the [Applying adjustment factors](https://databento.com/docs/examples/adjustment-factors/applying-adjustment-factors.md#procedure) tutorial. The only difference is that we'll now apply the factors to multiple listings simultaneously. ## Selection The selected stocks are prominent representatives of the tech sector, chosen to illustrate various scenarios where adjustment factors are applied: - **Microsoft Corp.** (`MSFT`): Pays regular quarterly cash dividends. - **Apple Inc.** (`AAPL`): Pays regular quarterly cash dividends. - **Tesla Inc.** (`TSLA`): Underwent two forward stock splits on 2020-08-31 and 2022-08-25. - **Amazon.com Inc.** (`AMZN`): Underwent a forward stock split on 2022-06-06, and issued a cash dividend on 2022-12-27. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbols = ["MSFT", "AAPL", "TSLA", "AMZN"] start = "2018-06" end = "2025-04" # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request adjustment factors data for the relevant time range df_factors = ref_client.adjustment_factors.get_range( symbols=symbols, start=start, end=end, countries=["US"], ) # Filter listings on Nasdaq/NGS (Global Select Market) df_factors = df_factors[df_factors["operating_mic"] == "XNGS"] # Filter factors with (A)pply status df_factors = df_factors[df_factors["status"] == "A"] # Filter option 1 (default) for all factors df_factors = df_factors[(df_factors["option"] == 1)] # Convert the ex_dates index to UTC timestamps df_factors.index = pd.to_datetime(df_factors.index, utc=True) # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request daily bars data to obtain close prices data = hist_client.timeseries.get_range( dataset=dataset, schema="ohlcv-1d", symbols=symbols, start=start, end=end, ) df_bars = data.to_df() def apply_factors(df_adjusted, df_bars, factors_symbol, symbol): for ex_date, factor in zip(factors_symbol.index, factors_symbol["factor"]): mask = (df_adjusted.index < ex_date) & (df_bars["symbol"] == symbol) df_adjusted.loc[mask, "close_adjusted"] *= factor # Apply the adjustment factors: # Create a new dataframe for the adjusted prices df_adjusted = pd.DataFrame(index=df_bars.index) df_adjusted["symbol"] = df_bars["symbol"] df_adjusted["close_adjusted"] = df_bars["close"].copy() for symbol in symbols: # Filter the bars and factors for the current symbol bars_symbol = df_bars[df_bars["symbol"] == symbol] factors_symbol = df_factors[df_factors["symbol"] == symbol] # Apply the factors using the `apply_factors` function defined above apply_factors(df_adjusted, df_bars, factors_symbol, symbol) # Plot the results for symbol in symbols: df_symbol = df_adjusted[df_adjusted["symbol"] == symbol] plt.plot(df_symbol.index, df_symbol["close_adjusted"], label=f"{symbol} adjusted") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Daily Close Price (USD)") plt.title("Adjusted Close Prices for Selected Stocks") plt.show() ``` ![Multiple adjusted close](https://databento.com/docs/multiple-adjustment-factors-0.svg.md) ## Security master | Title | Subtitle | Image | Supported languages | | [Enrich instrument definitions](https://databento.com/docs/examples/security-master/enrich-instrument-definitions.md) | Use the security master to enrich instrument definitions. | ![placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Listings and delistings](https://databento.com/docs/examples/security-master/listings-and-delistings.md) | Track security listings, delistings and symbology changes across time with the security master point-in-time (PIT) records. | ![placeholder](https://databento.com/docs/placeholder.svg.md) | Python, Historical | | [Market capitalization change](https://databento.com/docs/examples/security-master/market-cap-change.md) | Combine daily close prices with shares outstanding to analyze changes in market capitalization over time. | ![example3](https://databento.com/docs/market-cap-change-thumbnail.svg.md) | Python, Historical | # Enrich instrument definitions ## Overview This example demonstrates how to merge security master data with the [Definition schema](https://databento.com/docs/schemas/instrument-definitions.md) using the [Historical client](https://databento.com/docs/api-historical/client.md). This gives you the benefit of multiple symbology identifiers and additional security and listing information included with each record. For our purposes we'll assume the latest definition schema can be obtained on 2024-04-01. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. ## Instrument identification and symbology When identifying a listing on a specific venue, we suggest using the `nasdaq_symbol` field, as it aligns with the normalized `raw_symbol` for instruments. It's also important to correctly match the venue, being the `operating_mic` ISO 10838 Market Identifier Code (MIC) with the definitions `exchange` field. By utilizing the [ISIN](https://en.wikipedia.org/wiki/International_Securities_Identification_Number), you can more precisely identify the security within the security master and corporate actions datasets. When querying these datasets, specifying the `isin` symbology type ensures the most accurate results. ```python import databento as db import pandas as pd # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request the instrument definitions data = hist_client.timeseries.get_range( dataset="XNAS.ITCH", symbols=["AAPL"], start="2024-04-01", limit=1, ) df_definition = data.to_df() # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request the latest security master df_security_master = ref_client.security_master.get_last( symbols=["AAPL"], countries=["US"], ) # Merge the dataframes by joining the definition `symbol` with the security master `symbol` df_combined = pd.merge( df_definition, df_security_master, left_on="symbol", right_on="symbol", how="inner", ) # Display the combined records print(df_combined.head()) ``` ## Result ```plaintext ts_event rtype publisher_id instrument_id action side depth price size flags ts_in_delta sequence symbol ts_record listing_id listing_group_id security_id issuer_id listing_status listing_source listing_created_date listing_date delisting_date issuer_name security_type security_description primary_exchange exchange operating_mic nasdaq_symbol local_code isin us_code bbg_comp_id bbg_comp_ticker figi figi_ticker fisn lei sic cik gics naics cic cfi incorporation_country listing_country register_country trading_currency multi_currency segment_mic_name segment_mic structure lot_size par_value par_value_currency voting vote_per_sec shares_outstanding shares_outstanding_date ts_created 0 2024-04-01 08:00:05.048097430+00:00 0 2 38 T A 0 171.85 1 130 169598 280962 AAPL 2024-08-05 02:41:52+00:00 L-135825 LG-35825 S-33449 I-30017 L M 2001-05-05 1980-12-12 None Apple Inc EQS Ordinary Shares USNASD USNASD XNAS AAPL AAPL US0378331005 37833100 BBG000B9XRY4 AAPL US BBG000B9Y5X2 AAPL UW NaN HWUPKR0MPOU8FGXBT394 3571 320193 NaN 334111 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0.00001 USD V 1 15204137000 2024-07-19 2024-11-01 03:02:18.020386+00:00 ``` # Listings and delistings ## Overview This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve all point-in-time (PIT) security master records for Google Inc. We can use these to track Google's security listings across time. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. ## Google securities Google launched in 1998, quickly becoming the leading global search engine. It went public in 2004 through an IPO (prior to the security master dataset coverage). In 2015, Google restructured to form Alphabet Inc., a holding company designed to manage its diverse ventures more effectively, ensuring focused growth across various sectors. Alphabet Inc., the parent company of Google, issues three types of stock to accommodate different investor needs and maintain founder control: Class A, B, and C shares. Summary of Alphabet's share structure: - Class A Shares (`GOOGL`): Publicly listed, 1 vote per share - Class B Shares: Not publicly listed, 10 votes per share, held by founders and early investors - Class C Shares (`GOOG`): Publicly listed, no voting rights This structure ensures that while the company benefits from public capital through Class A and C shares, the founders retain overriding control through their Class B holdings. ## Security master PIT records We'll make a request from the security master API for Google securities publicly listed in the US starting 2015-01-01. Because there are so many records, we'll call `.head(40)` on the dataframe to view a selection. ```python import databento as db # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request security master records for Google df = ref_client.security_master.get_range( symbols=["GOOGL", "GOOG"], countries=["US"], start="2015", ) df.head(40) ``` ## Result ```plaintext ts_record listing_id listing_group_id security_id issuer_id listing_status listing_source listing_created_date listing_date delisting_date issuer_name security_type security_description primary_exchange exchange operating_mic symbol nasdaq_symbol local_code isin us_code bbg_comp_id bbg_comp_ticker figi figi_ticker fisn lei sic cik gics naics cic cfi incorporation_country listing_country register_country trading_currency multi_currency segment_mic_name segment_mic structure lot_size par_value par_value_currency voting vote_per_sec shares_outstanding shares_outstanding_date ts_created ts_effective 2015-10-05 00:00:00+00:00 2015-08-10 23:00:00+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN NaN 2015-10-05 2024-11-13 14:04:47.783780+00:00 2016-02-01 00:00:00+00:00 2016-02-12 13:28:12+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 292580627 2016-02-01 2024-11-01 06:45:30.486798+00:00 2016-02-01 00:00:00+00:00 2016-02-12 13:28:12+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 345539303 2016-02-01 2024-11-13 14:04:47.783780+00:00 2016-02-05 00:00:00+00:00 2016-02-15 13:33:55+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 292585668 2016-02-05 2024-11-13 14:04:47.783655+00:00 2016-04-11 00:00:00+00:00 2016-05-03 13:14:54+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 293658526 2016-04-11 2024-11-13 14:04:47.783655+00:00 2016-04-21 00:00:00+00:00 2016-05-04 07:22:00+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 293675151 2016-04-21 2024-11-01 06:45:30.486798+00:00 2016-10-27 00:00:00+00:00 2016-11-04 13:15:10+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 296087212 2016-10-27 2024-11-01 06:45:30.486798+00:00 2017-04-19 00:00:00+00:00 2017-05-01 07:39:14+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 297614546 2017-04-19 2024-11-13 14:04:47.783655+00:00 2018-01-31 00:00:00+00:00 2018-02-07 13:19:08+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 298492525 2018-01-31 2024-11-01 06:45:30.486798+00:00 2018-04-18 00:00:00+00:00 2018-04-25 13:13:11+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 298656198 2018-04-18 2024-11-01 06:45:30.486798+00:00 2018-07-18 00:00:00+00:00 2018-07-25 14:03:11+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349883498 2018-07-18 2024-11-01 06:45:30.486923+00:00 2018-10-19 00:00:00+00:00 2018-10-29 07:26:56+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349615045 2018-10-19 2024-11-13 14:04:47.783780+00:00 2018-10-19 00:00:00+00:00 2018-10-29 07:26:56+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349615045 2018-10-19 2024-11-13 14:04:47.783780+00:00 2019-01-31 00:00:00+00:00 2019-02-06 13:07:42+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 299360029 2019-01-31 2024-11-01 06:45:30.486798+00:00 2019-01-31 00:00:00+00:00 2019-02-06 13:07:42+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349291348 2019-01-31 2024-11-01 06:45:30.486923+00:00 2019-01-31 00:00:00+00:00 2019-02-06 13:07:42+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 349291348 2019-01-31 2024-11-01 06:45:30.486923+00:00 2019-07-19 00:00:00+00:00 2019-07-29 04:36:44+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 347344583 2019-07-19 2024-11-13 14:04:47.783780+00:00 2020-01-27 00:00:00+00:00 2020-02-06 13:20:30+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 299895185 2020-01-27 2024-11-13 14:04:47.783655+00:00 2020-01-27 00:00:00+00:00 2020-02-06 13:20:30+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 299895185 2020-01-27 2024-11-13 14:04:47.783655+00:00 2020-01-27 00:00:00+00:00 2020-02-06 13:20:30+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 340979832 2020-01-27 2024-11-13 14:04:47.783780+00:00 2020-04-07 00:00:00+00:00 2020-04-27 03:38:30+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300047170 2020-04-07 2024-11-13 14:04:47.783655+00:00 2020-07-23 00:00:00+00:00 2020-08-03 05:05:14+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300471156 2020-07-23 2024-11-01 06:45:30.486798+00:00 2020-07-23 00:00:00+00:00 2020-08-03 05:05:14+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 333631113 2020-07-23 2024-11-01 06:45:30.486923+00:00 2020-10-22 00:00:00+00:00 2020-11-02 03:37:23+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300643829 2020-10-22 2024-11-13 14:04:47.783655+00:00 2020-10-22 00:00:00+00:00 2020-11-02 03:37:23+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 329867212 2020-10-22 2024-11-13 14:04:47.783780+00:00 2021-04-06 00:00:00+00:00 2021-04-26 04:11:13+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 324511693 2021-04-06 2024-11-01 06:45:30.486923+00:00 2021-04-20 00:00:00+00:00 2021-04-30 13:06:34+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300746844 2021-04-20 2024-11-01 06:45:30.486798+00:00 2021-07-20 00:00:00+00:00 2021-07-29 13:17:56+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 320168491 2021-07-20 2024-11-13 14:04:47.783780+00:00 2021-10-19 00:00:00+00:00 2021-10-28 13:14:13+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 317737778 2021-10-19 2024-11-01 06:45:30.486923+00:00 2022-01-25 00:00:00+00:00 2022-02-03 13:09:07+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300754904 2022-01-25 2024-11-13 14:04:47.783655+00:00 2022-01-25 00:00:00+00:00 2022-02-03 13:09:07+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 315639479 2022-01-25 2024-11-13 14:04:47.783780+00:00 2022-04-05 00:00:00+00:00 2022-04-11 03:37:16+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 300758936 2022-04-05 2024-11-01 06:45:30.486798+00:00 2022-04-05 00:00:00+00:00 2022-04-25 03:51:25+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 315573679 2022-04-05 2024-11-01 06:45:30.486923+00:00 2022-04-05 00:00:00+00:00 2022-04-25 03:51:25+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 315573679 2022-04-05 2024-11-01 06:45:30.486923+00:00 2022-04-19 00:00:00+00:00 2022-04-29 03:51:45+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 313376417 2022-04-19 2024-11-13 14:04:47.783780+00:00 2022-07-27 00:00:00+00:00 2022-07-28 03:37:17+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 5996000000 2022-07-27 2024-11-13 14:04:47.783655+00:00 2022-07-27 00:00:00+00:00 2022-07-28 03:37:17+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 6163000000 2022-07-27 2024-11-13 14:04:47.783780+00:00 2022-10-18 00:00:00+00:00 2022-10-27 13:16:16+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 5973000000 2022-10-18 2024-11-13 14:04:47.783655+00:00 2023-01-26 00:00:00+00:00 2023-02-06 04:10:55+00:00 L-16476331 LG-6476331 S-4579561 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class A USNASD USNASD XNAS GOOGL GOOGL GOOGL US02079K3059 02079K305 BBG009S39JX6 GOOGL US BBG009S4MT03 GOOGL UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESVUFR US US US USD False Global Select XNGS NaN 100 0 USD V 1 5956000000 2023-01-26 2024-11-01 06:45:30.486798+00:00 2023-04-18 00:00:00+00:00 2023-04-27 07:20:35+00:00 L-16476332 LG-6476332 S-4579564 I-175414 L M 2015-08-11 2015-10-05 None Alphabet Inc EQS Ordinary Shares - Class C USNASD USNASD XNAS GOOG GOOG GOOG US02079K1079 02079K107 BBG009S3NB30 GOOG US BBG009S4MVF2 GOOG UW NaN 5493006MHB84DD0ZWV18 7370 1652044 NaN 51 US31 ESNUFR US US US USD False Global Select XNGS NaN 100 0 USD N NaN 5874000000 2023-04-18 2024-11-13 14:04:47.783780+00:00 ``` # Market capitalization change ## Overview This example demonstrates how to use the [Reference client](https://databento.com/docs/api-reference-data/client.md) to retrieve point-in-time (PIT) security master records for Apple Inc. alongside daily bar aggregates. By combining daily close prices with shares outstanding, we can analyze changes in Apple's total market capitalization over time. > **See also** > > [Security master](https://databento.com/docs/datasets/security-master.md) dataset guide for further details. ## Market capitalization Market capitalization, or "market cap," is the total value of a company's outstanding shares of stock, representing its overall market value. It is typically calculated by multiplying the current stock price by the total number of outstanding shares: \[ \text{Market Capitalization} = \text{Share Price} \times \text{Shares Outstanding} \] This straightforward calculation provides a snapshot based on the latest share price but does not account for complexities like share class variations, potential stock buybacks, or other corporate actions that might affect a company's actual valuation over time. ## Apple daily bars Using the [Historical client](https://databento.com/docs/api-historical/client.md) client, we can retrieve OHLCV-1d bars from 2018-05-01. These daily bars provide close prices that we can use to calculate and plot Apple's market capitalization. Note that these bars are aligned with UTC date boundaries, so the close price represents the price at UTC midnight, which may differ from the local market session close price. > **See also** > > [Aggregate bars (OHLCV)](https://databento.com/docs/schemas/ohlcv.md) schema guide for further details. ## Security master PIT records Next, we'll use the security master API to request point-in-time (PIT) records for Apple Inc. stocks (`AAPL`) publicly listed in the US since 2018-05-01. We apply the `countries` filter with `"US"` to ensure we capture only US-listed securities, in this case Apple Inc. listed on Nasdaq (`XNAS`). For securities listed on multiple exchanges, additional filtering on the returned DataFrame may be necessary to isolate the desired listing. ```python import databento as db import matplotlib.pyplot as plt import pandas as pd # Set parameters dataset = "XNAS.ITCH" symbol = "AAPL" start = "2018-05" end = "2025-04" # Create a historical client hist_client = db.Historical(key="$YOUR_API_KEY") # Request OHLCV-1d data to obtain close prices data = hist_client.timeseries.get_range( dataset="XNAS.ITCH", schema="ohlcv-1d", symbols=["AAPL"], start="2018-05", end="2024-11", ) df_bars = data.to_df() # Create a reference client ref_client = db.Reference(key="$YOUR_API_KEY") # Request security master records for Apple df_security_master = ref_client.security_master.get_range( symbols=["AAPL"], countries=["US"], start=start, ) df_security_master["shares_outstanding_date"] = pd.to_datetime( df_security_master["shares_outstanding_date"], utc=True, ) df_security_master.set_index("shares_outstanding_date") # Filter and resample the shares outstanding to daily frequency, forward filling missing values df_shares_outstanding = ( df_security_master[["shares_outstanding_date", "shares_outstanding"]] .set_index("shares_outstanding_date") .resample("D") .ffill() ) # Align both dataframes on the date index df_market_cap = df_bars[["close"]].join(df_shares_outstanding, how="left") # Calculate market capitalization by multiplying close price with shares outstanding df_market_cap["market_cap"] = df_market_cap["close"] * df_market_cap["shares_outstanding"] df_market_cap["market_cap"] = df_market_cap["market_cap"] / 1e9 # Convert to billions # Plot the results plt.plot(df_market_cap["market_cap"], label="market_cap") plt.legend() plt.xlabel("Date (UTC)") plt.ylabel("Market Capitalization (Billion USD)") plt.title(f"{symbol}") plt.show() ``` ![AAPL market cap](https://databento.com/docs/market-cap-change-0.svg.md)