Breaking changes to end timestamp behavior for historical API

To enhance your user experience, we're changing how our historical API handles input time ranges, specifically end timestamps provided as strings with less than one-second resolution. This will break compatibility and take effect on March 27, 2024 at 5 PM ET.

• "Less than one-second resolution" means a timestamp that's only specified up to year, month, day, hour, or minute resolution, e.g. yyyy-mm-ddTHH:MM.
• All mentions of "timestamps" in this announcement refer to strings in the ISO 8601 format. This change will not impact time ranges passed as a native timestamp type in our Python, C++, or Rust client libraries, e.g. DateTime objects, since they always serialize to at least one-second resolution or better.

When users request data with our historical API, they specify a time range with an inclusive start timestamp and an optional, exclusive end timestamp. However, our historical API forward-fills end timestamps with less than one-second resolution when passed as ISO 8601 strings.

This means that if the user provides an incomplete end timestamp (e.g., only the date, hour, or minute), it's rounded up to the next unit of time at the highest-specified resolution. For example, if only minutes are specified (2024-02-06T09:15), the data is returned up until the following minute (2024-02-06T09:16:00). This creates some scenarios where the input time range is effectively treated as end-inclusive.

Starting March 27, 2024, any unspecified intraday resolution in the end timestamp will default to 0 and any unspecified month or day resolution will default to 1. This ensures that all input time ranges will be treated as start-inclusive and end-exclusive.

The following table illustrates how the current and new behavior will differ:

Only the handling of inputs to historical API requests is changing—the data itself is not. As mentioned earlier in this announcement, users who input end timestamps with one-second resolution or better are unaffected.

The handling of the start timestamp remains unchanged. Pointwise input time ranges are still supported when an end timestamp isn't specified. For example, you can request data for February 6, 2024, with start=2024-02-06 and no end timestamp.

Our current forward-fill behavior was intended to accommodate users who mistakenly assume the time range includes the end timestamp, but this approach has led to three issues.

At Databento, we've adopted a convention of half-open time ranges with an exclusive end across our APIs. This is good practice for APIs that routinely involve high-resolution timestamps, whereas inclusive time ranges are more suitable for APIs for low-resolution data, such as those for daily-frequency data. Forward-filling certain end timestamps can create an inconsistent experience when requesting data through our historical API.

The current forward-fill behavior can result in users requesting, and consequently being billed for, more data than expected. The following example demonstrates this:

data = client.timeseries.get_range(
    dataset="GLBX.MDP3",
    symbols=["ESM2"],
    schema="ohlcv-1d",
    start="2022",
    end="2023",
)

When forward-filled, the above request will return two years of data from 2022-01-01T00:00:00 until 2024-01-01T00:00:00. After March 27, 2024, this example will only return one year of data because the end timestamp will be treated as end="2023-01-01T00:00:00".

This forward-filling behavior is subtle and limited to specific cases since it only applies to end timestamps with less than one-second resolution. Native timestamp types used with our Python, C++, or Rust client libraries, e.g., DateTime objects, always serialize to ISO 8601 timestamps or nanosecond UNIX timestamps with at least one-second resolution or better and are not forward-filled. Hence, it's easy for users to miss these subtleties and be unaware of this behavior.

Unexpected forward-fill behavior can also have implications for OHLCV data requests, as users must consider that OHLCV events are timestamped at the opening of each interval.

Starting February 28, 2024, users will receive a warning on historical API requests with forward-filled end timestamps. Again, only users who specify end timestamps as strings with less than one-second resolution will be affected.

In most cases, the new behavior will align with user expectations, but some code adjustments may be necessary. Existing pipelines and automations can avoid changes by specifying end timestamps with at least one-second resolution or by using native timestamp types or objects instead of strings in the client libraries. This will prevent queries from exhibiting different behavior after March 27, 2024.

While we recognize that this change may inconvenience some users, we believe it will improve our product and result in a more consistent API in the long run. If you have any questions or concerns, please reach out to us at support@databento.com.