Updates to handling of display factor and unit of measure fields in definition schema

We're updating how our client libraries and APIs handle the display_factor and unit_of_measure_qty fields in our definition schema to match their expected behavior and enhance usability. This change will be implemented on May 21, 2024 at 6 PM ET.

display_factor and unit_of_measure_qty are two useful fields in our definition schema. They can be used to calculate the original display price and notional value of a futures or options contract. Since these fields affect prices and require the same precision, they're encoded in the same memory-efficient, fixed-point integer format as prices.

Prior to this change, it was intended that these fields should be scaled depending on the pretty_px and price_type parameters in our APIs and client libraries. However, these two fields weren't being scaled correctly, making the intended behavior even more ambiguous.

Most of our services use or store data under the hood in Databento Binary Encoding. In DBN, display_factor and unit_of_measure_qty are represented as fixed-point integers (int64_t), where every unit implicitly corresponds to 1e-9. For example, a display price of $1.25 is encoded as an integer value of 1250000000.

As a convenience, we currently provide several ways to scale these implicitly-scaled, fixed-point integer values into an explicitly-scaled decimal representation. These behave correctly for all fields except display_factor and unit_of_measure_qty in the definition schema.

We've identified four scenarios where the display_factor and unit_of_measure_qty fields in the definition schema should be represented as explicitly-scaled decimals but are incorrectly returned as implicitly-scaled, fixed-point integers:

• When you specify pretty_px=True or any price_type other than 'fixed' in our client libraries. This deserializes the fixed-point integers into explicitly-scaled, decimal object representation.
• When you transcode DBN to CSV or JSON through our client libraries and specify pretty_px=True. This converts the fixed-point integers into explicitly-scaled decimal strings.
• When you fetch data in CSV or JSON directly from our HTTP API and specify pretty_px=True. This indicates that you want the conversion from fixed-point integers to explicitly-scaled decimal strings done for you on server side.
• When "Decimal prices" is selected as the price format for batch download requests through the portal.

After this upcoming fix, the display_factor and unit_of_measure_qty fields will now be correctly represented in decimal format with explicit scaling (e.g. "1.25" instead of 1250000000).

To better illustrate the change, below is an example calling DBNStore.to_df(price_type='decimal') in our Python client.

                                 display_factor     unit_of_measure_qty
ts_recv
2024-04-11 00:00:00+00:00              10000000           1000000000000

                                 display_factor     unit_of_measure_qty
ts_recv
2024-04-11 00:00:00+00:00                  0.01                    1000

Our raw data, which is in DBN encoding, is not affected. This means DBN files and the raw DBN records from our APIs will be unchanged after this fix, so existing DBN files will remain forward-compatible with newer versions of our client libraries. These affected fields only exist in the definition schema, so all other schemas are unaffected.

If you work with encodings other than DBN and use the display_factor or unit_of_measure_qty fields in fixed-point integer format, e.g. by specifying pretty_px=False or price_type='fixed', the behavior will also remain unchanged.

Since display_factor and unit_of_measure_qty are less commonly used than other fields in the definition schema, we anticipate that this fix will not affect most users.