Updates to CSV and JSON encoding fields (July 2023)

We made changes to the order and structure of CSV and JSON encoding fields for all schemas. This took effect on July 23, 2023 at 5 PM EST. The serialized output from both historical and live APIs will now consistently match the result from our dbn CLI tool.

• For CSV encoding, the field order will be modified for most schemas to prioritize the serialization of more important fields first.
• Regarding JSON encoding, the following adjustments will be made:
  • Header fields will be serialized within the nested hd object to ensure consistency with how levels is serialized.
  • All 64-bit integers will be encoded as strings to prevent loss of precision during conversion.
  • The depth field will be added to MBP schemas that were previously excluded.
  • JSON will be encoded in a more compact format, omitting all optional whitespace.
• Both JSON and CSV encodings will now always include an rtype field, facilitating easier differentiation between record types and schemas. This will be particularly important for Databento Live, where multiple record types can be sent in a single stream.

To ease the transition, we will regenerate any CSV and JSON jobs with the new serialization at your request.

In our ongoing mission to enhance your user experience, we're introducing changes to our APIs that will break compatibility. These will take effect on August 10, 2023, at 5 PM EST. We strongly recommend that you update to the latest version of our APIs or client libraries to support these changes. You can learn more about the updates here.

We understand that making these changes may require you to modify your code. However, we believe that these updates will improve your experience with our data, APIs, and services. If you have any questions or need assistance, please contact support.