Timestamping

Precision, accuracy, and resolution

In our documentation, we quantify these terms as follows:

• Precision. The maximum deviation between timestamps delivered in the data and any reference time.
• Accuracy. The maximum deviation between timestamps delivered in the data and Coordinated Universal Time (UTC).
• Resolution. An arbitrary unit of time. The minimum increment of timestamps delivered in the data.
• Jitter. Closely related to precision. The maximum deviation of a clock's periodicity from its intended periodicity.

Info

Unfortunately, common speech uses the opposite sign convention. A time source with high precision or accuracy is said to have low maximum deviation. Likewise, high resolution refers to a small minimum timestamp increment. A clock with high jitter has low precision.

The following diagram helps to visualize these terms:

Nanosecond resolution timestamps

You may have seen other data sources that claim to provide nanosecond-resolution timestamps. But this is a meaningless claim on its own, as any device is capable of nanosecond-resolution timestamps! You can guess the current time and write a number arbitrarily down by hand to the nanosecond resolution - that doesn't mean that your body is capable of nanosecond-precision timestamping.

Likewise, you will find that most sources of financial data that carry nanosecond-resolution timestamps will use software-based timestamping that relies on CPU registers. There are a host of issues with this:

• The precision of CPU registers varies substantially with thermal conditions.
• The timestamping application is subject to sources of high jitter, such as interrupts, kernel scheduler events, context switching, and interference from other applications etc. This is especially the case with data providers that use cloud-based virtual machines to capture the data.

To safeguard against these issues, we offload our timestamping to hardware, for instance FPGA, which achieves more deterministic results. Depending on the techniques used, software timestamps may have a precision of anywhere from 100 nanoseconds to over 10 milliseconds. In contrast, our hardware timestamps achieve a precision from 4 to 10 nanoseconds (depending on the market).

Time synchronization with Precision Time Protocol

You will also find that most sources of financial data rely on Network Time Protocol (NTP) to synchronize their clocks. In practice, software timestamps relying on NTP synchronization against public NTP servers can deviate from UTC up to the order of 100 milliseconds. Typical implementations of Precision Time Protocol (PTP) will achieve accuracy from sub-microsecond to somewhere under 50 microseconds.

At each data center, Databento uses a pair of GPS grandmaster clocks with PTP class 6 and clock accuracy under 25 ns against UTC as the time source. Our data servers achieve sub-microsecond synchronization to these GPS clocks.

Accuracy, precision, and resolution of Databento timestamps

Databento's data is intended to be suitable for high fidelity simulation, including backtesting of passive and arbitrage strategies, order routing, transaction cost analysis, and execution research. Our receive timestamps (ts_recv) have the following properties:

• Precision. We use hardware timestamping to achieve a precision from 4 to 10 nanoseconds.
• Accuracy. We use PTP time synchronization against GPS receiver-based time sources that are backed up by rubidium atomic oscillators to ensure holdover accuracy. Our timestamps achieve an accuracy from under 100 nanoseconds to under 1 microsecond, depending on the capture server's location.
• Resolution. All of our timestamps are in nanosecond resolution.

Any exception to the above for a given dataset is stated in the detail page of the dataset. Such exceptions generally arise when we have to backfill data for other sources or use different hardware for a particular market.

There is no guarantee that timestamps embedded by original publisher or market (ts_event, ts_in_delta) will have high accuracy, precision, or resolution. Some markets still use millisecond resolution timestamps, inherited from legacy FIX protocol. Many markets will use NTP time synchronization and software timestamps. Often, our timestamps will have even higher precision and accuracy than those provided by the market itself.