Upcoming changes to the historical batch API

We're changing the behavior of the historical API's batch.list_jobs endpoint:

1. batch.list_jobs will return a summary of each job containing only id, state, and ts_received.
2. A new batch.get_job_details endpoint will provide the full details for a single job.

Since we're moving most of the fields returned by list_jobs to get_job_details, this is a breaking change if your application reads any of the affected fields. We're rolling out these changes over a transition period to give you time to update your integration as needed.

Currently, batch.list_jobs returns the full set of fields for every job that matches the provided parameters. For users with many batch jobs, this results in large payloads and slower responses. We're separating the complete job details into a dedicated endpoint so that list_jobs returns only the information needed to identify jobs and check their current state.

The new response will include only three fields for each job:

• id
• state
• ts_received

During the transition period, list_jobs will accept a temporary short parameter that lets you switch between the new and legacy response formats. Set short=True to preview the new response:

client = db.Historical("YOUR_API_KEY")

jobs = client.batch.list_jobs(
    states=["queued", "processing", "done"],
    since="2022-06-01",
    short=True,
)

print(jobs)

This will return the condensed response:

[
  {
    "id": "GLBX-20220901-5DEFXVTMSM",
    "state": "queued",
    "ts_received": "2022-11-30T20:33:08.006406000Z",
  },
  ...
]

The new batch.get_job_details endpoint returns the full details for a single job. Pass the job_id from list_jobs to get the fields that aren't included in the condensed response.

client = db.Historical("YOUR_API_KEY")

job_details = client.batch.get_job_details(
    job_id="GLBX-20220901-5DEFXVTMSM",
)

print(job_details)

This will return all fields for the specified job:

{
  "id": "GLBX-20220901-5DEFXVTMSM",
  "user_id": "46PCMCVF",
  "api_key": "prod-001",
  "cost_usd": 6.72198,
  "dataset": "GLBX.MDP3",
  "symbols": "ESH2",
  "stype_in": "raw_symbol",
  "stype_out": "instrument_id",
  "schema": "trades",
  "start": "2022-01-03T00:00:00.000000000Z",
  "end": "2022-01-20T00:00:00.000000000Z",
  "limit": null,
  "encoding": "dbn",
  "compression": "zstd",
  "pretty_px": false,
  "pretty_ts": false,
  "map_symbols": false,
  "split_symbols": false,
  "split_duration": "day",
  "split_size": null,
  "packaging": null,
  "delivery": "download",
  "record_count": 6063235,
  "billed_size": 291035280,
  "actual_size": null,
  "package_size": null,
  "state": "queued",
  "ts_received": "2022-11-30T20:33:08.006406000Z",
  "ts_queued": null,
  "ts_process_start": null,
  "ts_process_done": null,
  "ts_expiration": null,
  "progress": 0
}

If your application reads any field other than id, state, or ts_received from list_jobs (for example, a job's cost, progress, or other timestamps), you must update it to get those details from get_job_details before the transition period ends.

During the transition period, you can use the short parameter to switch between the condensed and legacy list_jobs response formats.

• get_job_details is available.
• list_jobs accepts the short parameter, which defaults to False. Existing integrations continue to receive the legacy response unless short=True.

• The default value of the short parameter changes to True, and list_jobs returns the condensed response unless short=False.

• The short parameter and legacy response are removed.
• list_jobs returns the condensed response for all requests.