Backfill

The initial load — finding which endpoints carry the reporting fields you need, paging through a list endpoint, the tenancy and auth headers every call requires, and staying inside the rate limit.

A backfill is one job you run once per store and then never quite trust again unless you built it to be re-runnable. Build it to be re-runnable.

Which endpoints carry the reporting fields

Every commerce entity you would want to report on — products, orders, carts, payments, customers — is readable through the public API, under one gateway with every path versioned beneath /v1.

Do not take a field list from a doc, including this one. The authoritative parameters, fields and response schema for every endpoint your tenant exposes are in the API Explorer. Which capabilities exist is per tenant: core commerce exposes a standard set, and installed Apps add their own.

The practical way to decide what to backfill:

  1. Write down the figures you have to produce.
  2. For each one, find the entity that owns the number, in the API Explorer.
  3. Check the list endpoint's schema for the fields you need — especially the timestamps and the ids you will key on.
  4. Note anything you need that is only on the detail endpoint, not the list. That is the difference between one call per page and one call per record, and it usually decides the shape of your backfill.

The headers every call needs

HeaderRequiredPurpose
X-Revenexx-TenantYesThe tenant slug the request is scoped to.
X-Revenexx-Api-KeyOne credentialA scoped key (rvxk_…) for server-side and machine-to-machine calls.
Authorization: Bearer <jwt>One credentialA user token, for calls on behalf of a signed-in user.

Send either the API key or the Bearer token, never both. The tenant header is always required — a backfill is inherently machine-to-machine, so use a scoped API key. See API usage and Auth.

One tenant per run. The tenant header is what scopes the read, so a multi-tenant backfill is N runs with N credentials, not one run with a filter. Record the tenant on every row you store; a store that lost track of which tenant a row came from cannot be fixed afterwards.

Log the X-Request-ID from each response. Quoting it is what makes a support conversation about a strange page traceable.

Paging through everything

List endpoints page. The common conventions:

ParameterPurpose
limitHow many items to return.
offsetHow many items to skip.
orderSort field and direction, as field.direction.
One page
curl "https://api.revenexx.com/v1/products?limit=100&offset=0&order=created_at.desc" \
  -H "X-Revenexx-Tenant: <TENANT_SLUG>" \
  -H "X-Revenexx-Api-Key: rvxk_..."

A page carries the items plus the paging echo, so total tells you what you are in for:

Page
{
  "items": [],
  "total": 248,
  "limit": 100,
  "offset": 0
}

Increase offset by limit until a page returns fewer items than you asked for. The exact parameter names, the default and maximum limit, and the sortable fields are defined per endpoint — check the API Explorer rather than assuming. See Pagination & filtering.

Offset paging over a moving dataset skips and repeats rows. New records arriving while you page shift everything along, so a row can slide from page 4 into page 3 after you have read page 3. Two defences, and use both: sort by a stable key with order rather than relying on the default, and upsert on the platform's id so a repeated row is harmless and a skipped one is caught by your reconciliation pass.

Staying inside the rate limit

A backfill is the one job most likely to hit the per-tenant budget. Design for it:

  • Read the rate limits page and handle 429 by backing off rather than retrying immediately.
  • Use the largest limit the endpoint allows. Half as many calls for the same rows.
  • Run it off-peak, and single-threaded before you try concurrency. A parallel backfill that trips the limit is slower than a serial one that does not.
  • Checkpoint after every page, so a run that stops resumes rather than restarts.

Make it re-runnable

Four properties, and they cost almost nothing at the start:

  1. Upsert on the platform's id. Never insert blindly.
  2. Store the raw record, then transform. A modelling change is then a re-transform, not a re-backfill.
  3. Record a watermark — the highest modification timestamp you have seen — so the incremental sync can pick up where the backfill stopped. See Incremental sync.
  4. Record the run: when it started, what it covered, how many rows. The first question about any odd figure is "when did we last load this".

Where to go next

Was this page helpful?