Build your own reporting store

The path most partners end up on — deciding between pull and push, backfilling history from the API, keeping in sync, and modelling commerce data without getting the grain wrong.

Build your own reporting store

Pull versus push

What each feed is good for, and why most pipelines use both.

Backfill

Loading history — which endpoints carry the reporting fields, paging, and the headers every call needs.

Incremental sync

Watermarks, reconciling a webhook gap against an API re-read, and out-of-order arrivals.

Modelling

The grain traps — order versus line item, gross versus net versus refunds, currency and tax.

Because there is no analytics API, the query side has to live somewhere you control. That is not a workaround — for anything a customer will argue about, an authoritative store you own is what you want anyway.

This group is how to build one: which feed to use, how to load history, how to stay current, and how to model commerce data so the numbers survive contact with an accountant.

The pages

  • Pull versus push — reading the API on a schedule against receiving events, what each is good for, and the combination that holds up.
  • Backfill — the initial load: finding the endpoints that carry the fields you need, paging through them, and the tenancy and auth headers every call requires.
  • Incremental sync — watermarks, catching up after a gap, and handling arrivals that are late or out of order.
  • Modelling — the four traps: grain, refunds, currency, and tax.

The shape that works

The pipeline
1. Backfill        the API, paged, into a raw landing area
2. Stream          events into the same landing area, verified and deduped
3. Transform       raw → a model you report on
4. Reconcile       re-read the API for a window, compare, alert on drift

Four properties are worth designing in from the start:

  • Keep the raw records. Store the payload as received, then transform. A modelling mistake becomes a re-transform rather than a re-backfill.
  • Key everything on ids from the platform, not on your own sequence numbers, so a re-read is an upsert.
  • Timestamp on event time, not arrival time.
  • Reconcile on a schedule and alert on the difference. The pipeline that nobody checks is the pipeline that has been wrong since March.

Where to go next