Pull versus push
Two feeds, and the choice is not either/or. Pull is authoritative and complete; push is fast and cheap. A pipeline that uses only one of them has a known failure mode.
The comparison
| Pull — read the API | Push — receive events | |
|---|---|---|
| Latency | As often as you schedule it | Seconds |
| History | Everything the API can list | Only from the moment you subscribed |
| Completeness | The entity's current state, in full | A payload about one change |
| Effort to start | An API key and a loop | An endpoint somebody must register in Cockpit, plus signature verification |
| Failure mode | You read stale data between runs | You miss events while your endpoint is down, or double-count retries |
| Cost per change | You re-read whether or not anything changed | One delivery per change |
| Who is authoritative | The platform, at read time | Nobody — the payload was true when it was emitted |
When to pull
- History. A first load, or any window from before you were listening. There is no event backlog to replay from before an endpoint existed.
- Truth. A figure somebody will dispute. A re-read from the API is the closest thing you have to an answer.
- Joins. Anything that needs fields the event payload does not carry.
- Reconciliation. Comparing what your store believes against what the platform says — the single most valuable scheduled job in the pipeline.
The mechanics are pagination, filtering and rate limits, identical across endpoints: see API usage and Backfill.
When to push
- Freshness. A counter, a live tile, an alert.
- Reacting. Something that must happen because of the change, not in a later report.
- Cheap change detection. Knowing which twenty entities changed, so your sync re-reads twenty instead of twenty thousand.
The mechanics are registration, signature verification and at-least-once delivery: see Stream commerce activity.
The combination that holds up
Use events as a change feed, and the API as the source of truth:
event arrives → verify, dedupe on id
→ record the raw envelope
→ mark the entity as dirty
scheduled job → re-read every dirty entity from the API
→ upsert into your store
daily → re-read a full window, compare, alert on drift
You get event latency for the things that need it, API truth for everything that lands in a report, and one loop that is correct whether an event was missed, duplicated or delivered out of order.
If you only get one
Sometimes you do — nobody has Cockpit access to register an endpoint, or your infrastructure cannot expose an HTTPS receiver.
Pull only is a perfectly good pipeline. Schedule it as often as the rate limits and your data volume allow, filter by a modification timestamp so each run reads only what changed, and accept the latency. Every figure will be right; some will be an hour old.
Push only is the one to avoid for reporting. You have no history, no way to reconcile, and no recovery if your receiver is disabled for a day. If it is genuinely all you have, at minimum store every raw envelope so a future backfill has something to rebuild from.
Where to go next
- Backfill — the initial pull.
- Incremental sync — keeping current with either feed.
- Rate limits — the budget a pull schedule has to live inside.