Delivery guarantees

At-least-once delivery, best-effort ordering, the delivery log and manual redelivery — and what each one means for a number somebody will argue about.

Three properties of delivery decide how you have to write a reporting handler. All three are the platform contract, documented in Delivery; this page is what they cost you if you ignore them.

At-least-once

The platform keeps trying until your endpoint accepts the event, with exponential backoff between attempts. A transient outage on your side does not lose an event — and the same event can reach you more than once.

For reporting that is not a rounding error, it is a wrong total. An INSERT per delivery double-counts every retried event, and the double count is invisible: nothing in your store says which row was a duplicate.

Two safe shapes:

  • Upsert keyed on the event id. The second delivery of the same event overwrites the first row instead of adding one.
  • A processed-ids table. Record the id before you act, and drop anything already there. Slightly more work, and it survives a handler that writes to several places.

What does not work is deduplicating on the entity id, or on a timestamp, or on "it looks like one we already had".

Best-effort ordering

Order is not guaranteed. Retries and backoff mean a redelivered event can arrive after a later one — order.paid can land before order.created.

Three ways to be immune:

  1. Use time, not arrival. Sort and window on the envelope's time, and your series is right whatever order the deliveries came in.
  2. Make handling commutative. If the end state is the same regardless of order, ordering stops being a correctness problem. Upserting a row keyed on the entity, with a last-write-wins on time, gets you there.
  3. Re-read the entity when the order matters. If deriving state from a sequence of events is unavoidable, let the event be the trigger and the API be the source of truth — see Incremental sync.

Beware the handler that assumes a creation event exists before an update event. It works in testing, where events arrive one at a time in order, and fails on the first busy afternoon.

The delivery log, and replay

Every delivery attempt to an endpoint is recorded: which event, when, the response, and whether it succeeded. A past event can be manually redelivered from there.

For a reporting pipeline that is the recovery tool, and it is worth knowing before you need it:

  • Your endpoint was down. Redeliver the window rather than re-backfilling from the API.
  • You shipped a bug in the handler. Fix it, then redeliver the affected events.
  • You are standing up a new store. Replay history into it — as far back as the log reaches.

The log is not infinite and carries no retention promise, so it is a recovery mechanism, not an archive. Anything you will need in a year, store yourself — which is the argument for recording the raw envelope as it arrives.

Watch for auto-disable too: an endpoint that keeps failing is disabled so the platform stops sending to a dead URL, and it has to be re-enabled from Cockpit. For a metrics pipeline a disabled endpoint reads exactly like a quiet week.

The reconciliation habit

Guarantees describe the transport, not your correctness. The habit that makes reporting trustworthy:

Daily
1. Handle events as they arrive → your store
2. Once a day, re-read the affected entities from the API for the window
3. Compare, and fix your store where they disagree
4. Alert if the disagreement is not zero

Step 4 is the one people skip, and it is the one that tells you the pipeline broke before a customer does.

Where to go next

Was this page helpful?