Webhooks, retries and the duplicates they cause

<h2>A webhook is a promise to try</h2><p>When a payments provider says it will notify you of a successful charge, it is promising to try. Networks drop, receivers restart mid-request, a load balancer returns 502 for eleven seconds during a deploy. The sender cannot tell the difference between a request that never arrived and a response that never came back, so it does the only safe thing and sends again.</p><p>That is at least once delivery, and it is the strongest guarantee any practical webhook system offers. Exactly once sounds better and is not on the menu. What you can build is a receiver that behaves as if delivery were exactly once, which is a different and entirely achievable thing.</p><h2>Our retry schedule</h2><p>When a destination rejects a write or a receiver rejects our outbound webhook, we retry on a fixed ladder. It is published because guessing it is worse for everyone.</p><table><thead><tr><th>Attempt</th><th>Delay after previous</th><th>Elapsed since event</th></tr></thead><tbody><tr><td>1</td><td>immediate</td><td>0s</td></tr><tr><td>2</td><td>30 seconds</td><td>30s</td></tr><tr><td>3</td><td>2 minutes</td><td>2m 30s</td></tr><tr><td>4</td><td>10 minutes</td><td>12m 30s</td></tr><tr><td>5</td><td>1 hour</td><td>1h 12m</td></tr><tr><td>6</td><td>6 hours</td><td>7h 12m</td></tr></tbody></table><p>After the sixth attempt the run is parked, not discarded. It sits in the history with its payload intact and a replay button. A 429 with a Retry-After header jumps the ladder and waits exactly as long as it was asked to, because arguing with a rate limiter wastes both budgets.</p><h3>What we do not retry</h3><p>A 4xx that means you sent something wrong gets one attempt and stops. Retrying a 400 six times produces six identical failures and a slower error message. The exceptions are 408, 425 and 429, all of which mean try again rather than try differently.</p><h2>Where the duplicate comes from</h2><p>Here is the shape of an inbound event we receive:</p><pre><code>POST /hooks/inbound { "event_id": "evt_4c19aa72", "type": "payment.succeeded", "created": 1774872412, "data": { "amount": 4900, "currency": "usd", "email": "rosa@kestrelfreight.test" } } </code></pre><p>We accept it, start writing a contact into a CRM, and the CRM takes nine seconds. The sender's timeout is five. It gives up, marks the attempt failed, and sends the identical body again thirty seconds later. Our first write eventually succeeds. Now we are holding two requests for one payment, and without a guard we create two contacts and post two messages into a chat channel.</p><h3>Idempotency in four lines</h3><p>The whole defence fits in a paragraph of pseudocode:</p><pre><code>key = event_id or sha256(type + created + canonical(payload)) if seen(key): return 200 result = process(payload) mark_seen(key, ttl = 14 days) return 200 </code></pre><p>Three details decide whether it works. Return 200 on the duplicate, since returning an error teaches the sender to retry a message you have already handled. Store the key before you finish processing rather than after, and store it with the outcome so a crash halfway through does not turn into a permanent skip. Pick a time to live longer than the retry ladder, because a key that expires in an hour is no defence against a six hour retry.</p><p>When the sender gives you no event id, hash the fields that make the event unique. Never hash the raw body, since a whitespace change or a reordered JSON object produces a different hash for the same event.</p><h2>Ordering is not guaranteed either</h2><p>Two events emitted a second apart can arrive in either order, and after a retry they routinely do. A subscription upgrade followed by a cancellation can land as a cancellation followed by an upgrade, which leaves you with an active subscription for a customer who left.</p><p>The fix is to stop treating an event as an instruction and start treating it as a timestamped observation. Every write compares the event timestamp against what is already on the record. If the record was last touched by a newer event, the older one is recorded in the history and dropped. This is dull, it costs one comparison, and it removes a whole category of three in the morning confusion.</p><h2>When the receiver is down</h2><p>If your endpoint has been failing for an hour, sending the same volume the moment it recovers is how you knock it over a second time. We drain a parked queue at a quarter of normal rate for the first two minutes, then ramp. Every message still carries its original event id, so a receiver with a working idempotency key can be flooded without harm.</p><blockquote><p>We spent a week hunting a bug that turned out to be our own retry, arriving after a deploy, hitting an endpoint that had no idempotency key. The bug report said duplicate customers. The cause was thirty seconds of downtime.</p><p>Internal postmortem, Syncline flow runner</p></blockquote><h2>The rules we hold ourselves to</h2><ul><li>Every outbound webhook carries a stable event id and a delivery attempt number in the headers.</li><li>Retries are published and never randomised beyond a small jitter, so receivers can plan.</li><li>A parked run keeps its full payload and can be replayed by hand for fourteen days.</li><li>We fail loudly. A run that stopped is visible in the history with a reason, and never silently dropped.</li></ul><p>None of this makes delivery exactly once. It makes duplicate delivery harmless, which is the part that matters when you are the one reading the run history at the end of a bad afternoon.</p>