Skip to content
Syncline
engineering, product

How we think about reliability

Uptime is the least interesting number on our status page. What matters is how a flow behaves in the ten minutes when something downstream is broken.

Kit Ashford, Reliability Engineer2026-06-188 min read
<h2>Uptime is the least useful number we publish</h2><p>Our status page shows availability per component, and it is the number people ask about first. It is also the number that tells you the least. An integration platform can be up in every meaningful sense while a flow you depend on has not moved a record since Tuesday, because the app on the far end changed a permission and nobody read the email.</p><p>What we actually care about is a harder question. When something breaks, does the customer find out from us or from their own customer? Everything below follows from trying to make the answer the first one.</p><h2>Four components, four different promises</h2><table><thead><tr><th>Component</th><th>Target</th><th>Measured last quarter</th><th>What happens when it slips</th></tr></thead><tbody><tr><td>API</td><td>99.9 percent</td><td>99.96 percent</td><td>Reads fail fast, writes queue for ten minutes</td></tr><tr><td>Flow runner</td><td>99.9 percent</td><td>99.92 percent</td><td>Runs are delayed, never dropped, and drain in order</td></tr><tr><td>Inbound webhooks</td><td>99.95 percent</td><td>99.98 percent</td><td>Senders retry against our published ladder</td></tr><tr><td>App connections</td><td>Best effort</td><td>Varies by app</td><td>Flow parks with a named reason and an email to the owner</td></tr></tbody></table><p>The last row is the honest one. We do not control whether a third party API is answering, and promising a number we cannot hold would be a promise about somebody else's infrastructure. What we can promise is behaviour: a connection failure parks the flow with a reason a person can read, and it tells the flow owner rather than waiting to be discovered.</p><h2>The three failures we actually see</h2><ol><li><strong>Expired or revoked credentials.</strong> Comfortably the most common, and almost never an outage. Somebody left the company, an admin tightened a scope, a token hit its ninety day expiry. The flow is fine and the door is locked.</li><li><strong>Rate limits during a burst.</strong> An import of four thousand rows meets a hundred requests per ten seconds, and every request after the budget comes back 429. Correct behaviour from both sides, and it looks exactly like an outage from a dashboard.</li><li><strong>A shape change.</strong> A field renamed, a select option deleted, a required property added. The API is up, the credentials are valid, and the write is rejected for a reason that only makes sense if you know what changed yesterday.</li></ol><p>Only one of those three is an outage, and it is the rarest. Building for the other two is why our error messages name the field and the record rather than saying request failed.</p><h2>Degrade loudly, never quietly</h2><p>The worst possible failure mode is a flow that looks healthy while doing nothing. A green tick over an empty result is more expensive than a red cross, because nobody investigates a green tick until the monthly numbers look wrong.</p><p>So a run that read zero records says it read zero records. A run that skipped four records lists which four and why. A flow that has produced nothing for longer than its own normal gap raises a quiet notice, since a daily flow silent for three days is a fact worth surfacing even when nothing technically failed.</p><blockquote><p>We would rather send you one email you did not need than let a flow sit still for a fortnight. The second one costs a customer.</p><p>Internal reliability note, written after the March backlog</p></blockquote><h2>What changed after the March backlog</h2><p>In March a queue backed up for just over two hours. The cause was ordinary: a database migration on our side ran longer than its window, workers waited politely, and the queue grew. Nothing was lost, everything drained, and the incident write up is on the status page.</p><p>Three things changed afterwards, and only one of them was technical.</p><ul><li><strong>Queue depth became a customer facing signal.</strong> If your runs are delayed, the flow page says delayed with an estimate, instead of showing the last successful run and implying all is well.</li><li><strong>We drain at a quarter rate for the first two minutes.</strong> Dumping two hours of held work at full speed into a destination that allows a hundred requests per ten seconds turns our recovery into their outage.</li><li><strong>The status page updates before the postmortem.</strong> We now post the holding note within fifteen minutes and the explanation when we have one. Waiting until we understood the cause meant an hour of silence, and silence reads as nobody has noticed.</li></ul><h2>What we will not do</h2><p>We will not promise exactly once delivery. Every practical webhook system is at least once, so we publish our retry ladder and make duplicates harmless instead of pretending they do not exist.</p><p>We will not hide a third party outage behind a green tick on our own status page. If an app connection is failing for everyone, that row goes amber even though our own services are healthy, because the question you are answering at that moment is whether your flow works.</p><p>And we will not delete a failed run to make a dashboard look better. A parked run keeps its payload for fourteen days and can be replayed. The history is the only reason anybody trusts an automation they cannot watch, and trimming it to look tidy would be trading the product for a screenshot.</p>

Key takeaways

<ul><li>Most failures are expired credentials, rate limits and shape changes, and only one of those is an outage.</li><li>A green tick over an empty result costs more than a visible error, so a flow that produces nothing should say so.</li><li>Publish the retry ladder, drain backlogs slowly, and keep failed runs replayable rather than deleting them.</li></ul>

Kit Ashford

Reliability Engineer

Kit joined Syncline from a payments company and is responsible for the alerts that wake the on call engineer.

Try Syncline

Build the flow this article describes.

Start free

More from the blog

One flow, four clients, no copy and paste

Agencies rebuild the same automation for every client, then maintain twelve copies of it. A look at how templates and workspace variables were designed to fix that.