Idempotency, Retries and Backoff

The design-for-failure principle says every network call needs a retry plan. This note is the mechanics: how to make an operation safe to repeat, how to retry without amplifying a struggling dependency into an outage, and the patterns that stop a failure spreading.

It all starts from one unavoidable fact: a failed request tells you nothing about whether the work happened.

Client ──── POST /payments ────▶ Server
                                   │ charges the card ✓
       ◀──── timeout / TCP reset ───┘ response lost

Client's view: "it failed."   Reality: the payment succeeded.

Retry blindly and you charge twice. Don’t retry and you may drop a payment that never happened. The only way out is making the operation safe to repeat.

Idempotency

An operation is idempotent if performing it many times has the same effect as performing it once.

Not “returns the same response” — the same effect on state. DELETE /orders/42 is idempotent: after the first call the order is gone, and further calls leave it gone (a 404 response is fine; the state is unchanged).

SET balance = 100     idempotent     — repeating lands on the same state
balance = balance - 10  NOT           — each repeat subtracts again

Some operations are naturally idempotent — absolute assignment, deletes, “ensure this exists.” Others aren’t, and must be made so.

HTTP verbs

Worth memorizing, because the semantics are a contract callers and proxies rely on:

Verb	Idempotent?	Safe (no state change)?
GET, HEAD	Yes	Yes
PUT	Yes — full replacement	No
DELETE	Yes	No
POST	No	No
PATCH	Not necessarily	No

POST is the problem child: it means “create a new thing,” so replaying it creates another. That’s precisely where idempotency keys come in.

Idempotency keys

The standard fix for a non-idempotent operation. The client generates a unique key per logical operation and sends it with every attempt, including retries. The server records it and refuses to do the work twice.

POST /payments
Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7

{ "amount": 5000, "currency": "GBP" }

Server logic, in essence:

1. Look up the key.
   - Found + completed  → return the stored response, do NOT re-execute
   - Found + in-flight  → return 409 / ask the client to retry later
   - Not found          → claim the key, execute, store the response
2. Keep the record for a retention window (24h–7d is typical).

An alternative when you control the data model: natural idempotency via a unique business key. A UNIQUE constraint on (order_id, payment_attempt) makes the second insert fail harmlessly — no separate key infrastructure needed.

Retries: when, and when not

Retrying is only correct for transient failures. Retrying a deterministic error just wastes capacity and delays the real error reaching the caller.

Retry	Don’t retry
Timeouts, connection resets	400 Bad Request — it’ll fail identically
503 Service Unavailable, 429 Too Many Requests	401 / 403 — auth won’t fix itself
500 on a known-idempotent operation	404, 409 — deterministic
Throttling, brief network partitions	Anything non-idempotent without a key

Two more rules that matter as much as the list:

• Retry at one layer only. If the client, the SDK, and the gateway each retry 3×, a single logical request becomes 27 calls. Pick one layer to own retries.
• Honour Retry-After. When a server tells you when to come back, that beats any backoff you compute.

Backoff and jitter

Retrying immediately, or on a fixed interval, is how you turn a brief hiccup into a sustained outage — every client hammers the recovering service in lockstep.

Exponential backoff spaces attempts out geometrically:

attempt 1 → wait 100ms
attempt 2 → wait 200ms
attempt 3 → wait 400ms
attempt 4 → wait 800ms      (capped at some maximum)

But exponential backoff alone isn’t enough. If 10,000 clients fail at the same instant, they all wait 100ms, then all wait 200ms — the retries stay synchronized into a thundering herd that re-kills the service on every wave.

Jitter breaks the synchronization by randomizing the delay:

sleep = random(0, min(cap, base * 2 ** attempt))   // "full jitter"

Always pair backoff with a cap on delay, a maximum attempt count, and an overall deadline — otherwise a retry loop can outlive the request that needed the answer.

Retry amplification and budgets

The failure mode that turns a partial outage into a total one: a service degrades, every caller retries, offered load multiplies exactly when capacity is lowest.

Normal:    1,000 rps
Degraded:  1,000 rps × 3 retries = 3,000 rps against a service already struggling

Two defences:

• Retry budget — cap retries as a fraction of total traffic (e.g. retries may not exceed 10% of requests). Under widespread failure the budget is exhausted and retries stop, instead of scaling with the failure.
• Circuit breaker — stop calling a dependency that is clearly down.

Circuit breakers and bulkheads

Retries handle a single call. These handle a dependency that is failing persistently.

Circuit breaker — a state machine wrapping a dependency:

CLOSED     calls pass through; count failures
   │ failure threshold exceeded
   ▼
OPEN       fail fast immediately — no calls attempted (protects both sides)
   │ after a cooldown
   ▼
HALF-OPEN  let a few trial calls through
   ├─ success → CLOSED
   └─ failure → OPEN

The value is twofold: callers stop wasting time and threads on calls that will fail, and the struggling dependency gets breathing room to recover.

Bulkhead — isolate resources per dependency, so one slow dependency can’t consume every thread or connection in the pool. Named after ship compartments: a flooded compartment doesn’t sink the vessel. This is what stops one degraded downstream from taking your whole service down with it.

Delivery semantics

Retries force a choice about how many times a message can be processed:

• At-most-once — send, never retry. No duplicates, but messages can be lost.
• At-least-once — retry until acknowledged. Nothing is lost, duplicates happen. This is what almost every real queue and RPC layer gives you.
• Exactly-once — the appealing one, and not achievable end-to-end across an unreliable network in the general case.

For messages that fail repeatedly, route them to a dead-letter queue (DLQ) after N attempts. A poison message that can never succeed will otherwise retry forever, blocking the queue and burning capacity. The DLQ makes the failure visible and inspectable instead of infinite.

At a glance

Concept	The one-liner
Idempotent	Repeating the operation doesn’t change the outcome
Idempotency key	Client-generated ID, reused across retries, deduped server-side
Natural idempotency	A unique business key / constraint does the dedup for you
Retry only transient	Timeouts, 429, 503 — never 400/401/404
Exponential backoff	Space attempts geometrically, with a cap
Jitter	Randomize delay so clients don’t retry in lockstep
Retry budget	Cap retries as a share of traffic to prevent amplification
Circuit breaker	Fail fast when a dependency is persistently down
Bulkhead	Isolate resources so one dependency can’t exhaust the pool
At-least-once	What you actually get; assume duplicates
Exactly-once	At-least-once + idempotent consumer
DLQ	Park messages that can never succeed

The model worth keeping: retries are mandatory, and retries are dangerous. Idempotency makes them safe; backoff with jitter, budgets, and breakers make them polite. Get the first wrong and you corrupt data; get the second wrong and you take down the service you were trying to reach.