Idempotency is not a key, it's a contract

Every payments API has an Idempotency-Key header. Far fewer have written down what it means, and the ones that haven't are usually running a SELECT … WHERE key = ? before the insert and calling the job done.

A key is a token. A contract is a set of answers: what counts as the same request, how long the answer stays valid, what a caller gets when their retry lands while the first attempt is still running, and which failures are worth remembering. Skip those and you've made double charges rarer without making them impossible, which is the worst outcome available: now nobody is watching for them.

The promise is "same outcome", not "skip the duplicate"

HTTP already has idempotent methods. RFC 9110 defines PUT and DELETE that way: sending the request twice leaves the server in the same state as sending it once. POST is deliberately excluded, which is why the header exists at all. It's a way to bolt the property onto the one method that doesn't have it.

Notice what the definition covers and what it doesn't. It constrains server state. It says nothing about what the caller sees, and the caller is the entire reason you're doing this. Here's the handler people write first:

const seen = await db.keys.find(key);
if (seen) return res.status(200).json({ status: "ok" });

State is protected. The client is not. They retried because they never saw the first response: the socket died, the load balancer timed out, the phone lost signal in a lift. What they need back is the charge id from attempt one. What they got is {"status":"ok"}, which tells them a charge exists somewhere with an id they will never learn.

So the promise has two halves, and both matter:

1. The work happens at most once.
2. Every retry gets back the original response, with the same status and the same body, byte for byte.

The second half is what turns your endpoint into something a client can safely retry in a loop. Without it, retries are safe for you and useless for them.

What counts as "the same request"

The key is not the identity of the request. It's a name the client picked for one. The identity is the key plus what was in the body.

POST /payments   Idempotency-Key: 7f3a…   { "amount": 5000, "to": "acct_1" }
POST /payments   Idempotency-Key: 7f3a…   { "amount": 9000, "to": "acct_1" }

Replay the first response and you've silently refused to send $90 while telling the caller you sent it. Execute the second and the key bought you nothing. Neither is acceptable, which means there's only one correct answer: reject it. Store a fingerprint of the request alongside the key and compare on every hit.

import { createHash } from "node:crypto";

/** Stable hash of the parts of a request that change what it *does*. */
function fingerprint(path: string, body: unknown): Buffer {
  return createHash("sha256").update(path).update("\0").update(canonicalJson(body)).digest();
}

Stripe returns a 400 with an idempotency_error here. 422 is defensible too. What isn't defensible is picking one of the two bodies and proceeding.

The trap in this section is canonicalJson. Fingerprint the raw bytes and you'll ship false rejections, because a retry is very often a re-serialisation, not a replay of the same buffer. Two things reliably bite:

• Key order. A client that builds the payload from a hash map may emit fields in a different order the second time. Same request, different bytes.
• Timestamps. A client_ts or requested_at field filled in with Date.now() at send time changes on every attempt. Now every retry is a fingerprint mismatch and your safest clients get the most errors.

Canonicalise (sort keys, normalise numbers), or fingerprint an explicit allowlist of the fields that carry meaning. Both work. Hashing the whole body verbatim does not.

The race is the actual problem

Check-then-insert is a time-of-check-to-time-of-use bug wearing a business shirt. Two retries arrive 4 ms apart, both SELECT and miss, both charge. This is not an exotic interleaving; it's the normal shape of a retry storm, because whatever made the client retry (a timeout) also tends to make it retry more than once.

Let the database arbitrate. Insert first, on the way in.

create table idempotency_keys (
  account_id      uuid        not null,
  endpoint        text        not null,
  key             text        not null,
  fingerprint     bytea       not null,
  state           text        not null
                  check (state in ('in_flight', 'succeeded', 'failed')),
  response_status int,
  response_body   jsonb,
  resource_id     uuid,
  locked_until    timestamptz,
  created_at      timestamptz not null default now(),
  primary key (account_id, endpoint, key)
);

const claimed = await db.query(
  `insert into idempotency_keys (account_id, endpoint, key, fingerprint, state, locked_until)
   values ($1, $2, $3, $4, 'in_flight', now() + interval '60 seconds')
   on conflict (account_id, endpoint, key) do nothing
   returning key`,
  [accountId, endpoint, key, fp],
);

if (claimed.rowCount === 1) {
  // We own this key. Do the work, then write the response back.
}

Zero rows means someone else owns it, and the existing row tells you what to do:

The 409 is worth defending, because the tempting alternative is to block: wait on a row lock until the first attempt finishes, then return its response. It reads beautifully and it's a bad idea at load. Every waiter is a held connection, so a provider that's gone slow converts directly into pool exhaustion, and the requests you're holding are by definition from clients that already gave up once. Hand back 409 with a Retry-After and let the client's backoff do the waiting. It has a much better place to do it than your connection pool.

The key has to outlive the process that made it

Server-generated keys don't work. Fetching one is itself a network call that can time out, and now you need idempotency for your idempotency endpoint. So the client owns the key, and where the client stores it decides whether any of this functions.

// Broken: a new key on every attempt. This is a plain retry loop with extra steps.
for (const delay of backoff) {
  await post("/payments", body, { key: crypto.randomUUID() });
}

// Also broken: survives the loop, not a crash. The retry after restart is a new key.
const key = crypto.randomUUID();

The key belongs next to the intent, in whatever durable thing already represents "we mean to pay this": the row in your own database, the payload in the job queue. Generate it once, when the intent is created, and read it back on every attempt including the ones that happen after a deploy, a pod eviction, or somebody replaying a dead-letter queue by hand on Monday morning.

The idempotency key is a property of the job, not of the attempt. If you take one line from this post, take that one. Most idempotency that fails in production fails here, in the client, in code nobody thought of as payment code.

Scope it, or you'll replay someone else's response

Make the key composite: (account_id, endpoint, key). Global uniqueness on the key column alone has two failure modes, and one of them is a security incident.

Two endpoints sharing a key namespace means a client that reuses order-8812 for both /payments and /refunds gets the payment response back from the refund call. Annoying.

Two tenants sharing a namespace is worse. Keys are frequently derived from things that aren't secret (an order number, an invoice id, a checkout-2026-08-12-0001), so tenant B can arrive with a key tenant A already used and receive A's stored response body. You built a cross-tenant read out of a deduplication table. Scope by account and the whole class disappears.

The window is a promise about time, and 24 hours is usually a guess

Stripe expires keys after 24 hours. That's a reasonable default for a browser and a bad fit for a lot of what actually retries.

Think about the longest path a retry of your endpoint can take. A queue with seven-day retention. An operator replaying Friday's failures when they get in on Monday. A mobile client that was in a tunnel, then a plane, then a country with roaming disabled. If your window is 24 hours and your job queue retries for 7 days, then on day two the same key is a new key and the retry is a fresh charge, with the idempotency system fully installed, monitored, and reporting green.

The window must be at least as long as the longest retry horizon of any caller you have. When you can't bound that, split the record instead of expiring it:

• The response body is the expensive part. JSONB of a full resource, times every write request. Expire that on the usual timescale.
• The key, fingerprint and resource_id are 100 bytes. Keep them for a year.

A retry that arrives after the body is gone but while the key survives is still recognisable. Return 409 and point at resource_id. "You already did this, it's payment pay_…, go look" is an unhelpful answer that leaves the caller with a lookup to do. It's also infinitely better than charging them again, which is the only other option once you've forgotten the key entirely.

Idempotency is not atomicity

This is the part that separates a key column from a system that works.

const charge = await provider.charge(amount, card);   // ← crash here
await ledger.record(charge.id, amount);

The process dies between those two lines. The provider has taken the money. Your ledger doesn't know. The key row says in_flight and will say so until the lock expires. The key prevented a duplicate request; it did nothing about partial work, because it was never the kind of thing that could.

An expired in_flight lock is an unknown, not a failure. That distinction is the whole game. Treat it as a failure and re-run the handler and you have built a double-charge machine whose trigger is your own worst outage: the one where processes were dying mid-request. Two things make the recovery path safe, and you want both:

Push your key downstream. Derive the provider's idempotency key deterministically from yours, so a re-run is also a re-run to them:

const providerKey = createHash("sha256").update(key).update(":provider:v1").digest("hex");

Now re-running the handler after a crash is safe at the boundary that matters, because the provider recognises the second call as the same one. The :v1 is there so you can roll the derivation without every historical key changing meaning.

Reconcile before you retry. On finding an expired lock, ask the provider what happened to providerKey before doing anything else. If they have a charge, adopt it, write the ledger row you owe, and mark the key succeeded. Only when they've never heard of it do you execute.

Skipping the reconciliation step and relying purely on the downstream key mostly works, right up until the downstream key window is shorter than yours. Now you know to check that number.

Don't cache failures you'd want retried

The last decision the contract owes an answer to: when the work fails, does the key remember?

It depends entirely on whether re-running would fail the same way.

• Deterministic failures (schema validation, a hard decline, an unknown account) get stored and replayed. Re-running produces the identical error and costs you a provider call to learn it.
• Transient failures (provider 503, a socket timeout, a serialisation conflict, a pod OOM) release the key. Set it back to unclaimed, or clear the lock so the next attempt with the same key claims it cleanly.

Get this backwards and you build a trap. Seal the key as failed on a 500 and the client's retry, using the same key, doing exactly what your docs told them to do, receives a permanent 500. Their only way out is a new key, which is precisely the double charge the header existed to prevent. You'd have been safer with no idempotency at all, because then at least the retry would have worked.

The rule fits on one line: cache a failure if and only if re-running it would produce the same failure.

What the contract actually says

If your API documentation can't answer these, you don't have idempotency. You have a column.

• Who generates the key, and what should it be derived from?
• What's the scope: per account, per endpoint, or both?
• How long does a key live, and what happens on the first request after that?
• What defines "same request", and what's returned when the key matches but the body doesn't?
• What comes back while the first attempt is still running?
• Which failures are remembered and which are retryable?
• Is a replayed response marked as one? (An Idempotent-Replay: true header costs nothing and makes client-side debugging dramatically less miserable.)

Seven answers. Publish them and callers can write a retry loop with confidence. Leave them implicit and every integrator discovers your semantics through an incident.

What this doesn't buy you

Four honest limits:

• Not ordering. Idempotent isn't commutative. A retried update and a retried cancel can still land in either order, and each one being safe on its own says nothing about the pair.
• Not exactly-once. There's no exactly-once delivery over a network; the FLP result and every practical system agree. What you get is at-least-once delivery plus an idempotent receiver, which adds up to effectively-once. That's the real ceiling and it's enough.
• Not double-submit protection. A user hammering Pay twice sends two different keys for two genuinely distinct requests, and idempotency will faithfully process both. That's a business uniqueness constraint (one pending payment per invoice), enforced somewhere else entirely.
• Not correctness. Every one of these mechanisms protects a duplicate. None of them notices that the amount was wrong, the account was wrong, or the rate was stale.

The short version

• Return the original response, not a bare acknowledgement. Retries need the resource id.
• Fingerprint the request. Same key with a different body is an error, never a coin flip.
• Insert first and let a unique constraint settle the race. Check-then-insert loses.
• Answer in-flight requests with 409 and Retry-After. Don't block on a lock.
• The client generates the key, and stores it with the intent, not inside the retry loop.
• Scope keys per account and per endpoint. Global keys leak responses across tenants.
• Size the window against your slowest retry path, and keep the key long after you drop the body.
• An expired in-flight record means unknown. Reconcile before re-running.
• Pass a derived key downstream so a re-run is a retry all the way through.
• Remember deterministic failures. Release transient ones.

A concrete place to start: search your codebase for a SELECT on an idempotency table that runs before the corresponding INSERT. Every one of those is a double charge waiting for a slow afternoon, and the fix is a unique constraint and about twenty lines. Then go and read your queue's maximum retention, and compare it against your key expiry. Those two numbers disagree far more often than anyone expects.