If you ask any engineer working on payments whether their system is idempotent, they will say yes. If you then ask them to prove it, what they show you is usually a HTTP header, a database column, or a Set somewhere that tracks request IDs. None of these are idempotency. They are mechanisms that, if implemented carefully and consistently, can produce idempotent behavior — but they are not the same thing, and the gap between the mechanism and the actual property is where the bugs live.
Idempotency is the property that performing an operation multiple times has the same effect as performing it once. In a payment system, this means that if a charge request is sent twice — because of a network retry, a client crash, an overzealous queue worker, or any of the other ways requests get duplicated — the customer is charged exactly once. Not zero times, not two times, exactly once. The bar is high because the consequences of getting it wrong are real money moving in the wrong direction.
The reason teams get this wrong is not that they don't understand what idempotency means. It's that the implementation has subtle requirements that aren't obvious until they fail.
The naive implementation
A typical first implementation looks like this. The client generates a unique idempotency key for each operation. The server receives the request, checks whether it has seen the key before, and either processes the request (if new) or returns the cached response (if seen).
This looks correct. It is correct, in the sense that it handles the simplest case: a client retries a request after receiving no response, and the server, having already processed the original, returns the same result. The customer is charged once. Everyone is happy.
It is also wrong in at least four ways that matter, and most production payment systems exhibit at least two of them.
Failure mode 1: the race window
The check-and-store pattern has a race condition between the check and the store. If two retries arrive in parallel — which happens more often than you'd think, because clients with aggressive retry logic don't wait politely for one retry to complete before issuing another — they can both pass the check before either has stored a result. Both will then process the request. The customer is charged twice.
The fix is to make the check-and-store atomic. The store has to happen before the request is processed, and the store has to be conditional: "insert this key, and if it already exists, do not insert and tell me what's there." The atomic insert is the actual idempotency mechanism. The cache lookup is just an optimization.
Almost every implementation I've reviewed has this race condition. It rarely manifests in testing because tests don't run requests in parallel. It manifests in production the first time a queue worker restarts during a partial deploy and re-runs jobs that another instance is already processing.
Failure mode 2: storing the response before the operation completes
Even with atomic storage, there's a second race: between storing the key and completing the operation. If the server stores the key, then crashes before processing the request, the next retry will see the key and assume the operation already happened — but it didn't. The customer is not charged at all, and the merchant is not paid.
The fix requires distinguishing three states for each key: not seen, in progress, and complete with known result. When the server starts processing a request, it marks the key as in progress. When it finishes, it updates the entry with the result. If a retry arrives while the original is in progress, the server has to decide what to do: wait, return a "still processing" response, or fail with a specific error. Each choice has tradeoffs, and the right answer depends on the operation.
For a payment authorization, my preference is to wait. Authorization requests are idempotent in the sense that the second one shouldn't charge again, but the client wants the result of the original call, not an error. The wait should have a timeout, after which the client gets back a "still processing, original result not yet known" response that they can retry later.
Failure mode 3: the key namespace problem
Idempotency keys need to be unique, but unique with respect to what? Most implementations treat the key as globally unique, meaning that a given key can only be used for one operation ever. This sounds safe, but it creates a different bug: if a client accidentally reuses a key for a different operation, the system returns the cached response from the original operation instead of processing the new one. The client thinks the new operation succeeded, but nothing happened.
The fix is to scope the key. An idempotency key should be unique within a context — usually a customer, an operation type, and a time window. The same key used by two different customers should not collide. The same key used for an authorization and a refund should not collide. The system should either reject mismatched-context reuse (by storing the operation parameters with the key and comparing them) or scope the key namespace so collisions can't happen.
Reject-on-mismatch is the safer choice. Store the request body's hash with the key. When a retry arrives, verify that the hash matches. If it doesn't, the client is reusing a key for a different operation — return an error rather than the cached response. This catches client-side bugs early.
Failure mode 4: the network ambiguity problem
The fundamental challenge of idempotency in payment systems is that a request can fail in a way that leaves the server in a known state and the client in an unknown state. The server processed the charge successfully. The response was sent. The network dropped the response on the way back. The client doesn't know whether the charge happened.
The client retries. The server's idempotency mechanism recognizes the key, returns the cached response, and the client now has the correct answer. This is the case the naive implementation handles.
But there's a worse case. The request was sent. The server received it. The server forwarded it to the upstream payment processor. The processor's response timed out. The server does not know whether the processor processed the charge. It cannot return a definitive response to the client. The idempotency entry is in an indeterminate state.
If the server returns "unknown" to the client, the client doesn't know what to do. If the server retries with the processor, it might cause a duplicate charge. If the server marks the entry as failed, it will incorrectly tell the next retry that the operation didn't happen — when in fact, it might have.
The only correct handling is to model "unknown" as a real state and resolve it asynchronously. The server marks the operation as pending-uncertain. A reconciliation process queries the processor's records to determine what actually happened, then updates the idempotency entry with the resolved result. Until then, retries get an "unknown, please wait" response. This is operationally inconvenient, but it's the only way to avoid double-charging or losing charges in the long run.
Idempotency at the wrong layer
A common architectural mistake is to put idempotency at the wrong layer of the stack. Some teams implement it only at the API gateway, treating idempotency as a cross-cutting concern handled by infrastructure. Others implement it only inside individual service handlers, treating it as application logic.
Both approaches are wrong. Idempotency in payment systems has to be implemented at every layer where requests might be retried, and the layers have to coordinate.
The client retries with an idempotency key. The API gateway recognizes the key and routes to a deterministic handler. The handler checks its own idempotency table. If the operation requires a downstream call to a payment processor, that call has its own idempotency mechanism — usually a separate key that the handler manages. If the processor's response triggers a database write, the write has to be idempotent too.
This means a single client retry causes idempotency checks at three or four different layers, each with its own state and its own failure modes. If any layer skips its check, the chain is broken. If any layer's state diverges from the others, retries produce inconsistent results.
The right model is to think of idempotency as a property of the entire request path, not a single component. Every component along the path has to participate, and the keys at each level have to be linked so that operators can trace a single client request through every layer of the system.
The processor's idempotency is not your idempotency
Modern payment processors usually provide their own idempotency mechanisms. Stripe has idempotency keys. Adyen has reference IDs. Most major processors offer some way to mark a request as a retry of a previous one.
These mechanisms are useful, but they are not a substitute for your own idempotency. Three reasons:
They protect you against duplicate processor calls, not duplicate client requests. If your client sends two requests to your server, and your server forwards both to the processor with different idempotency keys, the processor will process both. Your idempotency layer has to dedupe before the call ever reaches the processor.
They have limits and expiration. Processor idempotency keys typically have a window — 24 hours for Stripe, varying for others. Retries that arrive after the window are processed as new requests. Your own idempotency layer may need a longer window, or a different definition of "the same request."
They don't tell you about ambiguous states. When a processor's idempotency mechanism is invoked on a retry, it returns the cached response. But if the original request was in flight when the retry arrived, the behavior depends on the processor and is sometimes undefined. You can't rely on it to handle the network ambiguity case correctly.
Use processor idempotency as a second line of defense, not your primary mechanism. Your primary mechanism is your own.
What correct idempotency looks like
A correct idempotency implementation in a payment system has these properties:
Atomic insert-or-fetch. Storing a new idempotency key is a single atomic operation, not a check followed by a write. The database (or wherever the keys live) supports a "insert if not exists, otherwise return existing" primitive. Most databases do; many implementations don't use it.
Three-state lifecycle. Keys can be in not-seen, in-progress, or complete state. The state transitions are explicit and durable. Crashes during processing leave the key in in-progress, and recovery logic exists to either resume the operation or transition it to a known state.
Request fingerprint validation. The stored entry includes a hash of the request parameters. Retries are validated against this hash, and mismatches are rejected with an error rather than returning a cached response for a different operation.
Pending-uncertain handling. When the server cannot determine the outcome of an operation — typically because the upstream processor's response was lost — the entry is marked pending-uncertain and resolved asynchronously by querying the processor's records.
Bounded retention. Idempotency entries have an expiration policy that's longer than the longest plausible retry window for the operation. For payment authorizations, this is at least 24 hours and often longer. Entries are not deleted earlier even under storage pressure, because deletion creates the same bug as never storing them.
Key scoping. Keys are scoped to a context that prevents accidental collisions across operations or customers, but allows intentional reuse within the same operation.
Observability. Every idempotency hit is logged with enough information to trace which retry it served and what the original operation was. When something goes wrong, operators can reconstruct the sequence of retries that led to the current state.
Why this matters more than it seems
I have seen teams treat idempotency as a checklist item: "yes, we have an idempotency key on the request." They check the box and move on. The check-the-box approach is what produces the bugs I described. Real idempotency in a payment system is a discipline that affects every layer of the architecture and that requires ongoing attention to corner cases.
The cost of getting it wrong is not abstract. It's customers who get charged twice and demand refunds. It's merchants who don't get paid because a transaction was lost. It's reconciliation reports that don't balance, and operators who have to manually figure out what happened. It's the small but constant erosion of trust in the system that comes from anomalies nobody can fully explain.
The cost of getting it right is engineering time spent on the unsexy work of distributed systems correctness. It's not glamorous. It does not produce features. It is exactly the kind of work that gets deferred until something breaks badly enough to demand attention.
Build it before you need it. Build it carefully. Test it under failure injection — not by mocking failures, but by actually killing processes during requests and verifying that retries produce the right outcome. Treat every double-charge or lost-charge incident as a sign that the idempotency layer needs another invariant.
If your idempotency mechanism is a column on a table and a check at the start of a handler, you do not have idempotency. You have a hopeful gesture in the direction of idempotency. The difference matters, and it will become visible at exactly the wrong time.
This is part of a series on payment systems architecture. See also why payment state machines are harder than you think and designing payment systems that don't break at scale.