Designing idempotent APIs sounds straightforward until you account for clock skew, distributed caches, and exactly-once semantics across service boundaries. Here's what I learned building a payment API that had to be right every time.
The naïve approach
The most common way engineers implement idempotency is by adding a unique key to the request headers, usually Idempotency-Key. The server checks if it has seen this key before. If it hasn't, it processes the request and saves the result. If it has, it returns the saved result.
This breaks down in distributed systems in several subtle ways:
- Concurrent requests: What happens if two requests with the same key arrive at the exact same millisecond before the first one finishes?
- Database rollbacks: If the transaction fails, should you save the idempotency key as "failed" or delete it so the client can retry?
- Changing payloads: What if the client retries with the same key but a different request body?
The state machine approach
To build a truly robust system, we had to model idempotency as a state machine. Every key goes through three states: STARTED, COMPLETED, and FAILED.
When a request comes in, we attempt to insert the key into the database with a STARTED state using a unique constraint. If the insert fails, we know another request is already processing it. We wait for it to finish and return the result.