Race conditions in payments: distributed locks on a serverless architecture
· 7 min
On a serverless architecture, every request can be served by a different instance of the process, great for scaling, but it breaks any assumption that two requests from the same user will be seen by the same piece of memory. That matters a lot when those requests involve a card charge or a paid call to an AI service.
The problem: two clicks, two charges
A double-click on a plan-upgrade button, or a retry after a slow response, could fire two nearly simultaneous subscription-mutation requests. Since prorated charges bill immediately on a plan change, those two concurrent requests risked resulting in two real charges on the payment provider for the same user action.
Why an in-memory lock doesn't fix it
The obvious fix, lock the operation with an in-memory mutex while it runs, only works if both concurrent requests land on the same process instance. In production, under a distributed architecture, that's not guaranteed: the two requests could be served by different instances, each with its own isolated memory, and the lock simply wouldn't exist for the second call.
The fix: an atomic lock in the database
Instead of relying on shared memory, the lock moved into the database, the one resource every instance actually shares. A single atomic INSERT with conditional conflict resolution (only yields the lock if the previous one has already expired) acts as a per-user mutex: the second concurrent request simply fails to acquire the lock and gets rejected before it gets anywhere near the payment provider. The same pattern was reused to lock the AI assistant, preventing a user from firing multiple parallel paid calls by hitting "send" again mid-response, with lock renewal during long generations and an explicit cancellation path.
Rate limiting had to go distributed too
The same problem showed up in the login rate limiter: an in-memory, per-instance limiter is trivially bypassed on a distributed architecture, an attacker just needs their attempts to land on different instances. Authentication flows moved to a rate limiter backed by the database, shared across every instance; low-stakes flows kept using the cheaper in-memory version.
The result
No duplicate charges, no duplicate paid AI calls, and distributed brute-force attempts against login started hitting a real limit, all without adding a single new piece of infrastructure, just reusing the database that was already on every request's path as a shared mutex.