Distributed SystemsStaff

The Duplicate Payment

A customer is charged twice. The retry looked harmless in code review.

IdempotencyEvent processingConsistency25 min

How this works

01
Read the case
Context, code, logs and metrics, the same material you would have in production, with nobody pointing at the problem.
02
Write your diagnosis
No multiple choice and no single right answer: you write what you think is happening, what you would do and what it costs.
03
Compare with the solution
The moment you submit, the solution written by an engineer opens next to your answers. Nobody scores you: the comparison is yours to make, and it stays open to reread.

Context

Checkout does not call the payment provider directly. When an order is confirmed, it publishes a payment.requested event to the Kafka topic payments.requested and returns. A separate service, payments-processor, consumes that topic, calls the PSP over HTTP and records the result in PostgreSQL.

checkout  ──▶  payments.requested  ──▶  payments-processor (4 instances)
                                              │
                                              ├──▶  PSP  POST /v1/charges
                                              └──▶  postgres  insert into payments

The PSP supports idempotency keys: send the same Idempotency-Key header twice and the second call returns the original charge instead of creating another. The team knew this and has used it since day one. The service has been in production for fourteen months.

> The code is in TypeScript because some language had to be picked. The same > handler in Java, Go or Python has the same bug, on the same line.

Problem

A customer opened a ticket on Saturday morning with two charges of R$ 1,299.00 for a single order, 40 seconds apart. It was not an isolated case: 41 orders had two captured charges each, all between 20:11 and 20:26 on Friday.

There was no deploy on Friday. The producer was checked — checkout published exactly one event per order. The PSP status page records a degradation in the same window, with p50 going from 180ms to 820ms.

Investigation

The PSP dashboard, for order ORD-84213:

 charge_id          amount   status     idempotency_key
 ch_3PqL2sK9x1Bd    129900   captured   6f7c5a41-2b0e-4e2a-9a71-0d3f8c2b7e14
 ch_3PqLdN7m0Rtz    129900   captured   b3e91d08-77af-4c5e-8b12-5aa6f0e4c993

Two charges, two different idempotency keys.

The handler, and the consumer configuration:

export async function onPaymentRequested(event: PaymentRequested) {
  const charge = await psp.charge({
    orderId: event.orderId,
    amountCents: event.amountCents,
    idempotencyKey: randomUUID(),   // Idempotency-Key
  });

  await payments.insert({
    orderId: event.orderId,
    pspChargeId: charge.id,
    status: "captured",
  });
}

properties

enable.auto.commit   = false
max.poll.records     = 500     # offsets committed after the whole batch
max.poll.interval.ms = 300000

, instance , 20:14:

Questions

With no deploy and no producer duplication, what made the same message be processed twice?
The PSP deduplicates by idempotency key and the code sends one on every call. Why did that not stop the second charge?
What would you change so a duplicate delivery can no longer become a duplicate charge — under rebalance, crash or manual replay?

What opens

Answer it, then compare with the solution.

Room to answer each question, with a draft saved while you think
The solution written by an engineer: root cause, the fix we would pick, the alternatives and the trade-offs
Your answers kept next to it, to reread whenever you want

Reading is free; answering is what uses a credit. Every new account starts with 3, no card.