At-Least-Once vs Exactly-Once Delivery: Why the Customer Got Charged Twice

A customer taps Pay once. Their card is charged €120 twice. Nobody double-clicked, and there is no bug in your checkout code.

The cause sits one level below your application, and once you see it, all three "delivery guarantees" you've heard of stop being vocabulary and become the only three options that exist.

• Mental model: The network can eat your acknowledgement, not just your message. So a sender genuinely cannot tell "it never arrived" apart from "it arrived, worked, and the reply was lost." Every delivery semantic is just a different answer to that one moment.

---

The floor: two worlds you cannot tell apart

Here's the whole system. A checkout service sends one message — charge order 4471 — to a payments worker.

checkout  ──[ charge order 4471 ]──►  payments worker
          ◄────────[ ack ]──────────  (card charged ✅)

Now delete the ack. Not the message — the reply.

From the payments worker's side, everything is fine: the card was charged, the ack was sent. From the checkout service's side, it sent a message into the dark and heard nothing back. It is now sitting in front of two possible worlds:

1. The message never arrived. Nothing happened. The customer has not been charged.
2. The message arrived, the card was charged, and only the ack got lost.

These two worlds look byte-for-byte identical to the sender. No amount of clever code fixes this — it isn't a bug, it's a property of networks. There is no "did it work?" packet you can send that couldn't itself be lost.

So the sender has to make a choice, and that choice is the entire topic.

---

Choice 1: don't retry → at-most-once

Send it, hope, move on.

queue.send(msg)   # that's it. no ack tracking, no retry.

Every message is delivered zero or one times, never twice. It's the fastest option and it will never double-charge anyone.

It also loses messages silently — and silence is the problem. When the ack disappears, you don't retry, so world #1 (nothing happened) just stands. You lost a sale and nothing in your system says so.

Perfectly fine for a metrics ping or a heartbeat, where message 4,471 not landing is genuinely irrelevant. For a payment, it's a bug that never files a ticket.

Choice 2: retry until you hear an ack → at-least-once

The obvious fix: keep redelivering until someone confirms.

while not acked:
    queue.send(msg)
    acked = wait_for_ack(timeout=5)

Nothing is ever lost. This is also what Kafka, RabbitMQ and SQS actually give you by default — the broker redelivers anything it can't confirm was processed.

And it's exactly how our customer paid €240. The ack died, the sender assumed world #1, redelivered charge order 4471, and the worker — which had already charged the card — charged it again. Duplicates aren't an edge case here; they are the normal, designed behaviour of at-least-once.

Choice 3: "exactly-once" — the marketing word

Every message processed once. Never lost, never duplicated. It's what you actually want, and it's on a lot of feature pages.

Here's the honest version: a broker can make genuinely strong promises about its own log — its offsets, its transactions, its internal state. Kafka's "exactly-once semantics" is real, and it is about Kafka reading from Kafka and writing back to Kafka atomically.

It can promise nothing about what your code does when the message lands. Charging a card, sending an email, calling a third-party API — those are your side effects, in your process, outside the broker's world entirely. No setting on the broker reaches into your worker and un-charges a card.

Semantic	Mechanism	Lost messages?	Duplicates?	Use it for
At-most-once	Fire and forget, no retry	✅ Yes — silently	❌ Never	Metrics, heartbeats, telemetry
At-least-once	Retry until acked	❌ Never	✅ Yes — by design	Almost everything (the real default)
"Exactly-once"	At-least-once + idempotent consumer	❌ Never	❌ Not observable	Payments, orders, anything with a side effect

Note the third row carefully: duplicates are still delivered. They're just no longer observable, because your consumer refuses to act on them twice.

---

What you actually build: at-least-once + an idempotent consumer

You don't get exactly-once by turning it on. You get it by making a duplicate delivery harmless.

Every message carries a stable id — stable meaning the retry carries the same one, so it can't be a UUID generated inside the worker. Before doing the work, the consumer claims that id in a dedup store:

INSERT INTO processed_messages (message_id)
VALUES ('order-4471')
ON CONFLICT DO NOTHING;

• Insert won (1 row) → first time I've seen this. Charge the card.
• Insert lost (0 rows) → already done. Ack, drop it, move on.

The critical part is where that write lives:

with db.transaction():                       # ONE transaction
    claimed = db.execute(INSERT_ON_CONFLICT, msg.id).rowcount
    if not claimed:
        return ack(msg)                      # duplicate — nothing to do
    charge_card(msg.order_id, msg.amount)    # the work
    db.insert_payment_record(msg.order_id)

The dedup key and the work must commit together. Write the key first and crash before the charge, and the retry sees "already done" and skips a payment that never happened. Charge first and crash before the key, and you're back to double-charging. Same transaction, or you've just built a more confident version of the original bug.

The honest costs

Nobody puts these on the feature page:

1. The dedup store is real infrastructure. It's a Postgres table or a Redis keyspace that grows with every message you process, and it's now on your critical path — if it's down, your consumer is down.

2. TTL means you picked a window, and it's a bet. You can't keep keys forever, so you expire them after, say, 24 hours. A retry arriving at hour 25 looks brand new and goes straight through. "Exactly-once" is really "exactly-once within my retention window" — choose it knowing your broker's maximum retry horizon.

3. Some side effects can never be deduped. No dedup store un-sends a receipt email. Which is why the check goes before the effect, never after — and why irreversible effects belong at the very end of the handler, after everything that can still fail has failed.

This is also why payment APIs hand you an idempotency key: the exact same idea, one hop earlier, at the API edge instead of the queue consumer.

---

The same problem, one abstraction up: AI agents and LLM pipelines

If you're building with LLMs, you already have this problem — it just doesn't look like a message queue.

An agent loop is a consumer, and tool calls are side effects. Every serious agent framework retries: on a 429, on a 529 overload, on a socket timeout, on a malformed tool-call payload. That retry logic is at-least-once delivery by another name, and the failure looks identical:

# the LLM call times out AFTER the tool already ran
result = tools.send_invoice(customer_id="c_88", amount=4200)   # ✅ ran
# ...connection drops before the result comes back
# agent retries the step → invoice sent twice

The model never "clicked twice" either. The tool executed, the response was lost, the orchestrator couldn't tell world #1 from world #2, and it did the only safe-looking thing: retried.

The fix transfers exactly:

• Give every tool call a stable id derived from the step — the run id plus the step index, not a fresh UUID per attempt.
• Claim it before executing, in the same transaction as recording the result.
• Split tools by reversibility. Read-only tools (search, retrieval, get_*) are naturally idempotent — retry them freely. Effectful tools (send, charge, post, delete, write-to-CRM) need the dedup claim.

The same rule governs ingestion pipelines: an at-least-once document queue feeding an embedding job will happily embed the same chunk twice and quietly corrupt your retrieval with duplicate neighbours. Key the vector on a content hash, upsert instead of insert, and the duplicate delivery becomes a no-op.

And provider webhooks — batch-job-completed, run-finished — are at-least-once too. Every provider tells you to expect redelivery. They're telling you to build an idempotent consumer.

---

The verdict

At-most-once is a deliberate choice you make only when losing a message is genuinely cheaper than handling a duplicate. That's telemetry, and not much else.

At-least-once is what you'll actually be running, whether or not you chose it — it's the default of every mainstream broker. So duplicates are not a failure mode to prevent; they're an input to design for.

Exactly-once is not a checkbox on your broker. It's at-least-once delivery plus a consumer that claims a stable id in a dedup store, in the same transaction as the work, with a retention window you chose on purpose and an ordering rule that puts the check before the irreversible effect.

Exactly-once is not a setting you turn on. It's a property your consumer has.

Your turn: your worker charges the card, then writes the dedup key — and crashes between the two. What does the retry do? (That's the bug the single-transaction rule exists to kill.)

---

🎥 Watch the reel: At-Least-Once vs Exactly-Once Delivery — the whole thing animated, with the Sneakr checkout running the entire way through.