The job called the payment provider. The call succeeded. Then the worker timed out before it marked the job done. The queue did what queues do: it retried. The provider did what it was asked: it charged again. The customer did what customers do and wrote an angry email.
Nobody made a mistake here. The queue promises at-least-once delivery, and “at least” is written in the contract. Network partitions, worker crashes, deploy restarts. Sooner or later every job runs twice, and the jobs that hurt are exactly the ones with external effects: payments, emails, webhooks, API calls.
Exactly-once delivery does not exist in practice. Exactly-once business effect is achievable, and it is your job, not the queue’s. The tool is an idempotency key.
Before touching the outside world, the job writes an operation row with a unique constraint on the key:
INSERT INTO operations (idempotency_key, status)
VALUES ('charge-order-1042', 'started');
The second execution of the same job hits the duplicate key error and knows the story: this operation already ran or is running. Read its status. Finished, then quietly exit. Started long ago and never finished, then you are in the genuinely hard case, and you must ask the external system what actually happened before retrying.
If the provider accepts an idempotency key of its own, always pass one. Then even the ugly timeout case is safe, because the provider refuses the duplicate charge itself. The good payment APIs support this, and it is a serious argument when choosing one.
The unique constraint does the real work here. Do not replace it with a SELECT check before insert, that is a race, the same one I wrote about in January. Let the database enforce uniqueness. It is the only participant that can.
Every job will run twice. I write them that way now. It took one angry email.