A failed job that retries every second, one hundred times, against a payment provider that is already down. That is a small DDoS with a queue in front of it. If the provider needs five minutes to recover, hammering it for the first two achieves nothing except log volume.
Symfony Messenger has everything needed. You just have to configure it on purpose:
framework:
messenger:
transports:
async:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
retry_strategy:
max_retries: 5
delay: 2000
multiplier: 4
failure_transport: failed
Two seconds, eight, thirty two, and so on. Exponential backoff gives the external service room to breathe. I also add jitter in a custom retry strategy, because a hundred messages that failed together will otherwise retry together, in one synchronized wave. Same failure, five times.
The part people skip is the difference between transient and permanent failures. Timeout is transient, retry makes sense. “Invalid card number” is permanent, five retries will produce five identical rejections. Throw UnrecoverableMessageHandlingException for the permanent ones and they go straight to the failure transport. Retrying a validation error is not persistence. It is denial.
The failure transport is the second half. It is a queue of business incidents that need a human. messenger:failed:show to look, messenger:failed:retry to replay after the cause is fixed. On one project we review it every morning, same as Sentry. A message that sits there is usually a customer who did not get something.
And one alert that earns its place: age of the oldest message in the queue. Queue length can stay low while one poisoned message cycles forever. Age cannot. If the oldest message is an hour old, something is stuck.
The hundred-retries config at the top was real. Git blame said it was me.