Messenger retries and the failure transport

A log full of the same stack trace, every second, all night. That is what a queue looks like when a handler throws on a malformed payload and the retry policy is “forever”. Symfony 4.3 is out this week, and Messenger in it finally has an answer for that night.

Retry is good for transient errors. Network blinked, deadlock, remote API returned 503. Wait, try again, it passes. But some errors are permanent. Malformed payload, entity deleted, a bug in the handler. Retrying those forever means a worker grinding the same poison message until morning. Infinite retry is not persistence, it is denial.

The setup I converge to:

framework:
    messenger:
        failure_transport: failed
        transports:
            async:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                retry_strategy:
                    max_retries: 3
                    delay: 1000
                    multiplier: 2
            failed: 'doctrine://default?queue_name=failed'

Three attempts with growing delay eats most transient failures. After that the message goes to the failure transport and stops hurting the workers. 4.3 ships the tooling for the morning after: messenger:failed:show to look at the corpses, messenger:failed:retry to replay them once the bug is fixed, messenger:failed:remove for the hopeless ones. Manual replay sounds primitive. It is exactly right. A human decided the cause is gone, a human pushes the button.

Two alerts turn this from a graveyard into a system. Size of the failure queue, because growth means a new class of error appeared. And age of the oldest failed message, because a message sitting there three days is a business operation nobody finished and nobody noticed. The second alert fires more often than you expect. Always some rare flow. Always something with money or email.

Classify the error, bound the retries, keep the corpses where you can see them. The rest is YAML.