Queues on Murat Useinov

Retry is a strategy, not a loop

Wed, 27 Apr 2022 20:20:00 +0300

A failed job that retries every second, one hundred times, against a payment provider that is already down. That is a small DDoS with a queue in front of it. If the provider needs five minutes to recover, hammering it for the first two achieves nothing except log volume.

Symfony Messenger has everything needed. You just have to configure it on purpose:

framework:
    messenger:
        transports:
            async:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                retry_strategy:
                    max_retries: 5
                    delay: 2000
                    multiplier: 4
        failure_transport: failed

Two seconds, eight, thirty two, and so on. Exponential backoff gives the external service room to breathe. I also add jitter in a custom retry strategy, because a hundred messages that failed together will otherwise retry together, in one synchronized wave. Same failure, five times.

Laravel 8: the interesting parts are not on the marketing page

Fri, 11 Sep 2020 23:15:00 +0300

->refundedTwice(). That is the line from Laravel 8 I care about, and it is nowhere on the release page. Jetstream gets the screenshots. Class-based factories and the queue changes get my attention, because both are about production.

Factories used to be closures registered through a global function. Now they are classes with states:

Order::factory()
    ->paid()
    ->has(OrderItem::factory()->count(3))
    ->create();

Test data quality decides test quality. Most integration suites on our project test the happy path fifty times, because the default factory returns a fresh, valid, boring record. The bugs live somewhere else. An order refunded twice. A user registered before the migration added the column. A subscription that expired in the middle of renewal. A factory state gives such a monster a name, and once the name exists, people write tests with it. Named ugly data is the cheapest test improvement I know.

Static state outlives the job

Sun, 22 Dec 2019 23:50:00 +0300

One static array, from one of our projects:

class Settings
{
    private static $cache = [];

    public static function get($tenantId, $key)
    {
        if (!isset(self::$cache[$tenantId])) {
            self::$cache[$tenantId] = self::load($tenantId);
        }
        return self::$cache[$tenantId][$key] ?? null;
    }
}

A reasonable per-request cache. Under PHP-FPM it died with the process. In a queue worker it lives forever.

PHP had one great architectural feature nobody put in the manual: the process died after every request. Leak memory, cache nonsense in a static, forget to close things. Did not matter. The dying process forgave everything.

Messenger retries and the failure transport

Thu, 30 May 2019 23:25:00 +0300

A log full of the same stack trace, every second, all night. That is what a queue looks like when a handler throws on a malformed payload and the retry policy is “forever”. Symfony 4.3 is out this week, and Messenger in it finally has an answer for that night.

Retry is good for transient errors. Network blinked, deadlock, remote API returned 503. Wait, try again, it passes. But some errors are permanent. Malformed payload, entity deleted, a bug in the handler. Retrying those forever means a worker grinding the same poison message until morning. Infinite retry is not persistence, it is denial.

One payment is enough

Mon, 21 Jan 2019 20:35:00 +0300

Charge the card, send the receipt, ack the job. The worker lost its Redis connection between step one and step three. The queue delivered the job again. The customer paid twice. Support learned some new words from him.

Retries are not an edge case. Laravel retries failed jobs by design, and you want that, because networks blink. So every job with a side effect must answer one question: what happens if this runs twice. “It will not run twice” is not an answer. It will.

Messenger: in-process today, queue tomorrow

Tue, 19 Jun 2018 22:50:00 +0300

$bus->dispatch(new SendWelcomeEmail($user->getId())); and then, in the same request, a handler:

class SendWelcomeEmailHandler
{
    public function __invoke(SendWelcomeEmail $message)
    {
        // load user by id, send the email
    }
}

That is Messenger in Symfony 4.1, still marked experimental. By default everything is synchronous, so at first it is a function call with extra steps.

The extra steps are for the transport. Change configuration, route this message class to AMQP, and the same handler runs in a worker process. The calling code does not change. Start synchronous, go async when you need it. I like this order much more than “install RabbitMQ on day one”.

Horizon, or the queue becomes visible

Sat, 20 Jan 2018 10:30:00 +0300

redis-cli llen queues:default and hope. That was my queue monitoring for years. Horizon replaces it with a real dashboard, and I did not know how much I needed one until I saw it.

The mental shift matters more than the UI. dispatch() is the beginning of the work, not the end. The job still has to wait in Redis, run, maybe fail, maybe retry. All of that was invisible. Now it is on one screen: throughput, wait time per queue, failed jobs with the full payload and the exception.

The queue will run your job twice

Mon, 03 Oct 2016 15:20:00 +0300

The job called the payment provider. The call succeeded. Then the worker timed out before it marked the job done. The queue did what queues do: it retried. The provider did what it was asked: it charged again. The customer did what customers do and wrote an angry email.

Nobody made a mistake here. The queue promises at-least-once delivery, and “at least” is written in the contract. Network partitions, worker crashes, deploy restarts. Sooner or later every job runs twice, and the jobs that hurt are exactly the ones with external effects: payments, emails, webhooks, API calls.

Queues in Laravel 4.2: the user should not wait for your SMTP

Thu, 27 Nov 2014 20:50:00 +0300

1.8 seconds for one registration request. I put a timer around it on a project last month. 1.5 of those seconds was the welcome email going out over SMTP.

The user waits almost two seconds and looks at a spinner, for a handshake with a mail server he will never hear about.

Laravel 4.2 makes the fix one line:

Queue::push('SendWelcomeEmail', array('user_id' => $user->id));

The controller returns in 300 ms. A worker picks the job up and sends the email. php artisan queue:listen to start, beanstalkd or Redis behind it, the failed_jobs table for jobs that died.