Documentation menu
On this page
Packages · v2.x
Queue
The Queue package (src/packages/Queue) runs work asynchronously: the application dispatches messages (any serializable PHP object), a transport stores them, and a worker (php bin/neo queue:work) handles them in the background with retries, exponential backoff and a failed storage.
It ships the sync, database, filesystem and redis transports, handler discovery with #[AsMessageHandler], self-handling jobs, console commands and a WebProfiler panel. No external dependency is required (ext-redis only for the redis transport).
Module
| Manager | NeoPHP\Package\Queue\QueueManager (final) |
| Interface | NeoPHP\Package\Queue\QueueManagerInterface |
| Attribute | #[Package(provider: QueueProvider::class)] |
| Requires | nothing |
Inject QueueManagerInterface in a service, a command or a controller:
use NeoPHP\Package\Queue\QueueManagerInterface;
public function __construct(private QueueManagerInterface $queue)
{
}
The module is enabled by default. To disable it in a project, add it to config/config.php (see the Kernel documentation):
NeoPHP\Package\Queue\QueueManager::class => false,
The public API of the module is its manager and its interface, Contract\, the attributes, the exceptions, the events and the classes documented below. The classes marked @internal are used by the framework only.
Quick start
- Generate a message and its handler:
php bin/neo make:message SendNewsletter
- Dispatch it from a controller:
use App\Message\SendNewsletter;
$this->dispatchMessage(new SendNewsletter(42));
- Consume the queue:
php bin/neo queue:work
The default transport is QUEUE_DSN (database://default in the generated .env): the tables are created on first use (auto_setup: true) or with php bin/neo queue:setup.
Messages and handlers
A message is a plain serializable object; its handler is a class of src/ carrying #[AsMessageHandler] whose __invoke() receives the message (the message class is read from the type of the first argument):
namespace App\Message;
use NeoPHP\Package\Queue\Attribute\AsMessage;
#[AsMessage(transport: 'async', queue: 'emails')]
class SendNewsletter
{
public function __construct(public int $newsletterId)
{
}
}
namespace App\MessageHandler;
use App\Message\SendNewsletter;
use App\Service\NewsletterSender;
use NeoPHP\Package\Queue\Attribute\AsMessageHandler;
#[AsMessageHandler]
class SendNewsletterHandler
{
public function __construct(protected NewsletterSender $sender)
{
}
public function __invoke(SendNewsletter $message): void
{
$this->sender->send($message->newsletterId);
}
}
#[AsMessageHandler] options:
| Option | Default | Description |
|---|---|---|
handles |
type of the first argument | message class (or parent class / interface) |
method |
__invoke |
method called on the handler (class attribute) |
priority |
0 |
order when several handlers handle the same message (highest first) |
The attribute can also be put on public methods (one handler class for several messages). Handlers are built by the container (constructor autowiring) and may handle a parent class or an interface of the message.
Handlers are discovered in src/ and in the NeoPHP packages installed with Composer (see the Package documentation), and cached in var/cache/queue/handlers.{env}.php (refreshed automatically in debug). Explicit handlers can be added in handlers: of config/packages/queue.yaml.
Self-handling jobs
A message implementing NeoPHP\Package\Queue\Contract\JobInterface without a handler is a job: its handle() method is called through the container, so its arguments are autowired:
namespace App\Message;
use NeoPHP\Component\Mailer\MailerManagerInterface;
use NeoPHP\Package\Queue\Contract\JobInterface;
class SendWelcomeEmail implements JobInterface
{
public function __construct(public string $email)
{
}
public function handle(MailerManagerInterface $mailer): void
{
}
}
JobInterface is a marker interface: handle() is not declared in it so that it can receive any service.
Dispatching
Inject MessageBusInterface (or QueueManagerInterface, which extends it) or use the controller helper:
use NeoPHP\Package\Queue\Contract\MessageBusInterface;
public function __construct(protected MessageBusInterface $bus)
{
}
$envelope = $this->bus->dispatch(new SendNewsletter(42), ['queue' => 'emails', 'delay' => 60]);
$this->dispatchMessage(new SendNewsletter(42), ['delay' => new DateTimeImmutable('tomorrow 08:00')]);
| Option | Description |
|---|---|
transport |
transport name (overrides the routing) |
queue |
queue name (default: #[AsMessage(queue:)], else the queue option of the transport, else default) |
delay |
seconds, or a DateTimeInterface (available at) |
priority |
integer, highest first (default 0) |
max_attempts |
attempts before the failed storage (default retry.max_attempts) |
dispatch() returns the Envelope (id, transport, queue, attempts, available at...). The container aliases are QueueManagerInterface, MessageBusInterface, QueueManager, queue and message_bus.
Routing
The transport of a message is, in order: the transport option, routing: of the configuration (class, parent class or interface), #[AsMessage(transport:)], then default_transport.
#[AsMessage] also accepts queue, priority and maxAttempts.
Transports
Transports are configured by DSN (name: dsn or name: { dsn: ..., options: { ... } }); the query string of the DSN is merged into the options:
| DSN | Transport | Storage |
|---|---|---|
sync:// |
SyncTransport |
none: the message is handled during dispatch() (exceptions are thrown to the caller) |
database://default |
DatabaseTransport |
tables neo_queue_jobs and neo_queue_failed of a connection of the Database component (SQLite, MySQL, PostgreSQL) |
filesystem://%kernel.root_path%/var/queue |
FilesystemTransport |
one JSON file per message in <dir>/<queue>/, reserved in <queue>/.reserved/, failed in <dir>/failed/ |
redis://[:password@]host:6379/0 |
RedisTransport |
sorted sets `neo_queue: |
Options (all transports): queue (default queue), retry_after (seconds after which a reserved message whose worker died is delivered again, default 90), auto_setup. Database: table, failed_table. Redis: prefix (neo_queue:), timeout.
The database transport stores integer timestamps (available_at, reserved_at, created_at, failed_at) and the message class (message_class). A message is claimed with UPDATE ... SET reserved_at = ? WHERE id = ? AND reserved_at IS NULL: only one worker gets it, on every driver, without table locks.
The filesystem transport claims a message with an atomic rename() and writes every file through a temporary file; it is meant for a single server.
Custom transports
Implement TransportInterface (or extend Contract\AbstractTransport) and a TransportFactoryInterface whose supports(string $dsn) recognizes your scheme, then register the factory class:
transports_factories: [ App\Queue\SqsTransportFactory ]
Configuration
config/packages/queue.yaml (every key is optional):
| Key | Default | Description |
|---|---|---|
default_transport |
first transport | transport of the unrouted messages |
transports |
async: QUEUE_DSN |
name => DSN or { dsn, options }; sync: 'sync://' is always added |
routing |
{ } |
message class => transport |
handlers |
{ } |
message class => handler class, list of classes or { handler, method, priority } |
retry.max_attempts |
3 |
attempts before the failed storage |
retry.delay |
1 |
first retry delay (seconds) |
retry.multiplier |
2 |
delay multiplier per attempt |
retry.max_delay |
3600 |
maximum delay (seconds) |
auto_setup |
true |
creates the tables / directories on first use |
retry_after |
90 |
see transports |
transports_factories |
[ ] |
custom transport factories |
profiler_stats |
true |
counters of the transports in the profiler panel |
Without configuration file, the async transport uses QUEUE_DSN when defined, else database://default when a database connection is configured, else filesystem://var/queue.
###> queue ###
QUEUE_DSN="database://default"
###< queue ###
Workers
php bin/neo queue:work [transport] [--queue=high,default] [--limit=100] [--time-limit=3600] [--memory-limit=128] [--sleep=1] [--stop-when-empty] [--once]
Queues are consumed in the given order (the first queue has priority). Each message prints a line: time, message class, id, queue, attempt, status (handled, retry in Ns, failed) and duration.
The worker stops gracefully (after the current message) on SIGTERM, SIGINT or SIGQUIT when ext-pcntl is available, when a limit is reached, or after php bin/neo queue:restart (it writes a timestamp in var/queue/restart, read by the workers after each message).
In production, run the workers with a process manager that restarts them (systemd, Supervisor) and call queue:restart after each deployment. On Windows (no pcntl), stop the worker with Ctrl+C between two messages or use --time-limit / queue:restart; the Task Scheduler can start php bin/neo queue:work --stop-when-empty every minute.
Worker::run() and Worker::process() can also be used in code ($container->get(Worker::class)).
Retries and failed messages
When a handler throws, the message is released with an exponential backoff: delay * multiplier ^ (attempt - 1) seconds, capped by max_delay (1 s, 2 s, 4 s... with the defaults). After max_attempts attempts, the message moves to the failed storage of its transport (table neo_queue_failed, directory failed/ or the redis failed hash) with the last error.
Exceptions implementing UnrecoverableExceptionInterface (for example UnrecoverableMessageException, a missing handler or an invalid signature) skip the retries.
php bin/neo queue:failed
php bin/neo queue:retry 12 # or: queue:retry all
php bin/neo queue:forget 12
php bin/neo queue:flush-failed
Security of the payloads
Messages are serialized with serialize() and signed with HMAC-SHA256 using APP_SECRET (framework.app.secret). A payload whose signature is invalid (modified in the database, the files or redis, or signed with another secret) is never unserialized: it goes to the failed storage with SerializationException. Changing APP_SECRET makes the pending messages unreadable.
Events
Dispatched when the Event component is available:
| Event | When |
|---|---|
MessageDispatchedEvent |
after dispatch() (isHandledSync()) |
WorkerMessageReceivedEvent |
a worker received a message |
WorkerMessageHandledEvent |
the message was handled (getDuration() in ms) |
WorkerMessageFailedEvent |
the handler failed (getError(), willRetry(), getRetryDelay()) |
Commands
| Command | Arguments and options |
|---|---|
queue:work |
transport, --queue, --limit/-l, --time-limit/-t, --memory-limit/-m (128), --sleep/-s (1), --stop-when-empty, --once |
queue:status |
transport: ready, delayed, reserved and failed messages per queue |
queue:failed |
--transport, --limit/-l (50) |
queue:retry |
id... or all, --transport |
queue:forget |
id, --transport |
queue:flush-failed |
--transport, global --force skips the confirmation |
queue:purge |
transport, --queue, global --force skips the confirmation (failed messages are kept) |
queue:restart |
stops the running workers after their current message |
queue:setup |
transport: creates the tables / directories |
make:message |
name: src/Message/<Name>.php and src/MessageHandler/<Name>Handler.php |
Profiler
When the WebProfiler is enabled, Helper/WebProfiler/QueueProfiler adds a "Queue" panel (no toolbar item): the messages dispatched during the request (class, transport, queue, delay, priority, handled sync or sent with its id, duration, error) and the counters of every transport (ready, delayed, reserved, failed; disable with profiler_stats: false).
The messages are recorded only when the profiler is enabled (QueueTrace is attached to the bus only in that case).
Exceptions
All in NeoPHP\Package\Queue\Exception\, extending QueueException (itself a FrameworkException):
| Exception | Thrown when |
|---|---|
ConfigurationException |
unknown transport or scheme, invalid DSN, missing APP_SECRET, missing ext-redis, invalid handler |
TransportException |
storage error (tables, files, redis connection) |
SerializationException |
the message cannot be serialized, the payload is not signed or its signature is invalid (unrecoverable) |
NoHandlerException |
no handler for the message (unrecoverable) |
UnrecoverableMessageException |
thrown by a handler to skip the retries |
Limitations
- At-least-once delivery: a worker killed during a handler, or a handler running longer than
retry_after, leads to a second delivery. Handlers should be idempotent. - The filesystem transport is for a single server (local disk); the redis transport ignores the priority between delayed messages until they are ready and requires Redis 5 (
ZPOPMIN) withEVAL: a message is moved between its states in one atomic step, so a worker killed in the middle does not lose it. - The
synctransport has no retry and no failed storage. - With
auto_setup: true, the profiler panel may create the queue tables on its first display.
Changelog
- v2.13.0 — The redis transport moves the messages between their states atomically; the created directories use the
0775permissions. - v2.8.0 —
Helper/Doctor/QueueCheck:neo doctorchecks the transports and reports the failed messages (see the Doctor documentation). - v2.1.0 — The handlers of the NeoPHP packages are discovered.
- v2.0.0 —
QueueManageris thefinalentry point of the module, declared with#[Package];QueueManagerInterfacereplacesContract\QueueInterface;Helper/Profileris renamedHelper/WebProfiler;Envelopemoves toMessage\Envelope; the internal classes are marked@internal. - v1.37.0 — Queue package: messages and
#[AsMessage]routing,#[AsMessageHandler]handlers (discovered and cached), self-handlingJobInterfacejobs,MessageBusInterface/QueueInterface,sync,database,filesystemandredistransports configured by DSN with custom factories, HMAC-signed payloads, delays, priorities, retries with exponential backoff, failed storage, workers with limits, signals andqueue:restart, events,dispatchMessage()in controllers,queue:*andmake:messagecommands, Queue panel of the WebProfiler.