Introduction
Laravel queues provide a unified API across a variety of different queue backends, such as Beanstalk, Amazon SQS, Redis, or even a relational database. Queues allow you to defer the processing of a time consuming task, such as sending an email, until a later time. Deferring these time consuming tasks drastically speeds up web requests to your application.
The queue configuration file is stored in config/queue.php
. In this file you will find connection configurations for each of the queue drivers that are included with the framework, which includes a database, Beanstalkd, Amazon SQS, Redis, and a synchronous driver that will execute jobs immediately (for local use). A null
queue driver is also included which simply discards queued jobs.
Connections Vs. Queues
Before getting started with Laravel queues, it is important to understand the distinction between "connections" and "queues". In your config/queue.php
configuration file, there is a connections
configuration option. This option defines a particular connection to a backend service such as Amazon SQS, Beanstalk, or Redis. However, any given queue connection may have multiple "queues" which may be thought of as different stacks or piles of queued jobs.
Note that each connection configuration example in the queue
configuration file contains a queue
attribute. This is the default queue that jobs will be dispatched to when they are sent to a given connection. In other words, if you dispatch a job without explicitly defining which queue it should be dispatched to, the job will be placed on the queue that is defined in the queue
attribute of the connection configuration:
// This job is sent to the default queue...
dispatch(new Job);
// This job is sent to the "emails" queue...
dispatch((new Job)->onQueue('emails'));
Some applications may not need to ever push jobs onto multiple queues, instead preferring to have one simple queue. However, pushing jobs to multiple queues can be especially useful for applications that wish to prioritize or segment how jobs are processed, since the Laravel queue worker allows you to specify which queues it should process by priority. For example, if you push jobs to a high
queue, you may run a worker that gives them higher processing priority:
php artisan queue:work --queue=high,default
Driver Prerequisites
Database
In order to use the database
queue driver, you will need a database table to hold the jobs. To generate a migration that creates this table, run the queue:table
Artisan command. Once the migration has been created, you may migrate your database using the migrate
command:
php artisan queue:table
php artisan migrate
Redis
In order to use the redis
queue driver, you should configure a Redis database connection in your config/database.php
configuration file.
If your Redis queue connection uses a Redis Cluster, your queue names must contain a key hash tag. This is required in order to ensure all of the Redis keys for a given queue are placed into the same hash slot:
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => '{default}',
'retry_after' => 90,
],
Other Driver Prerequisites
The following dependencies are needed for the listed queue drivers:
Creating Jobs
Generating Job Classes
By default, all of the queueable jobs for your application are stored in the app/Jobs
directory. If the app/Jobs
directory doesn't exist, it will be created when you run the make:job
Artisan command. You may generate a new queued job using the Artisan CLI:
php artisan make:job SendReminderEmail
The generated class will implement the Illuminate\Contracts\Queue\ShouldQueue
interface, indicating to Laravel that the job should be pushed onto the queue to run asynchronously.
Class Structure
Job classes are very simple, normally containing only a handle
method which is called when the job is processed by the queue. To get started, let's take a look at an example job class. In this example, we'll pretend we manage a podcast publishing service and need to process the uploaded podcast files before they are published:
<?php
namespace App\Jobs;
use App\Podcast;
use App\AudioProcessor;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
class ProcessPodcast implements ShouldQueue
{
use InteractsWithQueue, Queueable, SerializesModels;
protected $podcast;
/**
* Create a new job instance.
*
* @param Podcast $podcast
* @return void
*/
public function __construct(Podcast $podcast)
{
$this->podcast = $podcast;
}
/**
* Execute the job.
*
* @param AudioProcessor $processor
* @return void
*/
public function handle(AudioProcessor $processor)
{
// Process uploaded podcast...
}
}
In this example, note that we were able to pass an Eloquent model directly into the queued job's constructor. Because of the SerializesModels
trait that the job is using, Eloquent models will be gracefully serialized and unserialized when the job is processing. If your queued job accepts an Eloquent model in its constructor, only the identifier for the model will be serialized onto the queue. When the job is actually handled, the queue system will automatically re-retrieve the full model instance from the database. It's all totally transparent to your application and prevents issues that can arise from serializing full Eloquent model instances.
The handle
method is called when the job is processed by the queue. Note that we are able to type-hint dependencies on the handle
method of the job. The Laravel service container automatically injects these dependencies.
Note:
Binary data, such as raw image contents, should be passed through thebase64_encode
function before being passed to a queued job. Otherwise, the job may not properly serialize to JSON when being placed on the queue.
Dispatching Jobs
Once you have written your job class, you may dispatch it using the dispatch
helper. The only argument you need to pass to the dispatch
helper is an instance of the job:
<?php
namespace App\Http\Controllers;
use App\Jobs\ProcessPodcast;
use Illuminate\Http\Request;
use App\Http\Controllers\Controller;
class PodcastController extends Controller
{
/**
* Store a new podcast.
*
* @param Request $request
* @return Response
*/
public function store(Request $request)
{
// Create podcast...
dispatch(new ProcessPodcast($podcast));
}
}
">Tip!! The
dispatch
helper provides the convenience of a short, globally available function, while also being extremely easy to test. Check out the Laravel testing documentation to learn more.
Delayed Dispatching
If you would like to delay the execution of a queued job, you may use the delay
method on your job instance. The delay
method is provided by the Illuminate\Bus\Queueable
trait, which is included by default on all generated job classes. For example, let's specify that a job should not be available for processing until 10 minutes after it has been dispatched:
<?php
namespace App\Http\Controllers;
use Carbon\Carbon;
use App\Jobs\ProcessPodcast;
use Illuminate\Http\Request;
use App\Http\Controllers\Controller;
class PodcastController extends Controller
{
/**
* Store a new podcast.
*
* @param Request $request
* @return Response
*/
public function store(Request $request)
{
// Create podcast...
$job = (new ProcessPodcast($podcast))
->delay(Carbon::now()->addMinutes(10));
dispatch($job);
}
}
Note:
The Amazon SQS queue service has a maximum delay time of 15 minutes.
Customizing The Queue & Connection
Dispatching To A Particular Queue
By pushing jobs to different queues, you may "categorize" your queued jobs and even prioritize how many workers you assign to various queues. Keep in mind, this does not push jobs to different queue "connections" as defined by your queue configuration file, but only to specific queues within a single connection. To specify the queue, use the onQueue
method on the job instance:
<?php
namespace App\Http\Controllers;
use App\Jobs\ProcessPodcast;
use Illuminate\Http\Request;
use App\Http\Controllers\Controller;
class PodcastController extends Controller
{
/**
* Store a new podcast.
*
* @param Request $request
* @return Response
*/
public function store(Request $request)
{
// Create podcast...
$job = (new ProcessPodcast($podcast))->onQueue('processing');
dispatch($job);
}
}
Dispatching To A Particular Connection
If you are working with multiple queue connections, you may specify which connection to push a job to. To specify the connection, use the onConnection
method on the job instance:
<?php
namespace App\Http\Controllers;
use App\Jobs\ProcessPodcast;
use Illuminate\Http\Request;
use App\Http\Controllers\Controller;
class PodcastController extends Controller
{
/**
* Store a new podcast.
*
* @param Request $request
* @return Response
*/
public function store(Request $request)
{
// Create podcast...
$job = (new ProcessPodcast($podcast))->onConnection('sqs');
dispatch($job);
}
}
Of course, you may chain the onConnection
and onQueue
methods to specify the connection and the queue for a job:
$job = (new ProcessPodcast($podcast))
->onConnection('sqs')
->onQueue('processing');
Error Handling
If an exception is thrown while the job is being processed, the job will automatically be released back onto the queue so it may be attempted again. The job will continue to be released until it has been attempted the maximum number of times allowed by your application. The number of maximum attempts is defined by the --tries
switch used on the queue:work
Artisan command. More information on running the queue worker can be found below.
Running The Queue Worker
Laravel includes a queue worker that will process new jobs as they are pushed onto the queue. You may run the worker using the queue:work
Artisan command. Note that once the queue:work
command has started, it will continue to run until it is manually stopped or you close your terminal:
php artisan queue:work
">Tip!! To keep the
queue:work
process running permanently in the background, you should use a process monitor such as Supervisor to ensure that the queue worker does not stop running.
Remember, queue workers are long-lived processes and store the booted application state in memory. As a result, they will not notice changes in your code base after they have been started. So, during your deployment process, be sure to restart your queue workers.
Specifying The Connection & Queue
You may also specify which queue connection the worker should utilize. The connection name passed to the work
command should correspond to one of the connections defined in your config/queue.php
configuration file:
php artisan queue:work redis
You may customize your queue worker even further by only processing particular queues for a given connection. For example, if all of your emails are processed in an emails
queue on your redis
queue connection, you may issue the following command to start a worker that only processes only that queue:
php artisan queue:work redis --queue=emails
Resource Considerations
Daemon queue workers do not "reboot" the framework before processing each job. Therefore, you should free any heavy resources after each job completes. For example, if you are doing image manipulation with the GD library, you should free the memory with imagedestroy
when you are done.
Queue Priorities
Sometimes you may wish to prioritize how your queues are processed. For example, in your config/queue.php
you may set the default queue
for your redis
connection to low
. However, occasionally you may wish to push a job to a high
priority queue like so:
dispatch((new Job)->onQueue('high'));
To start a worker that verifies that all of the high
queue jobs are processed before continuing to any jobs on the low
queue, pass a comma-delimited list of queue names to the work
command:
php artisan queue:work --queue=high,low
Queue Workers & Deployment
Since queue workers are long-lived processes, they will not pick up changes to your code without being restarted. So, the simplest way to deploy an application using queue workers is to restart the workers during your deployment process. You may gracefully restart all of the workers by issuing the queue:restart
command:
php artisan queue:restart
This command will instruct all queue workers to gracefully "die" after they finish processing their current job so that no existing jobs are lost. Since the queue workers will die when the queue:restart
command is executed, you should be running a process manager such as Supervisor to automatically restart the queue workers.
Job Expirations & Timeouts
Job Expiration
In your config/queue.php
configuration file, each queue connection defines a retry_after
option. This option specifies how many seconds the queue connection should wait before retrying a job that is being processed. For example, if the value of retry_after
is set to 90
, the job will be released back onto the queue if it has been processing for 90 seconds without being deleted. Typically, you should set the retry_after
value to the maximum number of seconds your jobs should reasonably take to complete processing.
Note:
The only queue connection which does not contain aretry_after
value is Amazon SQS. SQS will retry the job based on the Default Visibility Timeout which is managed within the AWS console.
Worker Timeouts
The queue:work
Artisan command exposes a --timeout
option. The --timeout
option specifies how long the Laravel queue master process will wait before killing off a child queue worker that is processing a job. Sometimes a child queue process can become "frozen" for various reasons, such as an external HTTP call that is not responding. The --timeout
option removes frozen processes that have exceeded that specified time limit:
php artisan queue:work --timeout=60
The retry_after
configuration option and the --timeout
CLI option are different, but work together to ensure that jobs are not lost and that jobs are only successfully processed once.
Note:
The--timeout
value should always be at least several seconds shorter than yourretry_after
configuration value. This will ensure that a worker processing a given job is always killed before the job is retried. If your--timeout
option is longer than yourretry_after
configuration value, your jobs may be processed twice.
Worker Sleep Duration
When jobs are available on the queue, the worker will keep processing jobs with no delay in between them. However, the sleep
option determines how long the worker will "sleep" if there are no new jobs available:
php artisan queue:work --sleep=3
Supervisor Configuration
Installing Supervisor
Supervisor is a process monitor for the Linux operating system, and will automatically restart your queue:work
process if it fails. To install Supervisor on Ubuntu, you may use the following command:
sudo apt-get install supervisor
Laravel Forge, which will automatically install and configure Supervisor for your Laravel projects.
">Tip!! If configuring Supervisor yourself sounds overwhelming, consider using
Configuring Supervisor
Supervisor configuration files are typically stored in the /etc/supervisor/conf.d
directory. Within this directory, you may create any number of configuration files that instruct supervisor how your processes should be monitored. For example, let's create a laravel-worker.conf
file that starts and monitors a queue:work
process:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan queue:work sqs --sleep=3 --tries=3
autostart=true
autorestart=true
user=forge
numprocs=8
redirect_stderr=true
stdout_logfile=/home/forge/app.com/worker.log
In this example, the numprocs
directive will instruct Supervisor to run 8 queue:work
processes and monitor all of them, automatically restarting them if they fail. Of course, you should change the queue:work sqs
portion of the command
directive to reflect your desired queue connection.
Starting Supervisor
Once the configuration file has been created, you may update the Supervisor configuration and start the processes using the following commands:
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start laravel-worker:*
For more information on Supervisor, consult the Supervisor documentation.
Dealing With Failed Jobs
Sometimes your queued jobs will fail. Don't worry, things don't always go as planned! Laravel includes a convenient way to specify the maximum number of times a job should be attempted. After a job has exceeded this amount of attempts, it will be inserted into the failed_jobs
database table. To create a migration for the failed_jobs
table, you may use the queue:failed-table
command:
php artisan queue:failed-table
php artisan migrate
Then, when running your queue worker, you should specify the maximum number of times a job should be attempted using the --tries
switch on the queue:work
command. If you do not specify a value for the --tries
option, jobs will be attempted indefinitely:
php artisan queue:work redis --tries=3
Cleaning Up After Failed Jobs
You may define a failed
method directly on your job class, allowing you to perform job specific clean-up when a failure occurs. This is the perfect location to send an alert to your users or revert any actions performed by the job. The Exception
that caused the job to fail will be passed to the failed
method:
<?php
namespace App\Jobs;
use Exception;
use App\Podcast;
use App\AudioProcessor;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
class ProcessPodcast implements ShouldQueue
{
use InteractsWithQueue, Queueable, SerializesModels;
protected $podcast;
/**
* Create a new job instance.
*
* @param Podcast $podcast
* @return void
*/
public function __construct(Podcast $podcast)
{
$this->podcast = $podcast;
}
/**
* Execute the job.
*
* @param AudioProcessor $processor
* @return void
*/
public function handle(AudioProcessor $processor)
{
// Process uploaded podcast...
}
/**
* The job failed to process.
*
* @param Exception $exception
* @return void
*/
public function failed(Exception $exception)
{
// Send user notification of failure, etc...
}
}
Failed Job Events
If you would like to register an event that will be called when a job fails, you may use the Queue::failing
method. This event is a great opportunity to notify your team via email or HipChat. For example, we may attach a callback to this event from the AppServiceProvider
that is included with Laravel:
<?php
namespace App\Providers;
use Illuminate\Support\Facades\Queue;
use Illuminate\Queue\Events\JobFailed;
use Illuminate\Support\ServiceProvider;
class AppServiceProvider extends ServiceProvider
{
/**
* Bootstrap any application services.
*
* @return void
*/
public function boot()
{
Queue::failing(function (JobFailed $event) {
// $event->connectionName
// $event->job
// $event->exception
});
}
/**
* Register the service provider.
*
* @return void
*/
public function register()
{
//
}
}
Retrying Failed Jobs
To view all of your failed jobs that have been inserted into your failed_jobs
database table, you may use the queue:failed
Artisan command:
php artisan queue:failed
The queue:failed
command will list the job ID, connection, queue, and failure time. The job ID may be used to retry the failed job. For instance, to retry a failed job that has an ID of 5
, issue the following command:
php artisan queue:retry 5
To retry all of your failed jobs, execute the queue:retry
command and pass all
as the ID:
php artisan queue:retry all
If you would like to delete a failed job, you may use the queue:forget
command:
php artisan queue:forget 5
To delete all of your failed jobs, you may use the queue:flush
command:
php artisan queue:flush
Job Events
Using the before
and after
methods on the Queue
facade, you may specify callbacks to be executed before or after a queued job is processed. These callbacks are a great opportunity to perform additional logging or increment statistics for a dashboard. Typically, you should call these methods from a service provider. For example, we may use the AppServiceProvider
that is included with Laravel:
<?php
namespace App\Providers;
use Illuminate\Support\Facades\Queue;
use Illuminate\Support\ServiceProvider;
use Illuminate\Queue\Events\JobProcessed;
use Illuminate\Queue\Events\JobProcessing;
class AppServiceProvider extends ServiceProvider
{
/**
* Bootstrap any application services.
*
* @return void
*/
public function boot()
{
Queue::before(function (JobProcessing $event) {
// $event->connectionName
// $event->job
// $event->job->payload()
});
Queue::after(function (JobProcessed $event) {
// $event->connectionName
// $event->job
// $event->job->payload()
});
}
/**
* Register the service provider.
*
* @return void
*/
public function register()
{
//
}
}
Using the looping
method on the Queue
facade, you may specify callbacks that execute before the worker attempts to fetch a job from a queue. For example, you might register a Closure to rollback any transactions that were left open by a previously failed job:
Queue::looping(function () {
while (DB::transactionLevel() > 0) {
DB::rollBack();
}
});