Our goal is to start a worker when a task is added to the queue and to destroy the worker after its done processing the queue tasks. This will result in a much lower bill at the end of the month because the worker doesn’t have to be up the whole month.
Setting up Sidekiq
First let’s take a look at a default configuration of Sidekiq within a Rails project.
Add the gem
First we need to add Sidekiq to our Gemfile and run
Create a Sidekiq worker
The worker is very basic. It just performs some heavy task and sends an email after it finished.
1 2 3 4 5 6 7 8 9
In our Procfile we have to define the
worker line to start the Sidekiq server on a Heroku worker dyno.
Move heavy task to the queue
On the controller we probably have a line calling
object.generate_download which takes to much time. We’ll change this line to execute the
1 2 3 4 5 6 7
NOTE: It’s recommended not to add the whole object into the worker because this is stored in the Redis database. Also the object might be changed before it’s being processed by Sidekiq. Adding the
id and fetching the object in the worker is a better approach.
At this point we have configured our Rails project to add tasks to Redis and have Sidekiq perform executing the worker when a task is added. If we deploy our Rails app to Heroku we need to add the Redis To Go addon and add a Heroku worker dyno to our application.
To test this locally we need:
- Redis installed locally and running
- Run the Rails app (
bundle exec rails server)
- Run the Sidekiq server (
bundle exec sidekiq)
Adding the autoscaler
At this point the Heroku worker dyno has to be started, to pickup tasks from the queue. It’s likely you do not need this worker to run the whole day because the queue might be empty most of the time. Here’s where the autoscaler gem come’s in handy.
This gem acts as Middleware for Sidekiq and performs the following tasks:
- When a task is added it checks if a Worker is present.
- When a worker is already running it does nothing and the worker will pickup the task.
- If there isn’t a worker running it will create a Heroku worker dyno.
- If the worker finished processing the queue it will keep the worker alive for 60 seconds just in case we add a task to the queue within this time. If no tasks are added it will destroy the worker.
Add the gem
First we need to add the autoscaler gem to our Gemfile and run
Adding required ENV variables
The gem requires two environment variables to be set on your Heroku application. The
HEROKU_API_KEY is required to perform creating and removing a Heroku worker dyno and the
HEROKU_APP to know on which application it has to create/destroy the worker dyno on.
1 2 3
Tweaking the Sidekiq initializer
Because this gem acts as Middleware we need to create a
sidekiq.rb in our initializers folder. This file will check if we are running on Heroku and if so, activates the autoscaler as Middleware.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Ready to go!
At this point we’re ready to deploy our application to Heroku and let the autscaler automatically create and destroy a worker dyno whenever it needs to process tasks from the Sidekiq queue.