Delayed Job – Example on How to Deal with Long Running Rails Tasks

When an application needs to deal with some long running tasks, we probably should move those tasks to run asynchronously as background jobs, more precisely to use Rails Delayed Job. Let’s examine what we need to do in order to enqueue and execute some tasks in the background.

Set up Delayed Job

With Rails Active Job we can easily define our jobs and execute them on various queuing backends. Delayed Job is one of those backends, that perfectly match out needs.

Let’s assume that we already have a Rails application with one model named Team. For start, we will need to add delayed_job_active_record gem to app’s Gemfile,

gem 'delayed_job_active_record'

and run bundle install to install it.
Then we will run the following commands, to create delayed_jobs database table and delayed_job file in the bin directory
rails generate delayed_job:active_record
rails db:migrate

Next we will set delayed_job as ActiveJob queue adapter,
# config/application.rb

config.active_job.queue_adapter = :delayed_job

and set default delayed job configuration.
# config/initializers/delayed_job_config.rb

Delayed::Worker.destroy_failed_jobs = false
Delayed::Worker.sleep_delay = 60
Delayed::Worker.max_attempts = 3
Delayed::Worker.max_run_time = 5.minutes
Delayed::Worker.read_ahead = 10
Delayed::Worker.default_queue_name = 'default'
Delayed::Worker.delay_jobs = !Rails.env.test?
Delayed::Worker.raise_signal_exceptions = :term
Delayed::Worker.logger = Logger.new(File.join(Rails.root, 'log', 'delayed_job.log'))

In order to menage background processes we will add `gem to our Gemfile. That makes us completely ready to create and test out our first background job. Create the Job
[ruby]
rails generate job teams_statistic

This will create teams_statistic_job.rb file under the app/jobs directory, with the following content:
class TeamsStatisticJob < ApplicationJob
queue_as :default

def perform(*args)
# Do something later
end
end

Let’s suppose that, within Team class, we have one instance method for long and complex calculations, named calculate_statistic.
class Team < ApplicationRecord
...

def calculate_statistic
# some complex calculation

puts "I am calculating #{self.name} statistic!"
end
end

Now we can easily call that method from the job’s perform method, passing the team as an argument.
def perform(team)
team.calculate_statistic
end

Let’s enqueue a job through Rails console.
red_star = Team.find_by_name("Red Star")
TeamsStatisticJob.perform_later(red_star)

Because Delayed Job is database based queue system, previous command will insert one row in the delayed_jobs table.
Also through logs we can see that we just enqueued a job to be performed as soon as the queuing system is free.
Enqueued TeamsStatisticJob (Job ID: new-job-id) to DelayedJob(default) with arguments: GlobalID:...

Start background worker
We can start background workers with this rake task bundle exec rake jobs:work or by running bin/deleyed_job script (by the way, that’s the reason why we added daemons gem).
# commands that runs bin/deleyed_job script

RAILS_ENV=production bin/delayed_job start
RAILS_ENV=production bin/delayed_job stop

Let’s open new terminal and run the rake task to actually perform the job.
As our logs are showing, long running task is now successfully completed.
Starting job worker
Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=12) (queue=default)
RUNNING

I am calculating Red Star statistic! # calculate_statistic method output

Job ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper (id=12) (queue=default)
COMPLETED after 0.0771

And that is it.
However, there are many useful features (priorities, named queues, …) in the Delayed Job documentation, that I didn’t cover in this post. I encourage everyone to check those out…
                      

Ruby on Rails gives us, out of the box, a few very useful query methods that can reduce number of database queries. In most cases that will significantly improve application performance.
This post will try to explain and compare some of the most commonly used methods like joins and includes, and to explain few different techniques how we can resolve N+1 query “problem”.
For demonstration purpose I created an application with two database tables teams and players, with one-to-many association between them.
class Player < ActiveRecord::Base
  belongs_to :team
end

class Team < ActiveRecord::Base
  has_many :players
end

Also, we will need some database records for teams and players.
red_star = Team.create(name: "Red Star")
partizan = Team.create(name: "Partizan")

Player.create(full_name: "Ognjen Dobrić", team: red_star)
Player.create(full_name: "Nemanja Dangubić", team: red_star)
Player.create(full_name: "Novica Veličković", team: partizan)
Player.create(full_name: "Vanja Marinković", team: partizan)

Our goal will be to fetch all the players together with the team name of each player, so let’s play a little bit in the Rails console. We will start with **lazy loading** technique which the `ActiveRecord` query interface uses by default.
Lazy loading
Philosophy of lazy loading design pattern is to delay initalization of an object until the time when it is really needed.
> Player.all.map { |player| [player.full_name, player.team.name] }

This query will result with this array,
[
  [ "Ognjen Dobrić",     "Red Star" ],
  [ "Nemanja Dangubić",  "Red Star" ],
  [ "Novica Veličković", "Partizan" ],
  [ "Vanja Marinković",  "Partizan" ]
]

and this logs.
Player Load (0.4ms)  SELECT "players".* FROM "players"
Team Load (0.2ms)  SELECT  "teams".* FROM "teams" WHERE "teams"."id" = ? LIMIT ?  [["id", 1], ["LIMIT", 1]]
Team Load (0.1ms)  SELECT  "teams".* FROM "teams" WHERE "teams"."id" = ? LIMIT ?  [["id", 1], ["LIMIT", 1]]
Team Load (0.1ms)  SELECT  "teams".* FROM "teams" WHERE "teams"."id" = ? LIMIT ?  [["id", 2], ["LIMIT", 1]]
Team Load (0.1ms)  SELECT  "teams".* FROM "teams" WHERE "teams"."id" = ? LIMIT ?  [["id", 2], ["LIMIT", 1]]

Our result is just what we wanted, but the number of queries in our logs indicates that we have N+1 situation, which may be a problem is case when we have a lot of records.
N+1 means that we have executed one query for the players and then one query for each player’s team (in our case N is 4)
Let’s eager load teams and reduce the number of queries.
Eager loading
Eager loading technique allow as to preload all “players” associated data (“teams” in our case).
> Player.
    includes(:team).
    map { |player| [player.full_name, player.team.name] }

Our result is still the same, but the number of queries is now only two, which is great.
<bash]

Player Load (0.4ms) SELECT “players”.* FROM “players”

Team Load (0.8ms) SELECT “teams”.* FROM “teams” WHERE “teams”.”id” IN (1, 2)

[/bash]
The number of queries can be reduced even more, to just one, and includes() method is smart enough to figure out when will be the good situation to run two queries and when a single query will be the most suitable option.
In short, includes() combines behaviours of two Rails methods, preload() when we need two separate queries and eager_load() in cases when it is best to run a single query.
Let’s examine the next example. Suppose that we want to fetch only the Red Star players.
> Player.
    includes(:team).
    where(teams: { name: "Red Star" }).
    map { |player| [player.full_name, player.team.name] }

Which results with the new array.
[
  [ "Ognjen Dobrić",     "Red Star" ],
  [ "Nemanja Dangubić",  "Red Star" ]
]

As we can see the includes() method allows as to filter players by the team name and our logs show that we triggered only one query to accomplish that.
SQL (2.4ms)  SELECT "players"."id" AS t0_r0, "players"."full_name" AS t0_r1, "players"."created_at" AS t0_r2, "players"."updated_at" AS t0_r3, "players"."team_id" AS t0_r4, "teams"."id" AS t1_r0, "teams"."name" AS t1_r1, "teams"."created_at" AS t1_r2, "teams"."updated_at" AS t1_r3
FROM "players"
LEFT OUTER JOIN "teams" ON "teams"."id" = "players"."team_id"
WHERE "teams"."name" = ?  [["name", "Red Star"]]

In most cases eager loading will do the trick, but it is very obvious that we just loaded all columns from teams table but we only needed the team name.
Let’s try with joins
joins() is a Rails prefered way when we need to specify conditions on associations, so let’s test it.
> Player.
    joins(:team).
    where(teams: { name: "Red Star" }).
    map { |player| [player.full_name, player.team.name] }

This query will produce the following logs:
Player Load (0.2ms)  SELECT "players".*
FROM "players"
INNER JOIN "teams" ON "teams"."id" = "players"."team_id"
WHERE "teams"."name" = ?  [["name", "Red Star"]]

Team Load (0.1ms)  SELECT  "teams".* FROM "teams" WHERE "teams"."id" = ? LIMIT ?  [["id", 1], ["LIMIT", 1]]
Team Load (0.1ms)  SELECT  "teams".* FROM "teams" WHERE "teams"."id" = ? LIMIT ?  [["id", 1], ["LIMIT", 1]]

The result is still the same, and a good thing is that we don’t load unnecessary data into memory any more. But as we can see from the logs, N+1 situation is back again.
In order to eliminate unnecessary queries, we will need to use select() method together with joins().
`joins` combined with `select` method
> Player.
    joins(:team).
    where(teams: { name: "Red Star" }).
    select("players.*, teams.name as team_name").
    map { |player| [player.full_name, player.team_name] }

Our result is still ok and our logs show that we needed only one query to achieve that.
Player Load (0.5ms)  SELECT players.*, teams.name as team_name
FROM "players"
INNER JOIN "teams" ON "teams"."id" = "players"."team_id"
WHERE "teams"."name" = ?  [["name", "Red Star"]]

You can notice, that I changed player.team.name to player.team_name in map() block after I added select() method, that is because team_name is now a Player model instance attribute.
For this situation it is very useful if we add team_name instance method to Player class.
class Player < ActiveRecord::Base
  belongs_to :team

  def team_name
    read_attribute("team_name") || team.name
  end
end

What we will use depends on case by case. My personal guidelines are: “Start with eager loading when you need to use an associated data and go with joins when you only need to query an associated data.”
                      

  

At the beginning of their career many new Rails developers put complex presentation logic directly into the templates, that looks something like this:
<%= @player.first_name.capitalize + " " + @player.nick_name.capitalize + " " + @player.last_name.capitalize %>

At that point, that kind of solution seems reasonable, but after awhile they all ask the same question: Is this the right place for such a code?
Of course, the answer is always negative, and they rush to find a better place for that code.
By default, Player model seems like the right place for new clean code, and full_name method will probably serve the purpose very well. That now looks something like this:
class Player < ApplicationRecord
  def full_name
    "#{first_name.capitalize} #{nick_name.capitalize} #{last_name.capitalize}"
  end
end

and in the template we can easily call our new full_name method:
<%= @player.full_name %>

Great, everything works as expected, and the developer is very proud. This feeling will last as long as developer does not have a lot of similar methods, that can make Player model very huge and difficult to maintain. Which brings us to the Decorator pattern. Lets see how we can use it to keep our Rails templates and models clean.
What is the Decorator pattern?
Decorator pattern enables us to easily add an enhancement to an existing object and to avoid possible problems when we throw all methods in one class (Player model in our case).
So, lets create our decorator. For me, the best practice is to create decorators directory inside app directory and to put all of my decorators there. Then we will create PlayerDecorator class which will inherit from SimpleDelegator, which is one of native Ruby class from the Ruby Delegator library. Now we can add full_name method directly to PlayerDecorator class and remove one from the Player model.
class PlayerDecorator < SimpleDelegator
  def full_name
    "#{first_name.capitalize} #{nick_name.capitalize} #{last_name.capitalize}"
  end
end

We can decorate a Player model instance before passing it to a template (e.g. form some controller action). Or, we can decorate an Player model instance directly, within a view, like this:
<%= PlayerDecorator.new(@player).full_name %>

So, to conclude, when you move your complex presentation logic into the decorators, your templates and models will stay clean, and easy to maintain.
                      
I recently worked on a Rails project, which had parts of pages in different languages. That may be a problem if you have already translated their entire text to all required languages. You can even be tempted to hardcode parts of the text into other languages. Fortunately, there is an elegant way to solve that problem, just wrap parts of template or partials into blocks with desired locale, like this:
<% I18n.with_locale('en') do %>
  ...part of your template
  or
  <%= render partial: 'some/partial' %>
<% end %>

Example
Suppose, there is a template with only header and two paragraphs.
<h1><%= t('my_great_header') %></h1>

<p><%= t('first_paragraph') %></p>

<p><%= t('second_paragraph') %></p>

And locale in English and French for that template.
# in config/locales/en.yml
en:
  my_great_header: "My English great header"
  first_paragraph: "First English paragraph"
  second_paragraph: "Second English paragraph"

# in config/locales/fr.yml
fr:
  my_great_header: "My French great header"
  first_paragraph: "First French paragraph"
  second_paragraph: "Second French paragraph"

And client wants first paragraph to always be in English.
Just wrap first paragraph in block with locale 'en, like this:
<h1><%= t('my_great_header') %></h1>

<% I18n.with_locale('en') do %>
  <p><%= t('first_paragraph') %></p>
<% end %>
<p><%= t('second_paragraph') %></p>

and when you switch language to Franch result will be:
My French great header

First English paragraph

Second French paragraph

I hope that this helps. Have a nice day.
                      
In the lifetime of every application the time comes for it to be presented to everyone. That’s why we have to put our application on a special server which is designed for this purpose. In one word, we need to deploy our application. In this post you will see how to deploy app with Capistrano 3.
Capistrano is a great developers tool that is used to automatically deploy projects to remote server.
Add Capistrano 3 to Rails app
I will assume you already have a server set up and an application ready to be deployed remotely.
We will use gem ‘capistrano-rails’, so we need to add this gems to Gemfile:
group :development do
  gem 'capistrano', '~> 3.5'
  gem 'capistrano-rails', '~> 1.1.6'
end

and install gems with $ bundle install.
Initialize Capistrano
Then run the following command to create configuration files:
$ bundle exec cap install

This command creates all the necessary configuration files and directory structure with two stages, staging and production:
Capfile
config/deploy.rb
config/deploy/production.rb
config/deploy/staging.rb
lib/capistrano/tasks

Require needed gems in Capfile
Open the Capfile and add or uncomment this lines:
require 'capistrano/setup'
require 'capistrano/deploy'
require 'capistrano/bundler'
require 'capistrano/rails/assets'
require 'capistrano/rails/migrations'
Dir.glob('lib/capistrano/tasks/*.rake').each { |r| import r }

Add capistrano-rbenv gem
The capistrano-rbenv gem provides rbenv support for Capistrano 3.
Add this line to the Gemfile:
group :development do
  gem 'capistrano', '~> 3.5'
  gem 'capistrano-rails', '~> 1.1.6'
  gem 'capistrano-rbenv', '~> 2.0', require: false
end

And require this gem in Capfile require 'capistrano/rbenv'.
Add capistrano-passenger gem
The capistrano-passenger gem adds a task to restart your application after deployment via Capistrano 3.
group :development do
  ...
  gem 'capistrano-rbenv', '~> 2.0', require: false
  gem 'capistrano-passenger', '~> 0.2.0'
end

And require this gem in Capfile require 'capistrano/passenger'


Configure deploy.rb file Open config/deploy.rb and add options for deployment:

	


+ set all needed variables, this is the variant with two servers (_staging and production_) and with user created on server (_server setup is theme for different post_)
set :application, 'app-name'   # application name
set :deploy_user, 'user-name'   # name of user who is set on server
set :repo_url, 'git@github.com:nickname/repo_name.git'   # your repository url from github
set :branch, ENV.fetch('BRANCH', 'master')   # branch which you want to deploy from

set the path where you want to find your app on server, starting from server’s root
	
set :deploy_to, -> { "/path/to/app/#{fetch(:rails_env)}-#{fetch(:application)}" }


set config files, Capistrano uses a folder called shared to manage files and directories that should persist across releases
set :linked_files, fetch(:linked_files, []).push('config/database.yml', 'config/secrets.yml')
set :linked_dirs, fetch(:linked_dirs, []).push('log', 'tmp/pids', 'tmp/cache', 'tmp/sockets', 'vendor/bundle', 'public/system')


next, set ruby version, we use `gem ‘capistrano-rbenv’` for this setup
set :rbenv_type, :user
set :rbenv_ruby, '2.2.2'


after that, set option for restarting your application after deployment with gem 'capistrano-passenger'
set :passenger_restart_with_touch, true


here you can put all kinds of rake tasks for different needs that you can run every time when you deploy your application.
namespace :deploy do
  desc "Description of task"
  task :name_of_task do
      # do something
  end
end



Capistrano’s server settings
You need to tell Capistrano where to find your server.

This is an example of server’s settings for application where everything is on same machine (application, server, database).

In config/deploy/staging set:
server 'your.staging.server.com', user: fetch(:deploy_user), roles: %w{app db web}


and set rails environment
set :rails_env, 'staging'



Also set the same configuration for production server.
In `config/deploy/production` add:
server 'your.production.server.com', user: fetch(:deploy_user), roles: %w{app db web}
set :rails_env, 'production'

Deploy your application
Just run deploy task:
bundle exec cap staging deploy

or

bundle exec cap production deploy

and that is it, your app is live and you can visit it on server’s name url, in our example case your.staging.server.com.
Note: you can find complete documentation on Capistrano site.
Have a nice day!
 
                      
Sooner or later every new Ruby developer needs to understand differences between rake db:schema:load and rake db:migrate. Basically, these simple definition tells us everything we need to know:

rake db:migrate runs migrations that have not run yet
rake db:schema:load loads the schema.db file into database.

but the real question is when to use one or the other.
Advice:  when you are adding a new migration to an existing app then you need to run rake db:migrate, but when you join to existing application (especially some old application), or when you drop your applications database and you need to create it again, always run rake db:schema:load to load schema.
Example
I am working on application which use globalize gem for ActiveRecord model/data translations. Globalize work this way:

first specify attributes which need to be translatable
class Post < ActiveRecord::Base
translates :title, :text
end


then create translation tables
class CreatePosts < ActiveRecord::Migration
def up
create_table :posts do |t|
t.timestamps
end
Post.create_translation_table! title: :string, text: :text
end
def down
drop_table :posts
Post.drop_translation_table!
end
end

Note that the ActiveRecord model Post must already exist and have listed attributes for translations.
and run rake db:migrate.

Problem comes when you change your mind and decide to leave title to be untranslatable.
remove title from post translations table
class RemoveTitleFromPostTranslations < ActiveRecord::Migration
def up
remove_column :post_translations, :title, :string
end

def down
Entry.add_translation_fields! title: :string
end
end


add title to posts table
class AddTitleToPosts < ActiveRecord::Migration
def change
add_column :posts, :title, :string
end
end


remove title attribute from model translations
class Post < ActiveRecord::Base
translates :text
end


and run rake db:migrate.

Everything looking good, so where is the problem?
Here it is! If you decide to delete your database and create it again you need to use:

rake db:drop
rake db:create
rake db:schema:load

Because, if you try to use rake db:migrate instead of rake db:schema:load you will get BIG ERROR!, because for your first migration “create_posts” it is necessary that you have defined translatable attributes :title and :text in Post model, but you removed :title from Post model translations.
So just follow advice above, and good luck.
                      
If you use Vagrant, VirtualBox and Ubuntu to build your Rails apps and you want to test it with Cucumber scenarios, this is the right post for you. By default Vagrant and VirtualBox use Ubuntu without an X server and GUI.
Everything goes well until you need @javascript flag for your cucumber scenario. @javascript uses a javascript-aware system to process web requests (e.g. Selenium) instead of the default (non-javascript-aware) webrat browser.
Install Mozilla Firefox
Selenium WebDriver is flexible and lets you run selenium headless in servers with no display. But in order to run, Selenium needs to launch a browser. If there is no display to the machine, the browsers are not launched. So in order to use selenium, you need to fake a display and let selenium and the browser think they are running in a machine with a display.
Install latest version of Mozilla Firefox:
sudo apt-get install firefox
Since Ubuntu is running without a X server Selenium cannot start Firefox because it requires an X server.
Setting up virtual X server
Virtual X server is required to make browsers run normally by making them believe there is a display available, although it doesn’t create any visible windows.
Xvfb (X Virtual FrameBuffer) works fine for this. Xvfb lets you run X-Server in machines with no display devices.
Install xvfb on ubuntu:
sudo apt-get install xvfb
Lets run the Xvfb service in a display number which is less likely to clash even if you add a display at a later stage. Display 10 will do fine.
sudo Xvfb :10 -ac
The parameter -ac makes xvfb run with access control off. The server should be running now.
Headless Firefox
Before you can run a browser, you need to set the environment variable DISPLAY with the display number at which xvfb is running.
Open new tab in terminal and set the DISPLAY variable:
export DISPLAY=:10
and start mozilla firefox:
firefox
Now you run firefox headlessly in Ubuntu, and you can run your cucumber scenarios with @javascript flag.
Start virtual X server automatically
To run your X server automatically, after installing Xvfb, you will need to:

put content of this gist in /etc/init.d/xvfb (hint use sudo wget command to do that)
make it executable sudo chmod a+x /etc/init.d/xvfb
start xvfb on display number 10 export DISPLAY=:10
run X server sudo /etc/init.d/xvfb start
when you want to stop X server sudo /etc/init.d/xvfb stop

This is my way to run firefox headlessly in Virtual box Ubuntu, and to run cucumber scenarios with @javascript flag.
References:

Selenium Headless Automated Testing in Ubuntu
Xvfb init script for Ubuntu