System of a test: Proper browser testing in Ruby on Rails

Read how to speed up your Ruby tests on our blog:

TestProf: a good doctor for slow Ruby tests

TestProf II: Factory therapy for your Ruby tests

Ruby community is passionate about testing. We have plenty of testing libraries , there are hundreds of blog posts on the topic, we even had a dedicated podcast . Hell, the  top 3 most downloaded gems are parts of the  RSpec testing framework!

Rails, I believe, is one of the reasons behind the rise of Ruby testing. The framework makes writing tests as enjoyable as possible. In most situations, following the comprehensive Rails testing guide is good enough (at least, in the beginning). But there is always an exception, and, in our case, it is  system testing .

Writing and maintaining system tests for Rails applications can hardly be called “enjoyable”. My approach to handling this problem evolved a lot since my first Cucumber -driven test suite back in 2013. And today, in 2020, I finally reached the point when I’m ready to share my current setup with everyone. In this post, I’m going to cover the following topics:

• System tests in a nutshell
• Modern system tests with Cuprite
• Configuration example
• Dockerizing system tests

System tests in a nutshell

“System tests” is a common naming for automated end-to-end tests in the Rails world. Before Rails adopted this name, we used such variations as feature tests, browser tests, and even acceptance tests (though the latter are ideologically different ).

If we recall the testing pyramid (or diamond ), system tests live on the very top: they treat the whole program as a black box, usually emulating end-users activities and expectations. And that is why, in case of web applications, we need a browser to run such tests (or at least an emulation like Rack Test ).

Let’s take a look at the typical system tests architecture:

That’s not the only way of writing end-to-end tests in Rails. For example, you can use Cypress JS framework and IDE. The only reason stopping me from trying this approach is the lack of  multiple sessions support , which is required for testing real-time applications (i.e., those with AnyCable :wink:).

We need to manage at least three “processes” (some of them could be Ruby threads): a web server running our application, a browser and a test runner itself. That’s the bare minimum. In practice, we usually also need another tool to provide an API to control the browser (e.g., ChromeDriver). There were attempts to simplify this setup by building specific browsers (such as capybara-webkit and  PhantomJS ) providing such APIs out-of-box, but none of them survived the compatibility race with real browsers.

And, of course, we need to add a handful of Ruby gems to our test dependencies—to glue all pieces together. More dependencies—more problems. For example, Database Cleaner for a long time was a must-have add-on: we couldn’t use transactions to automatically rollback the database state, because each thread used its own connection; we had to use TRUNCATE ... or  DELETE FROM ... for each table instead, which is much slower. We solved this problem by using a shared connection in all threads (via the  TestProf extension ). Rails 5.1 was released with a similar functionality out-of-the-box.

Thus, by adding system tests, we increase the maintenance costs for development and CI environments and introduce potential points of failures or instability: due to the complex setup, flakiness is the most common problem with end-to-end testing. And most of this flakiness comes from communication with a browser.

My standard system_helper.rb as of 2019 contained more than 200 lines of code! It’s still valid if you want to continue using Selenium.

Although by introducing system tests in 5.1 Rails simplified keeping browser tests, they still require some configuration efforts to work smoothly:

• You need to deal with web drivers (Rails assumes that you use Selenium).
• You’re on your own with configuring system tests in a containerized environment (i.e., when using Docker for development ).
• The configuration is not flexible enough (e.g., screenshots path).

Let’s move closer to code and see how you can make system testing more fun in 2020!

Modern system tests with Cuprite

By default, Rails assumes that you will be running system tests with Selenium . Selenium is a battle-proven software for web browsers automation. It aims to provide a universal API for all browsers as well as the most realistic experience. Only real humans made of meat and bones could do better in terms of emulating user-browser interactions.

This power doesn’t come for free: you need to install browser-specific drivers, the overhead of realistic interactions is noticeable at scale (i.e., Selenium tests are usually pretty slow).

Selenium was created a while ago, at the times when browsers didn’t provide any built-in capabilities for automation. Several years ago, this situation changed with the introduction of CDP protocol for Chrome. Using CDP, you can manipulate browser sessions directly, no need for intermediate abstraction layers and tools.

A lot of projects leveraging CDP appeared since then, including the most well-known one— Puppeteer , a browser automation library for Node.js. What about the Ruby world? Ferrum , a CDP library for Ruby, although being a pretty young one, provides a comparable to Puppeteer experience. And, what’s more important for us, it ships with a companion project called Cuprite —a pure Ruby Capybara driver using CDP.

I’ve started using Cuprite actively only in the beginning of 2020 (I tried it a year before but had some problems with Docker environment) and I never regret it. Setting up system tests became bloody simple (all you need is

Chrome), and the execution is so fast that after migrating from Selenium some of my tests failed: they lacked proper async expectations and passed in Selenium only because it was much slower.

Let’s take a look at the most recent system tests configuration with Cuprite I worked on.

Annotated configuration example

This example borrows from my recent open-source Ruby on Rails project- AnyCable Rails Demo . It aims to demonstrate how to use the just released AnyCable 1.0 with Rails apps, but we can also use it for this post—it has a decent system tests coverage.

The project uses RSpec and its system testing wrapper. Most of the ideas could be applied to Minitest as well.

Let’s start with a minimal example enough to run tests on a local machine. This code lives in the demo/dockerless branch of AnyCable Rails Demo.

Let’s take a quick at the Gemfile first:

group :test do
  gem 'capybara'
  gem 'selenium-webdriver'
  gem 'cuprite'
end

What? Why do we need selenium-webdriver if we’re not using Selenium at all? It turns out that Rails requires this gem to be present independently of a driver you use. Hopefully, that has been fixed and we’ll be able to remove this gem in Rails 6.1.

I keep system tests configuration in multiple files in the spec/system/support folder and use a dedicated system_helper.rb to load them:

spec/
  system/
    support/
      better_rails_system_tests.rb
      capybara_setup.rb
      cuprite_setup.rb
      precompile_assets.rb
      ...
  system_helper.rb

Let’s take a look at each file from the list above and see what it is for.

system_helper.rb

The system_helper.rb file may contain some general RSpec configuration for system tests, but, usually, it’s as simple as the following:

# Load general RSpec Rails configuration
require "rails_helper.rb"

# Load configuration files and helpers
Dir[File.join(__dir__, "system/support/**/*.rb")].sort.each { |file| require file }

Then, in your system specs, you use require "system_helper" to  activate this configuration.

We use a separate helper file and a support folder for system tests to avoid all the excess configuration in the case when we only need to run a single unit test.

capybara_setup.rb

This file contains configuration for Capybara framework:

# spec/system/support/capybara_setup.rb

# Usually, especially when using Selenium, developers tend to increase the max wait time.
# With Cuprite, there is no need for that.
# We use a Capybara default value here explicitly.
Capybara.default_max_wait_time = 2

# Normalize whitespaces when using `has_text?` and similar matchers,
# i.e., ignore newlines, trailing spaces, etc.
# That makes tests less dependent on slightly UI changes.
Capybara.default_normalize_ws = true

# Where to store system tests artifacts (e.g. screenshots, downloaded files, etc.).
# It could be useful to be able to configure this path from the outside (e.g., on CI).
Capybara.save_path = ENV.fetch("CAPYBARA_ARTIFACTS", "./tmp/capybara")

This file also contains a useful patch for Capybara, which purpose we will reveal later:

# spec/system/support/capybara_setup.rb

Capybara.singleton_class.prepend(Module.new do
  attr_accessor :last_used_session

  def using_session(name, &block)
    self.last_used_session = name
    super
  ensure
    self.last_used_session = nil
  end
end)

The Capybara.using_session allows you to manipulate a different browser session, and thus, multiple independent sessions within a single test scenario. That’s especially useful for testing real-time features, e.g., something with WebSocket.

This patch tracks the name of the last session used. We’re going to use this information to support taking failure screenshots in multi-session tests.

cuprite_setup.rb

This file is responsible for configuring Cuprite:

# spec/system/support/cuprite_setup.rb

# First, load Cuprite Capybara integration
require "capybara/cuprite"

# Then, we need to register our driver to be able to use it later
# with #driven_by method.
Capybara.register_driver(:cuprite) do |app|
  Capybara::Cuprite::Driver.new(
    app,
    **{
      window_size: [1200, 800],
      # See additional options for Dockerized environment in the respective section of this article
      browser_options: {},
      # Increase Chrome startup wait time (required for stable CI builds)
      process_timeout: 10,
      # Enable debugging capabilities
      inspector: true,
      # Allow running Chrome in a headful mode by setting HEADLESS env
      # var to a falsey value
      headless: !ENV["HEADLESS"].in?(%w[n 0 no false])
    }
  )
end

# Configure Capybara to use :cuprite driver by default
Capybara.default_driver = Capybara.javascript_driver = :cuprite

We also define a few shortcuts for common Cuprite API methods:

module CupriteHelpers
  # Drop #pause anywhere in a test to stop the execution.
  # Useful when you want to checkout the contents of a web page in the middle of a test
  # running in a headful mode.
  def pause
    page.driver.pause
  end

  # Drop #debug anywhere in a test to open a Chrome inspector and pause the execution
  def debug(*args)
    page.driver.debug(*args)
  end
end

RSpec.configure do |config|
  config.include CupriteHelpers, type: :system
end

Below you can see a demonstration of how this #debug helper works.

better_rails_system_tests.rb

This file contains some patches to Rails system tests internals as well as some general configuration (see code comments for explanations):

# spec/system/support/better_rails_system_tests.rb

module BetterRailsSystemTests
  # Use our `Capybara.save_path` to store screenshots with other capybara artifacts
  # (Rails screenshots path is not configurable https://github.com/rails/rails/blob/49baf092439fc74fc3377b12e3334c3dd9d0752f/actionpack/lib/action_dispatch/system_testing/test_helpers/screenshot_helper.rb#L79)
  def absolute_image_path
    Rails.root.join("#{Capybara.save_path}/screenshots/#{image_name}.png")
  end

  # Make failure screenshots compatible with multi-session setup.
  # That's where we use Capybara.last_used_session introduced before.
  def take_screenshot
    return super unless Capybara.last_used_session

    Capybara.using_session(Capybara.last_used_session) { super }
  end
end

RSpec.configure do |config|
  config.include BetterRailsSystemTests, type: :system

  # Make urls in mailers contain the correct server host.
  # It's required for testing links in emails (e.g., via capybara-email).
  config.around(:each, type: :system) do |ex|
    was_host = Rails.application.default_url_options[:host]
    Rails.application.default_url_options[:host] = Capybara.server_host
    ex.run
    Rails.application.default_url_options[:host] = was_host
  end

  # Make sure this hook runs before others
  config.prepend_before(:each, type: :system) do
    # Use JS driver always
    driven_by Capybara.javascript_driver
  end
end

precompile_assets.rb

This file is responsible for precompiling assets before running system tests (I’m not going to paste it here in full, just the most interesting parts):

RSpec.configure do |config|
  # Skip assets precompilcation if we exclude system specs.
  # For example, you can run all non-system tests via the following command:
  #
  #    rspec --tag ~type:system
  #
  # In this case, we don't need to precompile assets.
  next if config.filter.opposite.rules[:type] == "system" || config.exclude_pattern.match?(%r{spec/system})

  config.before(:suite) do
    # We can use webpack-dev-server for tests, too!
    # Useful if you working on a frontend code fixes and want to verify them via system tests.
    if Webpacker.dev_server.running?
      $stdout.puts "\n⚙️  Webpack dev server is running! Skip assets compilation.\n"
      next
    else
      $stdout.puts "\n:turtle:  Precompiling assets.\n"

      # The code to run webpacker:compile Rake task
      # ...
    end
  end
end

Why precompiling assets manually if Rails can do that for you automatically? The problem is that Rails precompiles assets lazily (i.e., the first time you request an asset), that could make the first test example running much slower and even encounter random timeout exceptions.

To use Webpack dev server for tests, you need to add a dev_server configuration for test environment to your webpacker.yml and run it via RAILS_ENV=test ./bin/webpack-dev-server .

Another thing I want to pay attention to is the ability to use a Webpack dev server for system tests. That’s really useful when you’re working hard on frontend code refactoring: you can pause a test, open a browser, edit frontend code and see it hot reloaded!

Dockerizing system tests

Let’s move our configuration to the next level and make it compatible with our Docker development environment . The dockerized version of the test setup above lives in the  default branch of the AnyCable Rails Demo repository, feel free to check it out, but we’re going to cover all the interesting bits below.

The main difference in a Docker setup is that we run a browser instance in a separate container. It’s possible to add Chrome to your base Rails image or, probably, even use a host machine browser from a container (like it could be done with Selenium and ChromeDriver ). But, in my opinion, defining a dedicated browser service to  docker-compose.yml is a proper Docker way of doing this.

Currently, I’m using a Chrome Docker image from browserless.io . It comes with a handy Debug Viewer, which allows you to debug headless browser sessions (wait for short video at the end of the article):

services:
  # ...
  chrome:
    image: browserless/chrome:1.31-chrome-stable
    ports:
      - "3333:3333"
    environment:
      # By default, it uses 3000, which is typically used by Rails.
      PORT: 3333
      # Set connection timeout to avoid timeout exception during debugging
      # https://docs.browserless.io/docs/docker.html#connection-timeout
      CONNECTION_TIMEOUT: 600000

Add CHROME_URL: http://chrome:3333 to your Rails service environment and run Chrome in the background:

docker-compose up -d chrome

Now we need to configure Cuprite to work with a remote browser if its URL is provided:

# cuprite_setup.rb

# Parse URL
# NOTE: REMOTE_CHROME_HOST should be added to Webmock/VCR allowlist if you use any of those.
REMOTE_CHROME_URL = ENV["CHROME_URL"]
REMOTE_CHROME_HOST, REMOTE_CHROME_PORT =
  if REMOTE_CHROME_URL
    URI.parse(REMOTE_CHROME_URL).yield_self do |uri|
      [uri.host, uri.port]
    end
  end

# Check whether the remote chrome is running.
remote_chrome =
  begin
    if REMOTE_CHROME_URL.nil?
      false
    else
      Socket.tcp(REMOTE_CHROME_HOST, REMOTE_CHROME_PORT, connect_timeout: 1).close
      true
    end
  rescue Errno::ECONNREFUSED, Errno::EHOSTUNREACH, SocketError
    false
  end

remote_options = remote_chrome ? { url: REMOTE_CHROME_URL } : {}

This configuration above assumes that a user wants to use a locally installed Chrome if CHROME_URL is not set or browser is not responding.

We do that to make our configuration backward-compatible with the local one (we usually do not force everyone to use Docker for development; we let Docker deniers suffer with their unique local setups :smiling_imp:).

Our driver registration now looks like this:

# spec/system/support/cuprite_setup.rb

Capybara.register_driver(:cuprite) do |app|
  Capybara::Cuprite::Driver.new(
    app,
    **{
      window_size: [1200, 800],
      browser_options: remote_chrome ? { "no-sandbox" => nil } : {},
      inspector: true
    }.merge(remote_options)
  )
end

We also need to update our #debug helper to print a Debug Viewer URL instead of trying to open a browser:

module CupriteHelpers
  # ...

  def debug(binding = nil)
    $stdout.puts ":mag_right: Open Chrome inspector at http://localhost:3333"
    return binding.pry if binding

    page.driver.pause
  end
end

Since a browser is running on a different “machine”, it should know how to reach the test server (which no longer listens on localhost ).

For that, we need to configure Capybara server host:

# spec/system/support/capybara_setup.rb

# Make server accessible from the outside world
Capybara.server_host = "0.0.0.0"
# Use a hostname that could be resolved in the internal Docker network
# NOTE: Rails overrides Capybara.app_host in Rails <6.1, so we have
# to store it differently
CAPYBARA_APP_HOST = `hostname`.strip&.downcase || "0.0.0.0"
# In Rails 6.1+ the following line should be enough
# Capybara.app_host = "http://#{`hostname`.strip&.downcase || "0.0.0.0"}"

Finally, let’s add a couple of tweaks to our better_rails_system_tests.rb .

First, let’s make screenshot notices clickable in VS Code (Docker absolute paths are different from a host system):

# spec/system/support/better_rails_system_tests.rb

module BetterRailsSystemTests
  # ...

  # Use relative path in screenshot message
  def image_path
    absolute_image_path.relative_path_from(Rails.root).to_s
  end
end

Secondly, make sure tests use the correct server host (this has been fixed in Rails 6.1 ):

# spec/system/support/better_rails_system_tests.rb

config.prepend_before(:each, type: :system) do
  # Rails sets host to `127.0.0.1` for every test by default.
  # That won't work with a remote browser.
  host! CAPYBARA_APP_HOST
  # Use JS driver always
  driven_by Capybara.javascript_driver
end

In too Dip

If you use Dip to manage your dockerized development environment (and I highly recommend you do, it gives you yield the power of containers without the overhead of remembering all Docker CLI Commands), you can avoid launching chrome service manually by adding custom commands to your dip.yml and an additional service definition to your docker-compose.yml :

# docker-compose.yml

# Separate definition for system tests to add Chrome as a dependency
rspec_system:
  <<: *backend
  depends_on:
    <<: *backend_depends_on
    chrome:
      condition: service_started

# dip.yml
rspec:
  description: Run Rails unit tests
  service: rails
  environment:
    RAILS_ENV: test
  command: bundle exec rspec --exclude-pattern spec/system/**/*_spec.rb
  subcommands:
    system:
      description: Run Rails system tests
      service: rspec_system
      command: bundle exec rspec --pattern spec/system/**/*_spec.rb

Now, to run system tests I use the following command:

dip rspec system

And that’s it!

As a final note, let me show you how debugging with the Debug Viewer from Browserless.io Docker image looks like:

If you want to set up rock-solid development practices in your engineering team—regardless of the technical stack—feel free todrop us a line. Helping companies to  improve their engineering culture is one of the favorite things we do!