Switching to Headless Chrome for Rails System Tests

Introduction

I recently switched a Rails 5.1 application’s system tests from Capybara, Poltergeist, and PhantomJS, to Capybara, Selenium, and headless Chrome. We run the development and test environments of the application on a Vagrant box running Ubuntu 16.04 server.

With the release of headless Chrome, PhantomJS is no longer being developed or maintained. It also used a different browser engine that the major browsers, and I was noticing that some test cases didn’t run exactly like they would in a real browser. So when I saw that there was a pull request in to Rails to change to Selenium and headless Chrome, I thought it was time to try it myself.

Installation

The first thing I did was to install Google Chrome from a Google repository, so it’s easy to get updates:

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add
echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt-get update
sudo apt-get install google-chrome-stable

Configuration

(Note that this configuration will change if/when Rails comes with configuration for headless Chrome.)

I set up the configuration in test/application_system_test_case.rb. The Rails 5.1 driven_by allows some options to be set, but I couldn’t figure out how to set the options I needed, so I registered a separate driver for headless Chrome:

  Capybara.register_driver(:headless_chrome) do |app|
    capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
      # This makes logs available, but doesn't cause them to appear
      # in real time on the console
      loggingPrefs: {
        browser: "ALL",
        client: "ALL",
        driver: "ALL",
        server: "ALL"
      }
    )

    options = Selenium::WebDriver::Chrome::Options.new
    options.add_argument("window-size=1400,1200")
    options.add_argument("headless")
    options.add_argument("disable-gpu")

    Capybara::Selenium::Driver.new(
      app,
      browser: :chrome,
      desired_capabilities: capabilities,
      options: options
    )
  end

  driven_by :headless_chrome

The arguments for "headless" and "disable-gpu" were necessary to make testing with headless Chrome work. I had to set the window size, because the default window size caused my application’s menu to collapse to a mobile device menu (I’m using Bootstrap 4).

Upgrade Capybara

My Gemfile had the version of Capybara locked down. I found that a lot of problems went away simply by taking away the version constraint in the Gemfile and letting bundle upgrade capybara --conservative do its thing.

fill_in Doesn’t Fire Change Event

I had some JavaScript that ran when input fields changed, via the changed event. I had to add a newline to the end of the input text for force the change event to fire:

fill_in "Fragment", with: "Outage B\n"

fill_in Date Time Field With “” Doesn’t Work

Filling in a date time field with “” to clear it worked with Poltergeist/PhantomJS. With Selenium and headless Chrome it gives:

Selenium::WebDriver::Error::InvalidElementStateError: invalid element state: Element must be user-editable in order to clear it.

So I changed the places where I had to clear a date or date time field to this instead:

find_field("Outages Before").send_keys :delete

fill_in Date Field With Date Works Differently

Selenium and headless Chrome seem to process fill_in of a date field more like what the user would experience. My tests that worked with Poltergeist and PhantomJS didn’t work with headless Chrome and Selenium, although part of the problem may have been with the change event triggers I had on the date fields.

I got the tests to work by (mysteriously) entering the date as “12312017”, in other words, in the date order used by only one country in the whole known universe. I still had to assert against dates in the format “yyyy-mm-dd”.

I also discovered that PhantomJS and/or Poltergeist was more forgiving about date formats in asserts, so I had to change a bunch of asserts where I had used the “dd/mm/yyyy” format.

Alerts

Alerts aren’t automatically dismissed, so I had to go through all my tests and put an assert_accept block around actions like deletes, like this:

accept_alert do
  click_link "Delete"
end

Empty divs

Selenium and Chrome seem to treat an empty div as if it’s not visible. I had to change some selectors that were looking for an empty div to something like this:

assert_selector ".test-home-page", visible: :any

No Browser Logs in Real Time (or by Default)

Browser logs don’t appear by default, and don’t appear in real time. To show what was in the browser log, I had to put the following in the test script:

puts page.driver.manage.get_log(:browser)

To even have the browser log available at all, I had to set up the configuration as described at the beginning of this post.

Performance

My system tests take about 50 % longer with headless Chrome, compared to PhantomJS.

Tips for System Testing With Capybara

One of the great challenges of system testing applications is the fact that there are two (or more) independent processes: One to run the test script. The other to run the browser. The actions in the test script and the actions as performed by the browser and the back end, are not necessarily synchronized the way they are in unit or controller tests.

This can lead to tests the fail sometimes but not others. They may fail one in ten runs, or they may work on development machines, but fail on the continuous integration platform, or vice versa. Some say that unreliable tests are one of the key things that lead people away from automated testing.

I’ve struggled with unreliable tests on Rails applications using Capybara. Here are some of the solutions I’ve used successfully to make tests more reliable.

Spinner

I like this one because it’s what I should be doing for my users anyway. Anytime something is happening in the background that might take some time, the page should put up a spinner – something that shows that the user should wait.

There are lots of spinners available on-line. Some examples are:

• https://projects.lukehaas.me/css-loaders/
• http://cssload.net/en/spinners
• https://www.pexels.com/blog/css-only-loaders/ (the example below is based on the snake spinner from this link)

The spinner chosen is not important from the point of view of the test. The tests work on the presence or absence of the spinner element. The basic idea is this:

1. The Capybara test script initiates a JavaScript action that will take some time
2. The test browser begins running the JavaScript. The first thing the JavaScript does is put up the spinner by un-hiding the spinner element or adding the spinner element to the DOM. The spinner element should have a distinctive id or class

3. The Capybara script waits for the spinner to go away by doing something like:

   assert_no_selector ".spinner"

4. On completing the AJAX request, the JavaScript hides the spinner element or removes it from the DOM

Here is one example of this technique from a Rails 5.1 application with Turbolinks. All it needs is this JavaScript in the application:

$(document).on('turbolinks:load', function(e) {
  $('form.js-submit-on-change').change(function(event) {
    $("body").prepend('<div class="spinner"></div>');
    $(this).submit();
  }).on("ajax:success", function(e) {
  }).on("ajax:error", function(e) {
  }).on("ajax:complete", function(e) {
    $(".spinner").remove();
  });
});

And then this in the test case to ensure that the test is synchronized with the browser:

assert_no_selector ".spinner"

(This JavaScript works for the case where I had forms that I wanted to automatically submit on any change to any of their fields. A traditional form that has a submit button would want to catch the submit event for the form.)

For completeness, here’s some SASS to put in assets/stylesheets/spinner.scss to actually show a spinner:

.spinner {
  @extend .centered-in-window;
  z-index: 1;
  height: 64px;
  width: 64px;
  animation: rotate 1s infinite linear;
  -webkit-animation: rotate 1s infinite linear;
  border: 8px solid #fff;
  border-right-color: black;
  border-radius: 50%;
}

@keyframes rotate {
  0% {
    transform: rotate(0deg);
  }

  100% {
    transform: rotate(360deg);
  }
}

@-webkit-keyframes rotate {
  0% {
    -webkit-transform: rotate(0deg);
  }

  100% {
    -webkit-transform: rotate(360deg);
  }
}

.centered-in-window {
  position: fixed;
  top: 50%;
  left: 50%;
  /* bring your own prefixes */
  transform: translate(-50%, -50%);
}

Don’t forget to add @import "spinner"; to your assets/stylesheets/application.scss file.

Only the Back End Changes

When the user changes, say, a check box or a drop down menu (a select tag), nothing happens on the browser side other than the change the user made. But often, on the back end something is added, changed, or removed from the database. I tried to use Rails’ assert_difference to test that the right thing happened, but more often than not it would fail, because by the time the database was updated, Capybara had already finished executing the assert_difference.

Because of that, my tests were full of examples like this:

assert_difference "CisOutage.count" do
  click_on "Save"
  sleep 2
end

Each time I put in a sleep 2 in my test cases, I was making the test take almost two seconds longer than it needed to. If I didn’t have the sleep 2 in the test, the test would fail, because the CisOutage record wouldn’t have been saved in the database by the time the test case evaluated the CisOutage.count.

To fix this, I wrote my own assert_difference for system tests, shamelessly stealing from the Rails source and a few items from Capybara. Like the Rails assert_difference, it runs the expressions to get the “before” value, executes the block, then runs the expressions again to get the after values. Unlike the Rails version, if any of the expressions fail to produce the desired result, it sleeps for a tenth of a second, then retries the expressions, until all the expressions produce the desired result, or two seconds pass:

##
# Check for a difference in `expression`, but repeat the check until it's
# true, or two seconds pass. Taken from Rails source and leveraging
# some Capybara stuff.
def assert_difference(expression, difference = 1, message = nil, &block)
  expressions = Array(expression)

  exps = expressions.map do |e|
    e.respond_to?(:call) ? e : -> { eval(e, block.binding) }
  end
  before = exps.map(&:call)
  after = []

  retval = yield

  start_time = Capybara::Helpers.monotonic_time
  loop do
    after = exps.map(&:call)
    break if before.zip(after).all? { |(b, a)| a == b + difference } ||
             start_time + 2 < Capybara::Helpers.monotonic_time
    sleep 0.1
  end

  expressions.zip(after).each_with_index do |(code, a), i|
    error  = "#{code.inspect} didn't change by #{difference}"
    error  = "#{message}.\n#{error}" if message
    assert_equal(before[i] + difference, a, error)
  end

  retval
end

For Rails 5.1, I put this in the ApplicationSystemTestCase class in app/test/application_system_test_case.rb. For Rails 5.0 and 4.x, I would have to put it somewhere else.

This code could be improved in many ways, but it was good enough to allow me to remove 40 seconds worth of arbitrary sleep statements in my system test cases.

Now my test case looks like this:

assert_difference "CisOutage.count" do
  click_on "Save"
end

And the test typically waits at most a couple of tenths of a second before succeeding.

Only the Order Changes

To test “latest first/earliest first” buttons, and similar actions, I had to go beyond a simple assert_text on the page. At first I tried getting all the list items within the list I was interested in, like this:

click_link "Oldest First"
notes = all("li.note")
within(notes[0]) do
  assert_text "Note B"
  assert_text "1 day ago"
  assert_no_link "Edit"
  assert_text "Can Edit CIs/Outages"
end

This would fail if Capybara happened to grab the list before the back end had replied to the browser. More often than not, I found Capybara would populate the notes variable with nodes from the page before the back end responded with the re-ordered list. In the best case, I would simply get test failures. In other cases, Capybara would actually throw errors (the dreaded “stale node” error). This is because once the browser gets the response from the back end, the nodes in notes will no longer be on the current page of the browser.

My fix was to use a more specific selector to take advantage of Capybara’s waiting behaviour. This method works for one specific case I had:

  def assert_synchronized(text, ordinal = 0)
    assert_selector "li.note:nth-of-type(#{ordinal + 1}) .note-body", text: text
  end

This finds the (ordinal + 1)th list item, and then if it has class note, finds all nodes with class note-body with the li, and checks to see if they have the desired text.

When I put this assert_synchronized call somewhere in the test case, Capybara checks that the first item in the list has the text I expect, and will do its standard waiting behaviour before proceeding with the rest of the test.

I was using 0-based indexes in the rest of the code, but CSS selectors are 1-based, which is why the ordinal + 1. Also, in a more general case I’d have to make sure I was in the right list, but on this page there was only one list. The actual selector would be different for every page or case. The above is just one example.

Here’s how I fixed the above test case:

click_link "Oldest First"
assert_synchronized("Note B")
notes = all("li.note")
within(notes[0]) do
  assert_text "Note B"
  assert_text "1 day ago"
  assert_no_link "Edit"
  assert_text "Can Edit CIs/Outages"
end

I ran into a couple of challenges with this approach:

• The nth-of-type(x) selector is literally on the type, AKA HTML tag, and not the other selectors. In other words, something like li.note:nth-of-type(1) gets the first li regardless of its classes, then checks to see if the li has class note. So if the list is mixed and the first li does not have class note, the selector returns nothing
• I had a mixed list as described above, where not every li had the same class. To work around the problem, I was looking at the second item in the list. But the list only had three items in it, so looking at the second item wasn’t synchronizing Capybara with the back end. It took me a while before the light bulb when on and I realized why my test was getting out of sync