Tuesday, 31 October 2017

Switching to Headless Chrome for Rails System Tests

Introduction

I recently switched a Rails 5.1 application’s system tests from Capybara, Poltergeist, and PhantomJS, to Capybara, Selenium, and headless Chrome. We run the development and test environments of the application on a Vagrant box running Ubuntu 16.04 server.

With the release of headless Chrome, PhantomJS is no longer being developed or maintained. It also used a different browser engine that the major browsers, and I was noticing that some test cases didn’t run exactly like they would in a real browser. So when I saw that there was a pull request in to Rails to change to Selenium and headless Chrome, I thought it was time to try it myself.

Installation

The first thing I did was to install Google Chrome from a Google repository, so it’s easy to get updates:

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add
echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt-get update
sudo apt-get install google-chrome-stable

Configuration

(Note that this configuration will change if/when Rails comes with configuration for headless Chrome.)

I set up the configuration in test/application_system_test_case.rb. The Rails 5.1 driven_by allows some options to be set, but I couldn’t figure out how to set the options I needed, so I registered a separate driver for headless Chrome:

  Capybara.register_driver(:headless_chrome) do |app|
    capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
      # This makes logs available, but doesn't cause them to appear
      # in real time on the console
      loggingPrefs: {
        browser: "ALL",
        client: "ALL",
        driver: "ALL",
        server: "ALL"
      }
    )

    options = Selenium::WebDriver::Chrome::Options.new
    options.add_argument("window-size=1400,1200")
    options.add_argument("headless")
    options.add_argument("disable-gpu")

    Capybara::Selenium::Driver.new(
      app,
      browser: :chrome,
      desired_capabilities: capabilities,
      options: options
    )
  end

  driven_by :headless_chrome

The arguments for "headless" and "disable-gpu" were necessary to make testing with headless Chrome work. I had to set the window size, because the default window size caused my application’s menu to collapse to a mobile device menu (I’m using Bootstrap 4).

Upgrade Capybara

My Gemfile had the version of Capybara locked down. I found that a lot of problems went away simply by taking away the version constraint in the Gemfile and letting bundle upgrade capybara --conservative do its thing.

fill_in Doesn’t Fire Change Event

I had some JavaScript that ran when input fields changed, via the changed event. I had to add a newline to the end of the input text for force the change event to fire:

fill_in "Fragment", with: "Outage B\n"

fill_in Date Time Field With “” Doesn’t Work

Filling in a date time field with “” to clear it worked with Poltergeist/PhantomJS. With Selenium and headless Chrome it gives:

Selenium::WebDriver::Error::InvalidElementStateError: invalid element state: Element must be user-editable in order to clear it.

So I changed the places where I had to clear a date or date time field to this instead:

find_field("Outages Before").send_keys :delete

fill_in Date Field With Date Works Differently

Selenium and headless Chrome seem to process fill_in of a date field more like what the user would experience. My tests that worked with Poltergeist and PhantomJS didn’t work with headless Chrome and Selenium, although part of the problem may have been with the change event triggers I had on the date fields.

I got the tests to work by (mysteriously) entering the date as “12312017”, in other words, in the date order used by only one country in the whole known universe. I still had to assert against dates in the format “yyyy-mm-dd”.

I also discovered that PhantomJS and/or Poltergeist was more forgiving about date formats in asserts, so I had to change a bunch of asserts where I had used the “dd/mm/yyyy” format.

Alerts

Alerts aren’t automatically dismissed, so I had to go through all my tests and put an assert_accept block around actions like deletes, like this:

accept_alert do
  click_link "Delete"
end

Empty divs

Selenium and Chrome seem to treat an empty div as if it’s not visible. I had to change some selectors that were looking for an empty div to something like this:

assert_selector ".test-home-page", visible: :any

No Browser Logs in Real Time (or by Default)

Browser logs don’t appear by default, and don’t appear in real time. To show what was in the browser log, I had to put the following in the test script:

puts page.driver.manage.get_log(:browser)

To even have the browser log available at all, I had to set up the configuration as described at the beginning of this post.

Performance

My system tests take about 50 % longer with headless Chrome, compared to PhantomJS.

Saturday, 14 October 2017

Tips for System Testing With Capybara

One of the great challenges of system testing applications is the fact that there are two (or more) independent processes: One to run the test script. The other to run the browser. The actions in the test script and the actions as performed by the browser and the back end, are not necessarily synchronized the way they are in unit or controller tests.

This can lead to tests the fail sometimes but not others. They may fail one in ten runs, or they may work on development machines, but fail on the continuous integration platform, or vice versa. Some say that unreliable tests are one of the key things that lead people away from automated testing.

I’ve struggled with unreliable tests on Rails applications using Capybara. Here are some of the solutions I’ve used successfully to make tests more reliable.

Spinner

I like this one because it’s what I should be doing for my users anyway. Anytime something is happening in the background that might take some time, the page should put up a spinner – something that shows that the user should wait.

There are lots of spinners available on-line. Some examples are:

The spinner chosen is not important from the point of view of the test. The tests work on the presence or absence of the spinner element. The basic idea is this:

  1. The Capybara test script initiates a JavaScript action that will take some time
  2. The test browser begins running the JavaScript. The first thing the JavaScript does is put up the spinner by un-hiding the spinner element or adding the spinner element to the DOM. The spinner element should have a distinctive id or class
  3. The Capybara script waits for the spinner to go away by doing something like:

    assert_no_selector ".spinner"
  4. On completing the AJAX request, the JavaScript hides the spinner element or removes it from the DOM

Here is one example of this technique from a Rails 5.1 application with Turbolinks. All it needs is this JavaScript in the application:

$(document).on('turbolinks:load', function(e) {
  $('form.js-submit-on-change').change(function(event) {
    $("body").prepend('<div class="spinner"></div>');
    $(this).submit();
  }).on("ajax:success", function(e) {
  }).on("ajax:error", function(e) {
  }).on("ajax:complete", function(e) {
    $(".spinner").remove();
  });
});

And then this in the test case to ensure that the test is synchronized with the browser:

assert_no_selector ".spinner"

(This JavaScript works for the case where I had forms that I wanted to automatically submit on any change to any of their fields. A traditional form that has a submit button would want to catch the submit event for the form.)

For completeness, here’s some SASS to put in assets/stylesheets/spinner.scss to actually show a spinner:

.spinner {
  @extend .centered-in-window;
  z-index: 1;
  height: 64px;
  width: 64px;
  animation: rotate 1s infinite linear;
  -webkit-animation: rotate 1s infinite linear;
  border: 8px solid #fff;
  border-right-color: black;
  border-radius: 50%;
}

@keyframes rotate {
  0% {
    transform: rotate(0deg);
  }

  100% {
    transform: rotate(360deg);
  }
}

@-webkit-keyframes rotate {
  0% {
    -webkit-transform: rotate(0deg);
  }

  100% {
    -webkit-transform: rotate(360deg);
  }
}

.centered-in-window {
  position: fixed;
  top: 50%;
  left: 50%;
  /* bring your own prefixes */
  transform: translate(-50%, -50%);
}

Don’t forget to add @import "spinner"; to your assets/stylesheets/application.scss file.

Only the Back End Changes

When the user changes, say, a check box or a drop down menu (a select tag), nothing happens on the browser side other than the change the user made. But often, on the back end something is added, changed, or removed from the database. I tried to use Rails’ assert_difference to test that the right thing happened, but more often than not it would fail, because by the time the database was updated, Capybara had already finished executing the assert_difference.

Because of that, my tests were full of examples like this:

assert_difference "CisOutage.count" do
  click_on "Save"
  sleep 2
end

Each time I put in a sleep 2 in my test cases, I was making the test take almost two seconds longer than it needed to. If I didn’t have the sleep 2 in the test, the test would fail, because the CisOutage record wouldn’t have been saved in the database by the time the test case evaluated the CisOutage.count.

To fix this, I wrote my own assert_difference for system tests, shamelessly stealing from the Rails source and a few items from Capybara. Like the Rails assert_difference, it runs the expressions to get the “before” value, executes the block, then runs the expressions again to get the after values. Unlike the Rails version, if any of the expressions fail to produce the desired result, it sleeps for a tenth of a second, then retries the expressions, until all the expressions produce the desired result, or two seconds pass:

##
# Check for a difference in `expression`, but repeat the check until it's
# true, or two seconds pass. Taken from Rails source and leveraging
# some Capybara stuff.
def assert_difference(expression, difference = 1, message = nil, &block)
  expressions = Array(expression)

  exps = expressions.map do |e|
    e.respond_to?(:call) ? e : -> { eval(e, block.binding) }
  end
  before = exps.map(&:call)
  after = []

  retval = yield

  start_time = Capybara::Helpers.monotonic_time
  loop do
    after = exps.map(&:call)
    break if before.zip(after).all? { |(b, a)| a == b + difference } ||
             start_time + 2 < Capybara::Helpers.monotonic_time
    sleep 0.1
  end

  expressions.zip(after).each_with_index do |(code, a), i|
    error  = "#{code.inspect} didn't change by #{difference}"
    error  = "#{message}.\n#{error}" if message
    assert_equal(before[i] + difference, a, error)
  end

  retval
end

For Rails 5.1, I put this in the ApplicationSystemTestCase class in app/test/application_system_test_case.rb. For Rails 5.0 and 4.x, I would have to put it somewhere else.

This code could be improved in many ways, but it was good enough to allow me to remove 40 seconds worth of arbitrary sleep statements in my system test cases.

Now my test case looks like this:

assert_difference "CisOutage.count" do
  click_on "Save"
end

And the test typically waits at most a couple of tenths of a second before succeeding.

Only the Order Changes

To test “latest first/earliest first” buttons, and similar actions, I had to go beyond a simple assert_text on the page. At first I tried getting all the list items within the list I was interested in, like this:

click_link "Oldest First"
notes = all("li.note")
within(notes[0]) do
  assert_text "Note B"
  assert_text "1 day ago"
  assert_no_link "Edit"
  assert_text "Can Edit CIs/Outages"
end

This would fail if Capybara happened to grab the list before the back end had replied to the browser. More often than not, I found Capybara would populate the notes variable with nodes from the page before the back end responded with the re-ordered list. In the best case, I would simply get test failures. In other cases, Capybara would actually throw errors (the dreaded “stale node” error). This is because once the browser gets the response from the back end, the nodes in notes will no longer be on the current page of the browser.

My fix was to use a more specific selector to take advantage of Capybara’s waiting behaviour. This method works for one specific case I had:

  def assert_synchronized(text, ordinal = 0)
    assert_selector "li.note:nth-of-type(#{ordinal + 1}) .note-body", text: text
  end

This finds the (ordinal + 1)th list item, and then if it has class note, finds all nodes with class note-body with the li, and checks to see if they have the desired text.

When I put this assert_synchronized call somewhere in the test case, Capybara checks that the first item in the list has the text I expect, and will do its standard waiting behaviour before proceeding with the rest of the test.

I was using 0-based indexes in the rest of the code, but CSS selectors are 1-based, which is why the ordinal + 1. Also, in a more general case I’d have to make sure I was in the right list, but on this page there was only one list. The actual selector would be different for every page or case. The above is just one example.

Here’s how I fixed the above test case:

click_link "Oldest First"
assert_synchronized("Note B")
notes = all("li.note")
within(notes[0]) do
  assert_text "Note B"
  assert_text "1 day ago"
  assert_no_link "Edit"
  assert_text "Can Edit CIs/Outages"
end

I ran into a couple of challenges with this approach:

  • The nth-of-type(x) selector is literally on the type, AKA HTML tag, and not the other selectors. In other words, something like li.note:nth-of-type(1) gets the first li regardless of its classes, then checks to see if the li has class note. So if the list is mixed and the first li does not have class note, the selector returns nothing
  • I had a mixed list as described above, where not every li had the same class. To work around the problem, I was looking at the second item in the list. But the list only had three items in it, so looking at the second item wasn’t synchronizing Capybara with the back end. It took me a while before the light bulb when on and I realized why my test was getting out of sync

Friday, 23 December 2016

Changing IP of Brother Scanner Under Linux

We had a power outage this week. Some devices on my home network got new IP addresses when the power came back on. One of the devices that got a new IP address was my Brother MFC-9340CDW printer/scanner/fax. The printer was still working fine, but I couldn’t scan.

The tl;dr is to remove the old configuration and set the new one:

sudo brsaneconfig4 -r MFC-9340CDW
sudo brsaneconfig4 -a name=MFC-9340CDW model=MFC-9340CDW ip=192.168.0.124

I got the new IP address from poking buttons on the front panel of the printer.
The name of the device may be different, but the above is what I had from the default setup.
brsaneconfig4 -q gives all the devices supported, and also the last line is the current configuration.
That’s useful to see what the currently configured IP address is, according to the software on the computer.

To figure out what was wrong with simple-scan, I did this in a terminal to see debugging output:

simple-scan -d

Another useful command is:

brsaneconfig4 -p

which runs ping on the scanner. However, in my case another device had the scanner’s old IP address,
so the ping appeared to be working fine.

Friday, 18 November 2016

Disabling Warnings and Autocorrect in Rubocop

I finally found how to disable Rubocop messages and auto-correction on a file or individual line basis.

I have Atom set up to run Rubocop and auto-correct my files on save. Most of the time this is very handy. I especially like how most of the time it indents my code to the standard. But I was struggling to debug some test cases, and I wanted to use Capybara’s save_and_open_page to see what Capybara was actually looking at. When I saved the file, Rubocop and Atom helpfully deleted the line before I could even run the test case.

But then I discovered this:

save_and_open_page # rubocop:disable Lint/Debugger

Problem solved.

Since the Lint/Debugger cop is arguably not applicable to your test files, I sometimes put this at the top of a test file:

# rubocop:disable Lint/Debugger

In the above case, I could turn Rubocop back on if I needed it with:

# rubocop:enable Lint/Debugger

This is described in the manual at http://rubocop.readthedocs.io/en/latest/configuration/#disabling-cops-within-source-code, but I had trouble finding that section with Google.

Sunday, 28 August 2016

Enterprise Challenges to Continuous Delivery

On one of my recent projects I noticed a challenge to a continuous delivery (CD) approach in the enterprise that I haven’t seen mentioned: Software that requires a paid license.

Most CD approaches are based on easily creating instances of a virtual machine. Software with a paid license often has mechanisms, like the MAC address of the first network interface has to be registered with the software vendor’s license server, to prevent automatically creating multiple instances of a virtual machine. Or it may have a license compliance tool that will get very confused by instances of virtual machines appearing and disappearing rapidly.

It’s probably not impossible to do CD with your paid software. Obviously, the easiest way to deal with the problem is to use software that’s completely free to use. If you can’t do that, you’ll need to understand how the license restrictions work, and then tailor the continuous deployment approach around those restrictions. Not easy, but probably doable.

Configuring Applications

I once had experience with a browser-based application that would show pages that were links to another application. Since the URL of the other application would change depending on the environment (development, test, staging, production, etc.), the developers decided to put the URL in a field in a row in a configuration table in the database.

So far, so good. Per-environment configuration needs to be changeable. But it turned out that putting configuration in the database had some challenges. Configuration in the database is less convenient to put under revision control and to deploy with automated tools. So we didn’t have the configuration as part of an automated build tool, and it was a bureaucratic nightmare each time we had to change the URL. (Simple files are easier to integrate with revision control and automated deployment tools.)

But what really caused problems was that the database contained more than just the protocol, host, domain, and port – the stuff that would change for each environment. It included a template for query parameters as well. So the line in the configuration table looked something like this:

https://otherapplication.example.com:20400/search-page?parm1=%1,parm2=%2

The application would take the query string (?parm1=%1,parm2=%2), and fill in the placeholders (%1, %2) with values. The problem was, every time the requirements expanded and we needed more parameters, the configuration string had to change for every environment.

The parameters weren’t part of the environment set-up, so they never should have been put in the configuration table. When you’re parametrizing an application configuration, make sure you put only things that change per environment into the environment configuration parameters.

Unrelated to the above point, but important to note, is that we also found it very inconvenient that the placeholder marker in the URL was a percent sign. Percent signs are URL-encodeable. When we had to e-mail each other with updates to the URL, we were constantly tripped up by our e-mail and spreadsheet programs “helpfully” URL-encoding the query parameters for us, turning %1 into %251, for example.

Tuesday, 19 July 2016

Citrix in a Window on Linux

[Edit: This was not the solution. It worked the first time, but now it isn't working again.]

I was using Citrix Receiver quite successfully on Ubuntu 16.04 and Linux Mint 13 to remotely access my customer's network, but I couldn't make it start in a window. It was coming up in full screen mode. I could minimize the whole Citrix session by doing Ctrl-F2 (to tell Receiver to pass the next key to Linux), then Ctrl-Super-downarrow (Super is the "Windows" key). However, I wanted to be able to watch the Citrix session on one monitor, while I worked on other stuff on the other monitor.

I finally found this blog that told me how to set up the Receiver config files to get Receiver to start in a window: http://blog.eek-a-geek.info/2014/10/citrix-receiver-for-linux-131-on-64-bit.html. What it says is:

Edit "~/.ICAClient/All_Regions.ini", replacing the line "TWIMode=*" to "TWIMode=Off".

Edit "~/.ICAClient/wfclient.ini", adding a line "TWIMode=off" to the "[WFClient]" section, and adding a line "UseFullScreen=True" to the "[Thinwire3.0]" section.