Saturday, 14 October 2017

Tips for System Testing With Capybara

One of the great challenges of system testing applications is the fact that there are two (or more) independent processes: One to run the test script. The other to run the browser. The actions in the test script and the actions as performed by the browser and the back end, are not necessarily synchronized the way they are in unit or controller tests.

This can lead to tests the fail sometimes but not others. They may fail one in ten runs, or they may work on development machines, but fail on the continuous integration platform, or vice versa. Some say that unreliable tests are one of the key things that lead people away from automated testing.

I’ve struggled with unreliable tests on Rails applications using Capybara. Here are some of the solutions I’ve used successfully to make tests more reliable.

Spinner

I like this one because it’s what I should be doing for my users anyway. Anytime something is happening in the background that might take some time, the page should put up a spinner – something that shows that the user should wait.

There are lots of spinners available on-line. Some examples are:

The spinner chosen is not important from the point of view of the test. The tests work on the presence or absence of the spinner element. The basic idea is this:

  1. The Capybara test script initiates a JavaScript action that will take some time
  2. The test browser begins running the JavaScript. The first thing the JavaScript does is put up the spinner by un-hiding the spinner element or adding the spinner element to the DOM. The spinner element should have a distinctive id or class
  3. The Capybara script waits for the spinner to go away by doing something like:

    assert_no_selector ".spinner"
  4. On completing the AJAX request, the JavaScript hides the spinner element or removes it from the DOM

Here is one example of this technique from a Rails 5.1 application with Turbolinks. All it needs is this JavaScript in the application:

$(document).on('turbolinks:load', function(e) {
  $('form.js-submit-on-change').change(function(event) {
    $("body").prepend('<div class="spinner"></div>');
    $(this).submit();
  }).on("ajax:success", function(e) {
  }).on("ajax:error", function(e) {
  }).on("ajax:complete", function(e) {
    $(".spinner").remove();
  });
});

And then this in the test case to ensure that the test is synchronized with the browser:

assert_no_selector ".spinner"

(This JavaScript works for the case where I had forms that I wanted to automatically submit on any change to any of their fields. A traditional form that has a submit button would want to catch the submit event for the form.)

For completeness, here’s some SASS to put in assets/stylesheets/spinner.scss to actually show a spinner:

.spinner {
  @extend .centered-in-window;
  z-index: 1;
  height: 64px;
  width: 64px;
  animation: rotate 1s infinite linear;
  -webkit-animation: rotate 1s infinite linear;
  border: 8px solid #fff;
  border-right-color: black;
  border-radius: 50%;
}

@keyframes rotate {
  0% {
    transform: rotate(0deg);
  }

  100% {
    transform: rotate(360deg);
  }
}

@-webkit-keyframes rotate {
  0% {
    -webkit-transform: rotate(0deg);
  }

  100% {
    -webkit-transform: rotate(360deg);
  }
}

.centered-in-window {
  position: fixed;
  top: 50%;
  left: 50%;
  /* bring your own prefixes */
  transform: translate(-50%, -50%);
}

Don’t forget to add @import "spinner"; to your assets/stylesheets/application.scss file.

Only the Back End Changes

When the user changes, say, a check box or a drop down menu (a select tag), nothing happens on the browser side other than the change the user made. But often, on the back end something is added, changed, or removed from the database. I tried to use Rails’ assert_difference to test that the right thing happened, but more often than not it would fail, because by the time the database was updated, Capybara had already finished executing the assert_difference.

Because of that, my tests were full of examples like this:

assert_difference "CisOutage.count" do
  click_on "Save"
  sleep 2
end

Each time I put in a sleep 2 in my test cases, I was making the test take almost two seconds longer than it needed to. If I didn’t have the sleep 2 in the test, the test would fail, because the CisOutage record wouldn’t have been saved in the database by the time the test case evaluated the CisOutage.count.

To fix this, I wrote my own assert_difference for system tests, shamelessly stealing from the Rails source and a few items from Capybara. Like the Rails assert_difference, it runs the expressions to get the “before” value, executes the block, then runs the expressions again to get the after values. Unlike the Rails version, if any of the expressions fail to produce the desired result, it sleeps for a tenth of a second, then retries the expressions, until all the expressions produce the desired result, or two seconds pass:

##
# Check for a difference in `expression`, but repeat the check until it's
# true, or two seconds pass. Taken from Rails source and leveraging
# some Capybara stuff.
def assert_difference(expression, difference = 1, message = nil, &block)
  expressions = Array(expression)

  exps = expressions.map do |e|
    e.respond_to?(:call) ? e : -> { eval(e, block.binding) }
  end
  before = exps.map(&:call)
  after = []

  retval = yield

  start_time = Capybara::Helpers.monotonic_time
  loop do
    after = exps.map(&:call)
    break if before.zip(after).all? { |(b, a)| a == b + difference } ||
             start_time + 2 < Capybara::Helpers.monotonic_time
    sleep 0.1
  end

  expressions.zip(after).each_with_index do |(code, a), i|
    error  = "#{code.inspect} didn't change by #{difference}"
    error  = "#{message}.\n#{error}" if message
    assert_equal(before[i] + difference, a, error)
  end

  retval
end

For Rails 5.1, I put this in the ApplicationSystemTestCase class in app/test/application_system_test_case.rb. For Rails 5.0 and 4.x, I would have to put it somewhere else.

This code could be improved in many ways, but it was good enough to allow me to remove 40 seconds worth of arbitrary sleep statements in my system test cases.

Now my test case looks like this:

assert_difference "CisOutage.count" do
  click_on "Save"
end

And the test typically waits at most a couple of tenths of a second before succeeding.

Only the Order Changes

To test “latest first/earliest first” buttons, and similar actions, I had to go beyond a simple assert_text on the page. At first I tried getting all the list items within the list I was interested in, like this:

click_link "Oldest First"
notes = all("li.note")
within(notes[0]) do
  assert_text "Note B"
  assert_text "1 day ago"
  assert_no_link "Edit"
  assert_text "Can Edit CIs/Outages"
end

This would fail if Capybara happened to grab the list before the back end had replied to the browser. More often than not, I found Capybara would populate the notes variable with nodes from the page before the back end responded with the re-ordered list. In the best case, I would simply get test failures. In other cases, Capybara would actually throw errors (the dreaded “stale node” error). This is because once the browser gets the response from the back end, the nodes in notes will no longer be on the current page of the browser.

My fix was to use a more specific selector to take advantage of Capybara’s waiting behaviour. This method works for one specific case I had:

  def assert_synchronized(text, ordinal = 0)
    assert_selector "li.note:nth-of-type(#{ordinal + 1}) .note-body", text: text
  end

This finds the (ordinal + 1)th list item, and then if it has class note, finds all nodes with class note-body with the li, and checks to see if they have the desired text.

When I put this assert_synchronized call somewhere in the test case, Capybara checks that the first item in the list has the text I expect, and will do its standard waiting behaviour before proceeding with the rest of the test.

I was using 0-based indexes in the rest of the code, but CSS selectors are 1-based, which is why the ordinal + 1. Also, in a more general case I’d have to make sure I was in the right list, but on this page there was only one list. The actual selector would be different for every page or case. The above is just one example.

Here’s how I fixed the above test case:

click_link "Oldest First"
assert_synchronized("Note B")
notes = all("li.note")
within(notes[0]) do
  assert_text "Note B"
  assert_text "1 day ago"
  assert_no_link "Edit"
  assert_text "Can Edit CIs/Outages"
end

I ran into a couple of challenges with this approach:

  • The nth-of-type(x) selector is literally on the type, AKA HTML tag, and not the other selectors. In other words, something like li.note:nth-of-type(1) gets the first li regardless of its classes, then checks to see if the li has class note. So if the list is mixed and the first li does not have class note, the selector returns nothing
  • I had a mixed list as described above, where not every li had the same class. To work around the problem, I was looking at the second item in the list. But the list only had three items in it, so looking at the second item wasn’t synchronizing Capybara with the back end. It took me a while before the light bulb when on and I realized why my test was getting out of sync

Friday, 23 December 2016

Changing IP of Brother Scanner Under Linux

We had a power outage this week. Some devices on my home network got new IP addresses when the power came back on. One of the devices that got a new IP address was my Brother MFC-9340CDW printer/scanner/fax. The printer was still working fine, but I couldn’t scan.

The tl;dr is to remove the old configuration and set the new one:

sudo brsaneconfig4 -r MFC-9340CDW
sudo brsaneconfig4 -a name=MFC-9340CDW model=MFC-9340CDW ip=192.168.0.124

I got the new IP address from poking buttons on the front panel of the printer.
The name of the device may be different, but the above is what I had from the default setup.
brsaneconfig4 -q gives all the devices supported, and also the last line is the current configuration.
That’s useful to see what the currently configured IP address is, according to the software on the computer.

To figure out what was wrong with simple-scan, I did this in a terminal to see debugging output:

simple-scan -d

Another useful command is:

brsaneconfig4 -p

which runs ping on the scanner. However, in my case another device had the scanner’s old IP address,
so the ping appeared to be working fine.

Friday, 18 November 2016

Disabling Warnings and Autocorrect in Rubocop

I finally found how to disable Rubocop messages and auto-correction on a file or individual line basis.

I have Atom set up to run Rubocop and auto-correct my files on save. Most of the time this is very handy. I especially like how most of the time it indents my code to the standard. But I was struggling to debug some test cases, and I wanted to use Capybara’s save_and_open_page to see what Capybara was actually looking at. When I saved the file, Rubocop and Atom helpfully deleted the line before I could even run the test case.

But then I discovered this:

save_and_open_page # rubocop:disable Lint/Debugger

Problem solved.

Since the Lint/Debugger cop is arguably not applicable to your test files, I sometimes put this at the top of a test file:

# rubocop:disable Lint/Debugger

In the above case, I could turn Rubocop back on if I needed it with:

# rubocop:enable Lint/Debugger

This is described in the manual at http://rubocop.readthedocs.io/en/latest/configuration/#disabling-cops-within-source-code, but I had trouble finding that section with Google.

Sunday, 28 August 2016

Enterprise Challenges to Continuous Delivery

On one of my recent projects I noticed a challenge to a continuous delivery (CD) approach in the enterprise that I haven’t seen mentioned: Software that requires a paid license.

Most CD approaches are based on easily creating instances of a virtual machine. Software with a paid license often has mechanisms, like the MAC address of the first network interface has to be registered with the software vendor’s license server, to prevent automatically creating multiple instances of a virtual machine. Or it may have a license compliance tool that will get very confused by instances of virtual machines appearing and disappearing rapidly.

It’s probably not impossible to do CD with your paid software. Obviously, the easiest way to deal with the problem is to use software that’s completely free to use. If you can’t do that, you’ll need to understand how the license restrictions work, and then tailor the continuous deployment approach around those restrictions. Not easy, but probably doable.

Configuring Applications

I once had experience with a browser-based application that would show pages that were links to another application. Since the URL of the other application would change depending on the environment (development, test, staging, production, etc.), the developers decided to put the URL in a field in a row in a configuration table in the database.

So far, so good. Per-environment configuration needs to be changeable. But it turned out that putting configuration in the database had some challenges. Configuration in the database is less convenient to put under revision control and to deploy with automated tools. So we didn’t have the configuration as part of an automated build tool, and it was a bureaucratic nightmare each time we had to change the URL. (Simple files are easier to integrate with revision control and automated deployment tools.)

But what really caused problems was that the database contained more than just the protocol, host, domain, and port – the stuff that would change for each environment. It included a template for query parameters as well. So the line in the configuration table looked something like this:

https://otherapplication.example.com:20400/search-page?parm1=%1,parm2=%2

The application would take the query string (?parm1=%1,parm2=%2), and fill in the placeholders (%1, %2) with values. The problem was, every time the requirements expanded and we needed more parameters, the configuration string had to change for every environment.

The parameters weren’t part of the environment set-up, so they never should have been put in the configuration table. When you’re parametrizing an application configuration, make sure you put only things that change per environment into the environment configuration parameters.

Unrelated to the above point, but important to note, is that we also found it very inconvenient that the placeholder marker in the URL was a percent sign. Percent signs are URL-encodeable. When we had to e-mail each other with updates to the URL, we were constantly tripped up by our e-mail and spreadsheet programs “helpfully” URL-encoding the query parameters for us, turning %1 into %251, for example.

Tuesday, 19 July 2016

Citrix in a Window on Linux

[Edit: This was not the solution. It worked the first time, but now it isn't working again.]

I was using Citrix Receiver quite successfully on Ubuntu 16.04 and Linux Mint 13 to remotely access my customer's network, but I couldn't make it start in a window. It was coming up in full screen mode. I could minimize the whole Citrix session by doing Ctrl-F2 (to tell Receiver to pass the next key to Linux), then Ctrl-Super-downarrow (Super is the "Windows" key). However, I wanted to be able to watch the Citrix session on one monitor, while I worked on other stuff on the other monitor.

I finally found this blog that told me how to set up the Receiver config files to get Receiver to start in a window: http://blog.eek-a-geek.info/2014/10/citrix-receiver-for-linux-131-on-64-bit.html. What it says is:

Edit "~/.ICAClient/All_Regions.ini", replacing the line "TWIMode=*" to "TWIMode=Off".

Edit "~/.ICAClient/wfclient.ini", adding a line "TWIMode=off" to the "[WFClient]" section, and adding a line "UseFullScreen=True" to the "[Thinwire3.0]" section.

Monday, 30 May 2016

WebEx on Ubuntu 16.04

Java

You need Java installed. I used the Open JRE. Some places on the web say you need the Oracle version, but it works for me with the Open JRE and IcedTea:

sudo apt-get install openjdk-8-jre icedtea-8-plugin

That’s all you need to get the meeting to work, but…

Missing i386 Libraries

But you won’t be able to share screens without a bunch of missing i386 libraries. The WebEx plugin is 32-bit, so you need to install some libraries that aren’t installed by default.

Check to see if you’re missing libraries by going into ~/.webex/ and then into a sub-directory whose name is all digits and underscores. Once there, run:

ldd *.so | grep "not found" | cut -f1 -d' ' | tr -d '\t' | uniq

I got about a dozen missing libraries on a relatively new install of Ubuntu 16.04. You may get different results, depending on what’s been installed on your system since you initially installed Ubuntu 16.04.

I installed the following packages [updated with suggestions from readers] (fewer than a dozen, because some packages pull in multiple libraries as dependencies):

sudo apt-get install libxmu6:i386
sudo apt-get install libgtk2.0-0:i386
sudo apt-get install libpangox-1.0-0:i386
sudo apt-get install libpangoxft-1.0:i386
sudo apt-get install libxtst6:i386
sudo apt-get install libasound2:i386
sudo apt-get install libxv1:i386

If you check again with the above ldd command, the only library you should still be missing is libjawt.so. This library doesn’t seem to be needed.