Cucumber test automation

These past few weeks I have been involved in updating my Visual Editor test plan, as well as working as writing a test case to test the visual editor headings.  I have also started to train one of our QA volunteers by running a pair programing session with her. This time I had the role of the teacher rather than the trainee, and I am pleased that I have gained enough knowledge that I was able to do an adequate job explaining our tools and processes to her.  The way I ran this session with her is that I  worked on a test case I needed to get done, and as I worked I showed her what the steps were. and how to debug it.  That seemed to work well for her, though in the next session, we are planning on her working on her own test case.

In this blog entry, I wanted to explain more about the FOSS automation tools we are using to do browser automation testing here at Wikimedia.  These include Cucumber, Ruby, and Selenium with Watir web driver.  We also use Page Object Design Pattern, which is a testing design pattern used to improve the maintenance of  automation tests.  I am excited to learn these tools not only because are they easy to learn and use, but also because they are up-and-coming browser automation testings tools starting to be seen more and more in the software testing world.  Another plus is that when you mention you are working with Cucumber, you are sure to get a good reaction from others.  Usually the reaction is something like this: "Huh?  What is that? Never heard of it".  In my daughter's case, her reaction was to insist to me that Cucumber is a vegetable and cannot be anything else but something you can eat!
 
One of Cucumber's strengths is that it is a communication tool as well as an automated testing tool.
Cucumber lets software development teams describe how software should behave in plain text.  The text is written in a business-readable domain-specific language and thus can be used for written specifications, automated tests and as a development-aid - all rolled into one.  Cucumber works with Ruby, Java, .NET, Flex or web applications written in any language.  In our case, we are running it with Ruby.



Writing a test case in Cucumber is really as simple as writing a test case in plain english, except for the fact that it must be put into Cucumber's Given, When, and Then syntax.  Here is an explanation and example of Cucumber taken directly from our Quality Assurance/browser testing document:

Cucumber demands specifications in a particular form, Given/When/Then:

• A Given statement should only ever mention starting conditions.
• A When statement should always be some sort of verb, or action.
• A Then statement always has the words "should" or "should not", or equivalent.

Feature: My Feature
  Scenario: Testing some aspect of My Feature
    Given <some initial condition>
    When <I take some action>
    Then <I should observe some result in the browser>

Any Given/When/Then statement may also have an arbitrary number of "And" clauses:

Scenario: Testing some complicated aspect of My Feature
  Given <some initial condition>
    And <some other condition>
  When <I take some action>
    And <I take some other action>
    And <I take yet another action>
  Then <I should observe some result in the browser>
    And <I should not observe some other result>

In order to write a cucumber test for the feature Search which should work in any given wiki, you should first write down how it is expected to behave in "plain english". It would like like this:

On any random page in any wiki, type "main" into the search box.  
A list of results should appear, and "Main Page" should be the top result.  
On another random page in any wiki, type "ma" into the search box and click the Search icon.  
I should then be on a page with a set of search results, and "Main Page" should be one of the results.

Now, here it is after being cucumberized:

Feature: Search

Scenario: Search suggestions
    Given I am at random page
    When I search for: main
    Then a list of suggested pages should appear
       And Main Page should be the first result

 Scenario: Fill in search term and click search
    Given I am at random page
    When I search for: ma
        And I click the Search button
    Then I should land on Search Results page 

See how simple that is?  Easy Peasy :)

The next step would be to run this cucumber script from the command line, where it will detail the steps you need to take to put the test case into Ruby.  More on this in my next entry.