Regression testing for any web application with TeamCity, Selenium, and JUnit

Maintaining a web application generally involves regression testing, and why not automate what you can? This post describes how to setup regression testing using TeamCity, Selenium, and JUnit. The steps are high-level, and detailed instructions are linked. The approach described here is independent of your web application technology/platform, so it can work with Java, .NET, PHP, etc. You will need to write *some* Java to make it work well, however.

Tools

TeamCity is a continuous integration system that is well suited for managing and running tests and tracking results over time. It has a free license for the Professional version. Selenium is an automated web testing framework that is compatible with all major browsers. Like all OpenQA projects, it has an Apache 2 license. Why not use OpenQA's other product, Bromine, instead of TeamCity? I have experimented with Bromine, but for continuous integration, it doesn't match TeamCity. TeamCity handles source control, builds, triggering, schedules, and unit tests. Bromine does not.

Continuous Integration Server

Choose a machine to run your continuous integration. This should ideally be a server-class machine because your build and test process, and perhaps deployment, will depend on it. Linux is a good choice.

Install TeamCity.

Get the server and a least one agent running.

If your application requires a build step and you want TeamCity to do that for you, setup a TeamCity build configuration. Otherwise this part is optional. Most likely you will have the build configuration check out your application from Subversion or other source control system. TeamCity can run your build for you via Ant, Maven2, and other Java and .NET runners.

You might want to setup a separate build configuration for regression testing, or you can have it run as part of your main build configuration, depending on your needs. If using Ant, you could have a target for regression testing. Be sure to check this Ant script or whatever build script you are using into your source control system.

The regression testing build configuration will need to be able to run JUnit. Ant is a very simple way to do that, because it has JUnit tasks available, or you can easily write your own. Again, this is independent of your web application technology.

Why JUnit? Because that is how we will hook in Selenium. More on that later. For now you can start with an empty JUnit task.

Regression Testing Server

Choose another machine (could be the same one) for automated web regression testing. This one needs to have browsers installed, which Selenium will control. Windows may be your best choice here; it depends on your user base, of course. If you need several platforms, you can setup multiple machines. Virtual machines may also be a consideration for you.

Install Selenium RC (remote control).

Start the Selenium RC server.

There are some good tips for setting up Selenium RC here.

Writing Tests

The easiest way to write a Selenium tests is to record it using Selenium IDE. One limitation is that it only runs in Firefox. If you are not targeting Firefox, I still encourage you to give Selenium IDE a try, because recording scripts is a lot faster than writing every line of code yourself, and it requires less time up front learning the Selenium scripting grammar. Of course, you will still need to tweak tests by adding your own assert statements (tests such as checking for entered text, user task completion indicators, expected or unexpected error conditions, etc.).

Whether you write or record your tests first, get a version of the tests in Java. Again, Selenium IDE makes this easy, because you can switch the test format from HTML to Java by clicking a menu item and copy or save the new format. (I recommend keeping a copy of the default HTML format file around too.) The Java tests should extend the class SeleneseTestCase, which in turn extends the JUnit class TestCase.

You may want to hand-edit your Java test further, to make sure it includes a good Java class name and any additional parameters you might want. I recommend parameterizing your selenium server, port, browser, and URL. This makes it much easier to add or adjust locations for your Selenium RC server(s), target browser(s), and application URL(s). You can do this by extending SeleneseTestCase with your own class MySeleneseTestCase like this and having your test extend MySeleneseTestCase:

package my.seleniumtests;

import com.thoughtworks.selenium.*;

public abstract class MySeleneseTestCase extends SeleneseTestCase {

private static final String DEFAULT_SERVER = "mydefaultserver";
private static final int DEFAULT_PORT = 4444;
private static final String DEFAULT_BROWSER = "*firefox";
private static final String DEFAULT_URL = "http://mydefaulturl";

private String server = System.getProperty("mySeleniumServer");
private String port = System.getProperty("mySeleniumPort");
private String browser = System.getProperty("mySeleniumBrowser");
private String url = System.getProperty("mySeleniumUrl");

public void setUp() throws Exception {
 selenium = new DefaultSelenium(
  server == null ? DEFAULT_SERVER : server,
  port == null ? DEFAULT_PORT : Integer.parseInt(port),
  browser == null ? DEFAULT_BROWSER : browser,
  url == null ? DEFAULT_URL : url);
 selenium.start();
}

}

Adding Tests to Automated System

Collect all your Java tests into a known location, and ideally check them into your source control system so that TeamCity can check them out and build them.

Make sure your regression testing build configuration in TeamCity has the ability to run JUnit. Pass the JUnit runner the names of your tests and any appropriate system properties.

If using Ant and tests that extend MySeleneseTestCase as described above, you can include lines like these in your Ant target that runs the tests:

<junit showoutput="true">
 <sysproperty key="mySeleniumServer" value="${mySeleniumServer}" />
 <sysproperty key="mySeleniumPort" value="${mySeleniumPort}" />
 <sysproperty key="mySeleniumBrowser" value="${mySeleniumBrowser}" />
 <sysproperty key="mySeleniumUrl" value="${mySeleniumUrl}" />
 <classpath>
  <pathelement path="${project.class.path}" />
  <fileset dir="${dist.dir}">
   <include name="**/*.jar" />
  </fileset>
 </classpath>

 <test name="my.seleniumtests.MyFirstTest" />
</junit>

The above would let you set the mySelenium* Ant properties in your build configuration parameters using the -DmySeleniumProperty=mySeleniumValue syntax. Point these properties (or if you hard-coded locations in your tests, then set those) to your Selenium RC server, Selenium RC port (4444 is default), test browser, and application URL.

Review

When everything is setup, you should have:

Source Control

• Your application
• Build script for application (optional) and regression testing (can be same script)
• Selenium tests in JUnit/Java format

TeamCity Server

• (Optional) Build configuration for application
  • Connected to source control system containing application code (application and build script)
  • Hooks into build script for application
• (Possibly separate) Build configuration for regression testing
  • Connected to source control system containing testing code (tests and build script)
  • Hooks into build script for regression testing
• Server running
• At least one agent running

Selenium RC Server

• Browser(s) for testing installed on same machine
• Server running

Test

Do a trial run with at least one Selenium test. The easiest way is to click the Run button for your TeamCity regression testing build configuration. (You can also setup automatic build triggers based on source code changes, timing, etc.) It should check out the source code from the source control system you indicated, perform a build of the Selenium tests if necessary, run the Selenium tests via JUnit, and report back the test results to TeamCity.