Selenium Remote Control

http://selenium-rc.openqa.org/

Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser.
Selenium RC comes in two parts.

1. A server which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them.
2. Client libraries for your favorite computer language.

TestNG

http://testng.org/doc/

TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionality that make it more powerful and easier to use, such as:

• JDK 5 Annotations (JDK 1.4 is also supported with JavaDoc annotations).
• Flexible test configuration.
• Support for data-driven testing (with @DataProvider).
• Support for parameters.
• Allows distribution of tests on slave machines.
• Powerful execution model (no more TestSuite).
• Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc...).
• Embeds BeanShell for further flexibility.
• Default JDK functions for runtime and logging (no dependencies).
• Dependent methods for application server testing.

TestNG is designed to cover all categories of tests: unit, functional, end-to-end, integration, etc...

Why use TestNG with Selenium RC ?

Selenium IDE by itself is great for creating test cases and executing single tests. TestRunner is ok at running a test suite. But if you want full automation against different server and client platforms, you need a way to invoke the tests from a command line process, flexibility in how you create your test suites, and reports that tell you what happened. TestNG provides these features. You can use Selenium RC with other languages as well, but since Sakai is written in Java it makes sense to stick with the language most of us are familiar with already.

Converting Selenium Tests to the TestNG format

To get started you need some Selenium Tests in the default HTML format. The IDE has built in support for converting this format to other formats. The format is written in javascript. You can find an example of the format we use attached to this page. This format relies on the base abstract class which holds some common functions. You can find all this stuff in contrib: test_suite

To Convert an html test to your preferred format:

1. Add the format to Selenium
   1. From the IDE menu; Options > Options > Formats > Add
   2. Type the name of the format (e.g. SakaiTestNG) and then paste the formatting code into the textarea and select OK.
2. The format contains certain properties that will need to be set. These include Testname, Description, dependsOn and groups. These properties tell TestNG when to use each test and whether or not the test has any dependencies (i.e. other test that must be run first)
3. Close and reopen Selenium IDE
   1. Selenium IDE will not recognize the properties you've set until it is restarted
4. Load your html Selenium Test case
5. Export the test into the TestNG format you created in step 1

You can find more information on creating a custom format here: http://wiki.openqa.org/display/SIDE/Adding+Custom+Format

Manual considerations

You may want to control the groups or dependencies for tests manually depending on your needs. You can pass this data in from the Selenium IDE, but you may find it easier to manually maintain this in the java file directly. See the annotations in following for an example.

public class Create1000Users extends AbstractTestCase {
@Test(groups = "Scale", description = "creates 1000 users", dependsOnGroups = "setup")
@Parameters({"webSite"})
	public void testDevSS01(String webSite) throws Exception {
	session().open(webSite);
		session().type("eid", "admin");
		session().type("pw", "admin");
		session().click("//input[@value='submit']");
		session().waitForPageToLoad("30000");
		session().click("//a[@class='icon-sakai-users']");
		session().waitForPageToLoad("30000");

Formatting Dates

You may also want to use Java to format dates in your test cases. If you don't format that dates in the TestNG java code you can also do this in javascript with a custom user-extension.js file. See more about that in the next section.

Uploading files

The big issue we ran into was with uploading files. Selenium IDE wants a local file path for the file, this obviously becomes an issue when you try to run the tests from different clients. The Java client API has a method called attachFile() which lets you point to a url for the resource. What happens when the test runs is the file is downloaded to a temp directory on the client and then uploaded to the browser from there. This is great because you don't have to put the resources on all the clients or figure out how to right a single path that will work on windows and linux.

For example,

     session().attachFile("//input[@id='content_0']", fileServer + "resources.xls");

The location of the fileServer is parametrized and gets set at run time, see the sample testng.xml file below.

FCK Editor

Tests that execute use cases involving the FCKEditor will need to turn the FCK Editor off. You can do this by installing the Firefox plugin Adblock Plus.

Firefox Profile

Modify the project.properties file to set the location of your Firefox profile, see http://support.mozilla.com/en-US/kb/Profiles for more info on that.
The default user_extension__file location should work without modification.

# for RC standalone or with hub
user_extension_file=${basedir}/user-extensions.js
firefox_profile=/Users/jbush/Library/Application Support/Firefox/Profiles/yrhn6hoy.default

Using a custom user-extension.js file

Some tests requires custom javascript. A common use case is setting dates on pages. Sakai has a wide range of date pickers that requires different date formats. In order to write tests that do things like pick a date next week, you need to invoke some custom javascript. See the example user-extensions.js here, https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite/user-extensions.js.

Creating test suites (Using the TestNG xml file)

The great part about TestNG is you can configure what tests you run, in what order, and set the parameters all from a simple xml file. This allows for quite a lot of flexibility.

In the example below, notice the parameters. We are passing in the location of our Sakai instance, where selenium RC is running on what port, what browser we want to invoke, a default timeout, and the location of files we will upload to the Resource tool.

That is followed by a series of tests run in order. We populate the Sakai instance with some data; things like users, sites, and resources. Then we run all tests in a certain package with a group named "Sakai-Reg-Suite". Selenium runs this is any order it chooses, and that is perfectly ok because they are only dependent on the Create User and Create Sites test. Then at the end we tear the whole thing down. That way we can run the test again from a different client (like internet explorer) and we don't have to clean the Sakai instance.

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="rSmart CLE Test Suite" verbose="10">
   <parameter name="webSite" value="http://testserver.com:8080/xsl-portal"/>
   <parameter name="seleniumHost" value="localhost"/>
   <parameter name="seleniumPort" value="4444"/>
   <parameter name="browser" value="*firefox"/>
   <parameter name="timeout" value="10000"/>
   <parameter name="fileServer" value="http://testserver.com/selenium/TestingData/" />

   <test name="Sakai Regression Test: Create Users">
      <classes>
         <class name="org.sakaiproject.test.DevUSERS01"/>
      </classes>
   </test>

   <test name="Sakai Regression Test: Create Sites">
      <classes>
         <class name="org.sakaiproject.test.DevSS01"/>
      </classes>
   </test>


   <test name="Sakai Regression Tests: Tools">
      <groups>
         <run><include name="Sakai-Reg-Suite"/></run>
      </groups>
      <packages>
         <package name="org.sakaiproject.test"/>
      </packages>
   </test>


   <test name="Sakai Regression Test: Delete Sites">
      <classes>
         <class name="org.sakaiproject.DevDELETESITES"/>
      </classes>
   </test>

   <test name="Sakai Regression Test: Delete Users">
      <classes>
         <class name="org.sakaiproject.test.DevDELETEUSERS"/>
      </classes>
   </test>


</suite>

You can create a series of these tests pointing to different Sakai instances and fire up tests at will. Another handy features I've discovered is the ability to include other testng files. This is helpful because you can maintain different tests that populate, tear down, or do both, and run them separately as needed. Its a great way for developers to seed Sakai data quickly and consistently.

<suite name="Sakai Test Suite" verbose="10">

   <suite-files>
      <suite-file path="sakai-suite-populate.xml"/>
      <suite-file path="sakai-suite-populate-clean.xml"/>
   </suite-files>

</suite>

Launching Remote Control

Before you can run a test suite. You need to launch remote control. Basically this process acts as a server proxy which listens for commands, executes the browsers and invokes the commands in the browser. The only requirement is the selenium server jar (attached here).

ant launch-remote-control

Once it launches you'll get some output like this:

Macintosh-9:lib jbush$ java -jar selenium-server-1.0-SNAPSHOT.jar -firefoxProfileTemplate "/Users/jbush/Library/Application Support/Firefox/Profiles/yrhn6hoy.default" -userExtensions "/Users/jbush/dev/projects/cm_tools/selenium-grid/Setup/user-extensions.js" 
11:54:00.607 INFO - Java: Apple Inc. 1.5.0_16-133
11:54:00.608 INFO - OS: Mac OS X 10.5.5 i386
11:54:00.613 INFO - v1.0-SNAPSHOT [1123], with Core v1.0-SNAPSHOT [2214]
11:54:00.704 INFO - Version Jetty/5.1.x
11:54:00.705 INFO - Started HttpContext[/,/]
11:54:00.706 INFO - Started HttpContext[/selenium-server,/selenium-server]
11:54:00.707 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
11:54:00.715 INFO - Started SocketListener on 0.0.0.0:4444
11:54:00.715 INFO - Started org.mortbay.jetty.Server@84ec44

Running a test suite

OK, now on to the fun part running some tests. Once RC is up and you have your tests and a TestNG xml file, you launch a separate process to run a test suite. TestNG is a java framework, we invoke it from an ant script. Here's a snippet of what we do:

 <target name="run-suite" depends="compile" description="Run Selenium tests">
     <echo>Running Suite from file: ${suite}</echo>

      <testng classpathref="runtime.classpath"
              outputDir="${basedir}/target/reports">
              <xmlfileset file="${suite}"/>
     </testng>
   </target>

To run this ant task you simply pass in the location of a TestNG xml file like this:

ant -Dsuite=rsmart-suite-single.xml

This will fire up the browser, run through the tests, and give you a few nicely formatted HTML reports with the results. The output shows up in the target/reports folder. A full example of what a build.xml might look like can be found here.

Sakai Selenium Tests in Contrib

We are starting to collect Selenium tests in SVN. Anyone can take these and convert them using the process described above into the TestNG format, or any other format.

The TestNG project can be found here. This provides all the test cases currently available and the skeleton to compile and run selenium RC and TestNG. It also has plenty of examples of what TestNG xml files look like, so that can jump start your own project.