SAK-12334 * Integration of TestHarness and TestRunner

During the December, 2007 Sakai Conference in Newport Beach, a Test BOF was convened to find consensus on a plan of action for the future of test framwork(s) in Sakai.

This was the outcome:

Requirements

1. Run as a normal unit test integrated with Maven's "test" goal, acting as if the test code was in a deployed Sakai web application calling component service methods.
2. Optionally be runnable inside a deployed component while Tomcat is running ('Online' Integration Test).
3. Optionally be runnable inside a deployed web application while Tomcat is running ('Offline' Integration Test).
4. Allow building and "deployment" of test-specific components before the test is run. Undo "deployment" at end of test.
5. Allow optional test-specific configurations of the component manager.
6. Simple configuration of dependencies.
7. Autowiring of component bean singletons.the
8. Easy control over whether the component manager should stay running between tests, should shut down and restart between tests, or should be completely unloaded from memory (clearing out static variables) between tests.
9. It should be possible to run tests while keeping test libraries (notably a particular version of JUnit) out of deployed builds.

Online vs Offline Integration Testing

Online testing takes place in a running application server. This is the most realistic test possible, particularly as far as component and webapp classloader issues go.
Offline testing takes place in Maven or Eclipse. This provides the fastest development turnaround, and allows for automated regression testing.

Differentiating Integration Tests From Unit Tests

The following is a proposal for additions to the master project's pom.

Summary:

Maven binds specific plug-ins to the build life-cylce phases by default and the bindings are configurable. Within each life-cycle phase, goals are executed in a default or configured order, goal handlers are also called mojos. The test life-cycle phase is bound to the surefire plug-in by default. For the default life-cycle (as opposed to the clean or site life-cycles), the test phase is executed after validate and compile.... but before the packaging phase. The test life-cycle phase is bound to the surefire plug-in by default. And there is no binding for the integration-test phase by default (For more info about maven life-cycles: http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html)

There are advantages to integration testing coming after a full packaging phase, the obvious example being mock object packaging.

This proposal would bind the surefire plug-in to the integration-test phase as well as the test phase, and to create a separate surefire configuration for the integration phase, using different test class name filters to separate the classes tested, and to pass configuration directives to the test classes. Developers can control the behavior of the test-harness classes using pom-defined properties (assuming all the inheritance works )

• Default test phase is skipped (technicality?) so that two executions can be defined for the surefire plugin (the default test phase plug-in)
• The first execution activates the surefire plugin to run during the test life-cycle phase, this time configured to not be skipped, and to ignore conspicuously-named IntegrationTest classes.
• The second execution activates the surefire plugin during the integration-test life-cycle phase, after the packaging cycle phase. During this execution, only the (named) IntegrationTest Classes are run.

<plugin>
         <inherited>true</inherited>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-surefire-plugin</artifactId>
         <configuration>
			<skip>true</skip>
		 </configuration>
		    <executions>
          		<execution>
          		  <id>sakai-integration-test</id>
          		  <phase>integration-test</phase>
          		  <goals>
          		    <goal>test</goal>
          		  </goals>
          		  <configuration>
          		  	<skip>${sakai.integration-test.skip}</skip>

			   		<includes>
           		  		<include>**/*IntegrationTest*.java</include>
				  		<include>**/*IntegrationTest.java</include>
				  		<include>**/*IntegrationTestCase.java</include>
		   		  	</includes>
		   		  	<systemProperties>
	               	 <property>
	                  <name>test.tomcat.home</name>
	                  <value>${test.tomcat.home}</value>
	               	 </property>
	           	  	</systemProperties>
          		  </configuration>
          		</execution>
          		<execution>
          		  <id>sakai-unit-test</id>
          		  <phase>test</phase>
          		  <goals>
          		    <goal>test</goal>
          		  </goals>
          		  <configuration>
          		  <skip>false</skip>
	       		  <excludes>
	              	<exclude>**/*IntegrationTest*.java</exclude>
					<exclude>**/*IntegrationTest.java</exclude>
					<exclude>**/*IntegrationTestCase.java</exclude>
			   	  </excludes>
			   	  <systemProperties>
	               <property>
	                  <name>test.tomcat.home</name>
	                  <value>${test.tomcat.home}</value>
	               </property>
	           	  </systemProperties>
          		  </configuration>
          		</execution>
        	</executions>
           
       </plugin>

.... add that at the same level as the other <plugin/> tags.

Note that the 'test.tomcat.home' property is being passed explicitly to the surefire plug-in. System properties do not appear to be passed into the surefire plug-in's environment, unless explicitly passed on in this way.

Not running integration tests by default

Now that we have made the integration-test phase of maven 'hot' we need to turn it off by default.

In order to keep phase from running, an addtional proposal is to add the following to each of the profiles of the Sakai's root pom.xml :

      <properties>
        <sakai.integration-test.skip>true</sakai.integration-test.skip>
      </properties>

...add the new properties tag at the same level as the <modules/> tags in each profile.... the naming if this new property is intended to mirror the maven convention for skipping surefire-plugin tests which is 'maven.test.skip=true'

With that last addition to the root POM, the defiult and mini profiles would pass the sakai.integration-test.skip=true into the master project pom and only the unit tests (the execution with id=sakai-unit-test) would then run. Overriding this can be done by adding a property definition on the command line: -Dsakai.integration-test.skip=false

More POM enhancements

The following addition to the master/pom.xml run the sakai maven plugin's 'deploy' goal in the pre-integration-test phase:

        <executions>
          <execution>
            <phase>pre-integration-test</phase>
            <goals>
            	<goal>deploy</goal>
            </goals>
            <configuration>
          	  <deployDirectory>${test.tomcat.home}</deployDirectory>
              <skip>${sakai.integration-test.deploy.skip}</skip>
            </configuration>
          </execution>
        </executions>

... add that new <executions/> tag at the same level as the <onfiguration/> tag for the sakai plugin (groupId=org.sakaiproject.maven.plugins, artifactId=sakai)

By default, we would not want to run the deploy goal before integration and so this change would require another default-to-skip setting in the root POM's profiles:

      <properties>
        <sakai.integration-test.deploy.skip>true</sakai.integration-test.deploy.skip>
      </properties>

...this would be just added as a second property in practice: along with the setting for *sakai.integration-test.skip.

Again, with that addition, the default and mini profiles would pass the sakai.integration-test.deploy.skip=true into the master project pom and this new execution for the sakai plugin deploy goal for during the pre-integration-test phase would not be run. This can be overriding on the command line: -Dsakai.integration-test.deploy.skip=false

Testing options

Some ideas for options that can be switched via this mechanism:

1. Enable/disable auto-start for testservice tests (default: on?)
   • ... enable/disable auto-start for testservice tests just for maven-based component bootups

Branch

https://source.sakaiproject.or/svn/test-harness/branches/SAK-12334

How to create tests

Unit tests

Unit tests are outside the scope of this project.

Why mention it? Because what is in scope is often defined by what is not in scope....and, since this technique involves 'messing with' the unit test runner (surefire plug-in), to point out that long as unit tests class names do not overlap (see below) with the pattern matching used to find integration test, then the two executions for the surefire plug-in will not interfere with each other.

Using TestService (Online or Offline)

Using build-time bootstrapped components.

Add the test-harness project to your pom.... and exten that master project's pom

blah

Create a test class that extends org.sakaiproject.testrunner.utils.SakaiTestCase

Name the class so that it fits one of these patterns: *IntegrationTest*.java, *IntegrationTest.java, or *IntegrationTestCase.java

To Do