Test Runner

Information

This page documents the test runner project and explains what it is for and how to use it. (See the summary for more information)
Current code (SVN): https://source.sakaiproject.org/contrib//caret/testrunner/trunk/

TestRunner Summary

The TestRunner project is an effort to make it easier to write integration, load, and data validation tests for Sakai. It allows the developer to easily write a test which can access real Sakai services (without having to look them up or create mocks). It can run any tests which extend the standard Junit TestCase but the primary use is for execution of tests within Sakai. There is a tool that is being developed to make it easy to configure and run tests via a GUI.

The TestRunner is meant to support the writing of complex integration tests and performance checking tests. It is designed to support the goals from the Sakai Technical Goals (towards a roadmap) proposal. In particular, it is made to address these points without putting a heavy burden on the programmers:

• Kernel codebase should include automated testing to ensure stability and reliability of code
• Optional load/integration/validation testing should be easy (and possible) to run

Features and benefits (Why would someone want this?)

• Removes the need for Mock objects
  • Difficult - Mocking up Sakai services is problematic and difficult to do correctly and is a barrier to writing tests
  • Integration instead of Unit - Most tests of Sakai code cannot be written as actual unit tests and are actually integration tests because they have so many dependencies on other Sakai code
  • Saves time - Creating all the mock objects needed by code which has many Sakai dependencies can take hours (or days), time which is better spent on writing actual code
  • Invalid Mocks - The mock objects cannot completely emulate the real thing and code which may work when tested against a mock will actually fail when run in a real system
• Valid testing results
  • Tests are executed inside the Sakai environment and are therefore much more reliable than a test which was run in complete isolation
• Load testing
  • Load tests can be easily written and setup to run in a way that yields accurate results (inside the running system)
  • Since a test is basically any code you want to run you can use them to artificially load your system and see where bottlenecks are
• Data validation
  • The test runner can be used to run tests which will check for data validity (and even correct it if desired)
  • Allows a system to validate that the data it has is ok after or before a migration/upgrade or large scale export
• Easy to use and highly configurable
  • All test runner settings are configurable via spring (and also sakai.properties)
  • Tests can be packaged as components or as a webapp and then dropped in (easily portable without rebuilding)
  • Supports test types (Integration, Load, Data Validation) and types can be individually enabled or disabled
  • Tests use standard syntax and are just an extension of the standard junit TestCase
  • Test classes support annotation based injection via autowiring or explicit bean id injection (using the syntax that will be supported in Spring 2.5 when it is released)
  • Can register your tests programmatically or via Spring beans using the TestExecutor
  • Can disable all tests and control when tests are run or use the tool to run tests
• Flexible and trustworthy code
  • The code has extensive unit testing to test the TestRunner codebase
  • Sample tests (small ones) are included with the TestRunner which are executed to verify it is working
  • Maven 1 and Maven 2 are supported and it should run in Sakai 2.2+

How do I use it?

• NOTE: Take a look at the API (it has extensive documentation in the interface)

1. Use svn to checkout the testrunner project from SVN (location at top of this page) and place it in your sakai source directory and then build it using maven 1 or maven 2
2. Create a test by extending either JUnit TestCase or TestRunner SpringTestCase
   • NOTE: You must use SpringTestCase if you want the ability to access Sakai services
   • Here is a simple test that extends the TestRunner SpringTestCase and gets the Sakai UserDirectoryService (using Autoiwiring automatic bean injection)

     public class SampleTestSakaiUser extends SpringTestCase {
        private UserDirectoryService userDirectoryService;
        @Autowired
        public void setUserDirectoryService(UserDirectoryService userDirectoryService) {
           this.userDirectoryService = userDirectoryService;
        }
     
        public void testCanGetSakaiUDSBean() {
           assertNotNull(userDirectoryService);
        }
     
        public void testCanUseUDS() {
           assertTrue(userDirectoryService.countUsers() > 1);
           assertTrue(userDirectoryService.getUsers(1, 1).size() == 1);
           User user = null;
           try {
              user = userDirectoryService.getUser(UserDirectoryService.ADMIN_ID);
           } catch (UserNotDefinedException e) {
              fail("Exception: " + e.getMessage());
           }
           assertNotNull(user);
           assertEquals(UserDirectoryService.ADMIN_ID, user.getId());
        }
     }
     

3. Add the needed maven dependencies to a project which will run the tests,
   • Name it something like sakai-YOURAPP-tests if you are making a project specially for
     running tests, otherwise you can just put it in the project with your implementation code
   • Maven 2 dependencies for your tests project (where you put your tests code)

     <dependency>
         <groupId>org.sakaiproject</groupId>
         <artifactId>sakai-testrunner-logic-api</artifactId>
         <version>${sakai.version}</version>
         <scope>provided</scope>
     </dependency>
     <dependency>
         <groupId>junit</groupId>
         <artifactId>junit</artifactId>
         <version>3.8.1</version>
         <scope>provided</scope>
     </dependency>
     

   • Maven 2 dependencies for your pack (if building a component) or tool (if building a special test running webapp)

     <dependency>
         <groupId>org.sakaiproject</groupId>
         <artifactId>sakai-YOURTOOL-tests</artifactId>
         <version>${sakai.version}</version>
         <scope>provided</scope>
     </dependency>
     <dependency>
         <groupId>org.sakaiproject</groupId>
         <artifactId>sakai-testexecutor</artifactId>
         <version>1.0</version>
         <scope>runtime</scope>
     </dependency>
     

4. Register the test with the TestRunner
   1. Register via Spring (using a TestExecutor bean definition in your components.xml file)

      <bean class="org.sakaiproject.testrunner.util.TestExecutor">
         <property name="testClassname"
            value="org.sakaiproject.testrunner.impl.tests.SampleTestSakaiUser" />
         <property name="testType" value="testrunner.integration" />
         <property name="registerTest" value="true" />
      </bean>
      

   2. Register programmatically via a service

      public class SampleTestRunner {
         private final static Log log = LogFactory.getLog(SampleTestRunner.class);
      
         private TestRunnerService testRunnerService;
         public void setTestRunnerService(TestRunnerService testRunnerService) {
            this.testRunnerService = testRunnerService;
         }
      
         private String myTestsId = "aaronz";
      
         public void init() {
            // register some tests
            testRunnerService.registerTest(myTestsId, TestRunnerService.TESTING_TESTS_INTEGRATION, SimpleTest.class);
            testRunnerService.registerTest(myTestsId, TestRunnerService.TESTING_TESTS_INTEGRATION, SampleTestSakaiUser.class);
      
            // call a method to run the tests
            if (runMyTests()) {
               log.info("My Sample tests running within a service passed!!");
            }
      
            // unregister my tests since I don't need them anymore
            testRunnerService.unregisterTests(myTestsId, null);
         }
      
         /**
          * This runs my tests and returns a boolean which indicates if they passed or failed,
          * you would probably want to do something useful with the output
          * @return true if tests passed, false otherwise
          */
         private boolean runMyTests() {
            Map<Class<? extends TestCase>, TestResult> m = testRunnerService.runTests(myTestsId, null);
            return TestRunnerUtils.checkTestsSuccess(m);
         }
      }
      

      • This would also require a simple spring bean definition to initialize the SampleTestRunner

        <bean class="org.sakaiproject.testrunner.impl.SampleTestRunner" init-method="init">
           <property name="testRunnerService" ref="org.sakaiproject.testrunner.TestRunnerService" />
        </bean>
        

5. Startup Sakai and the TestExecutor will run the test automatically (unless the TestRunnerService is configured to disable tests)

Code examples

There are some examples that use this in the following projects:

• memory - https://source.sakaiproject.org/svn/memory/branches/SAK-11913/memory-test
• content - https://source.sakaiproject.org/svn/content/branches/SAK-12105/content-test