Requirements for test-harness

Requirements for test-harness BOF - Tuesday, July 1st

(Thanks to Michelle Wagner for her notes.)

We all agree that Sakai needs more test coverage. One step towards that is making sure that the few developers who are already writing tests don't splinter their efforts. At the last Sakai conference, Aaron Zeckoski, Jon Gorrono, and Ray Davis (also known as "I" ) agreed to mend the current test-harness / test-runner split. Jon took leadership of task SAK-12334.

Not long before the Paris conference, Jon and I realized that my first guess about how to handle the task was oversimplified and we needed more input from others regarding test-harness requirements. Unfortunately, Jon couldn't make it to the conference and so I arranged for the BOF. Its primary goal was to collect issues and solutions for the test-harness in particular. The overarching goal, however, is to improve Sakai testing in general.

Overview of testing approaches

Revised and moved to a central location in the Development space.

Problems with test-harness

1. Getting started is too difficult for most Sakai developers. It's the "staring at a blank page" problem: the initial hump of learning how to set up test code is time-consuming enough that people are likely to just keep on with what they already know. (This issue applies to all flavors of testing, not just Sakai-service integration tests.)

POSSIBLE SOLUTIONS: a) Update existing working examples to share consistent scaffolding. Improve documentation and send pointers. b) Create placeholders for selected flavors of tests across the core product suite and ask the community to start filling them in. c) Look into Maven archetype? d) Aaron's AppBuilder already does a good job of creating templates for project-level testing. Should it also include optional generation of component integration testing once test-harness improvements are in place?

2. To use test-harness, Sakai developers are expected to extend a convenience class which extends JUnit's TestCase. But tying developers down to that particular subclass keeps them from being able to take advantage of other test superclasses (notably Spring's "AbstractAnnotationAwareTransactionalTests").

SOLUTION: Since every method but one in SakaiTestBase is static, I've split out the test-harness's most important functionality into its own static utility class, "ComponentContainerEmulator". This not only makes it possible to run integration tests using other test classes, but also makes server emulation available for functions like data loading and data migration. (For an example, see GradesFinalizerExec.java.) As we add more test-helpers, we should keep as much as possible in test-class-agnostic utility classes.

3. We're missing some cool Spring 2.5 annotation and autowiring features. Of course, this is true across all aspects of Sakai development, but Aaron and Steve say they particularly speed up test writing.

POSSIBLE SOLUTIONS: This is a tough one. a) The most obvious solution is to upgrade Sakai to Spring 2.5.x, but my first experiments (see SAK-12253) showed some conflicts with Sakai's current classloading approach, and Spring co-founder Juergen Hoeller suggests that we rethink our component manager design. We've been hoping to do that anyway, but prioritizing it to make testing easier sounds a little tail-wagging-the-doggish. Also, some at the meeting said they've had issues upgrading Spring even outside Sakai and so they're feeling wary. b) Alternatively, we could jury-rig our own Sakai-specific copies of a few especially attractive Spring 2.5.x features, and then swap them out for the real things when we can upgrade. This is what Aaron and Steve did in the test-runner project, and they definitely feel it was worthwhile. But this sort of "selective re-branding" sticks in the craw of other Sakai developers (Ray and Ian, for example).

4. There's no standard or automatic way to run all integration tests across all projects. Currently tests are triggered in a project specific way: you need to find the magic directory and know the magic invocation.

POSSIBLE SOLUTIONS: a) Jon Gorrono has created a branch that can hook into Maven's defined (but currently unimplemented) "integration-test" goal, which might then be triggered with a special profile defined in Sakai's base POM. b) Aaron suggested we might be able to get the same results more simply by just running slower tests (of any sort) through the profile alone. c) Josh suggested that integration tests be packaged in their own JARs and then explicitly included in a single test suite, running all tests in a single component manager (which is much faster than starting and shutting down the component manager for each test). This solution has two problems, though. First, the central test suite would have to be edited every time a new test was added to any project. Second, due to internal component caching, the only way to guarantee a "clean start" for an integration test is to shut down the component manager between tests. Since the full suite of long-winded tests will likely be run automatically as part of overnight regressions, the extra overhead shouldn't matter much.

5. The test-harness runs tests in a much simpler classloader environment than Sakai components and web applications will encounter in the real Tomcat server. This has been known to produce "false positives": the tests indicate that the expected functionality is there, but then in a live system classloader conflicts break the code.

POSSIBLE SOLUTIONS: This needs more investigation. I'm sure we'll hit some sort of limit on just how realistic we can get, but we might be able to do better than we are now, perhaps by running the test code in a "component-like" classloader environment rather than in the "shared-like" classloader environment used now. Aaron suggests that we look at Maven's own approach to class loading.

6. There's a life-cycle conflict with test-specific Sakai components that need to be configured semi-realistically (for example, the fake components used in the Component Manager integration tests). For test-harness to find the test components, they need to be deployed, but the "test" phase of Maven runs before the "deploy" phase. (For some tests, we can just code the "components" inside the test class itself. For an example, see the "TestProvider" in AuthenticatedUserProviderTest.java.)

POSSIBLE SOLUTIONS: This is why Jon is experimenting with the "integration-test" phase. The (very few) online examples of Maven "integration-test" use something like HttpUnit to do functional testing, and therefore rely on deploying to a server before running the tests. I think I'm the only person doing this at present, so I'll take responsibility for confirming the fix.

Follow-up with Noah Botimer

Even talking as rapidly as we could, the BOF just barely got through the initial agenda before our scheduled time was up, allowing no room to discuss other items or suggestions. Afterward, Noah Botimer spoke with me about what he'd like to see from the testing working group:

1. We should come up with about five tasks that community members can do ASAP to help with the effort, without any dependencies on pending work and with reasonable estimates for the investment required.
2. We should organize our own tasks so that others can join in ASAP.
3. Since the Spring upgrade / Component Manager design conflict is still unclear, I and Aaron should document the issues well enough to help others decide on priorities.