Make it easy to develop integration tests

Description

Many of the integration tests in the tool layer have extensive mocking and workarounds. Even though they will run somewhat slower, my interest is in making it easy to test against a fully bootstrapped kernel with an in-memory database.

The strategy behind this is that it's exceedingly difficult to mock everything that one of these giant Action classes uses, and then your tests aren't really verifying production code, so we should test against the real objects. With enough generic support code, it should be practical to set up test data builders and cover the core features. Once a feature is covered, we can refactor the pieces out, unit test them, and decide whether to trim down the feature tests or not. Either way we will need tests at our backs to do the kind of work planned with session state, and the more real they are, the better, barring browser automation.

I have done enough research to believe there is a pretty straightforward path to having a kernel that can start up and shut down per integration test case, making it easy enough to separate them by feature and keep fixture data focused. It should also be easy to inject services wherever they are needed to aid in offloading some of the tangled dependencies.

Environment

None

Test Plan

This is about testing – provided we stay green and regular development / QA can carry on in master, we're in good shape.

Activity

Noah Botimer June 22, 2022 at 6:51 AM

Regarding the pull requests:

 

I am happy to do a walk-through, code review, or just plain chat about it. There is a lot of stuff there, and it will take time to absorb. It’s all intended to be safe and, as much as possible, opt-in. For example, the new kernel loading is off by default. I’m not sure how much explanation people will need beyond the tests and extensive Javadoc, or if they would prefer to discuss big picture and/or implementation before looking it through.

I talked with and the other day about what I was up to. I know and have been the most recent people to work in the areas affected. Please let me know how you would like to approach this – I am eager to see it merged and make some headway on improving these well-loved tools.

P.S., Wherever you are, Ian, I can see the K2 summit from here.

Noah Botimer June 22, 2022 at 6:43 AM

After a long time away from this, I have returned and feel victorious. There are real acceptance tests, running against a real kernel and in-memory database. Any needed services are trivially injected by annotation. We can call methods directly on an “action” class (velocity tool servlet), or we can pass requests through the RequestFilter and ToolPortal. Test classes get all of that setup by adding the test-helper in the pom, extending a base test class, and customizing the context with whatever is needed for the tool (like loading the service beans).

I believe these are all of the ingredients required to enable practical integration testing, behind which we can refactor. The first targets would be extracting the inner classes and the large, use-case-specific chunks of code into their own classes, so we can trace the dependencies (on instance variables, outside services, “action state”, request attributes, and session state) more narrowly, and work toward code with small, explicit context.

This can happen one feature at a time, working in main the whole time, and covering features that have never really been tested. If the open PRs are merged, we can clean things up, safely, and without a grand rewrite.

Automation for Jira June 22, 2022 at 5:46 AM

A pull request has been created, "SAK-43201 - Add alternative to ComponentManager", you can see it at https://github.com/sakaiproject/sakai/pull/10676

Noah Botimer February 9, 2020 at 9:23 PM

There is an open pull request for this – happy to discuss. I do think some forthcoming examples of application will be illustrative.

Noah Botimer February 9, 2020 at 8:41 PM

I did a ton of digging into the component manager and, in particular, the relationship that almost everything has with the static cover. There are some pretty offensive things in there, and I think we can do better. Some of the issues:

  • The kernel tests use a magic sakai.test property to indicate a source directory that holds a literal test.properties file

  • The test.properties entry is in the core, default config (sakai-configuration.xml)

  • The ComponentManager cover creates the instance of SpringCmpMgr with a weird reentrant lock and deferred refresh

  • The lock / deferred refresh allows the loading thread to keep poking at the context while it's initializing

  • There are gobs of circular dependencies, even with the very first few beans, making startup very touchy

Enough problems... Let's talk direction...

  • I want to be able to use a base class for a test case and have a kernel come up with all of the beans in context

  • I want to be able to make a component manager instance and inject it into the cover so all the code that uses it can get a configured instance

  • I want to be able to customize properties, config, and components in the newly made component manager

  • I want to be able to use annotations and Java config to have both services and tool-specific code injected without ceremony

  • I want to be able to use the same base test infrastructure across tools/modules without a lot of setup or duplicate maintenance

I have some code ready that I think will prove compelling...

Fixed

Details

Assignee

Reporter

Components

Fix versions

Priority

Created February 9, 2020 at 8:28 PM
Updated March 3, 2025 at 7:56 AM
Resolved September 15, 2022 at 10:23 PM