QA -- Improve Overall Quality of Releases

The beginning of the academic year was extremely difficult for a number of institutions. This is a strong indication that our current development and release practices need some thought and new strategy. I think we're all in agreement that focus needs to be placed on ensuring higher quality software. Such a hot topic is very apparent in the community -

• Berkeley in Crisis email messages
• "If I had a simple set of "automated tests" which ran, which tested misc things like - 1. uploading files 2. downloading files I would have caught this problem immediately. It seems if we could classify a few simple "basic health" tests of the system, these would be great tools all sites could use. I've run automated tests to retrieve data files in the past, but haven't done the upload portion." - Steve Miley @ USCB
• "Acceptance or functional testing is really important, but in my experience often really hard to automate. " - Colin Clark U of Toronto
• Serious scalability issues with Oncourse at Indiana U

Starting with the basics

1) What is it we're trying to produce?
This point is directly related to Release Practices. Before attempting to define how to go about ensuring high quality, a common goal set is essestial.
2) What is the level of quality we are striving for? What is it we want to be able to say to speak to when we discuss the quality of the software?
One of the first steps in improving the level of quality of the software is coming to a common understanding of the level of quality the community deems acceptable in the long run as well as short short term goals. As I see it, this will be a concept that will need to be revisited a number of times over the coming years until there is a sustainable practice in place.

Using a well defined and common set of vocabulary

Aaron Z's Take
I wrote up something like this here (and included a number of references): http://confluence.sakaiproject.org/confluence/display/SAKDEV/Sakai+Kernel+Roadmap
(under Reliable and supports High Availability)
Thus:
1) "logic" testing is what the industry calls unit testing and we should too
2) integration testing is defined as the phase of software testing in which individual software modules are combined and tested as a group.
It follows unit testing and precedes system testing. The purpose of integration testing is to verify functional, performance and reliability requirements placed on major design items.
I'm not sure this really captures that: "Does this code work test with real collaborators (test without full system)"
3) If you are mentioning Sakai integration testing technologies you should include test-runner http://confluence.sakaiproject.org/confluence/display/SAKDEV/Test+Runner
4) Functional testing is typically used to mean integration testing in most cases where I have seen it used. If you mean requirements testing you might want to say that instead. Otherwise we risk inventing our own meaning for a term that is heavily used in the industry.
5) This might help for your system, testing bit: http://en.wikipedia.org/wiki/System_testing
6) Load testing would normally occur after system testing. Not much sense in optimizing for load when the thing does not work right as a whole.

What approaches should we take to testing, and who can do the testing?

Generally agreeing we need higher quality software isn't enough. I believe that making steps in the right direction is likely going to mean lesser focus on the types the things (features) the community has typically focused on in the past.

Lance listed off a few ways to improve the quality to the management list and I encourage everyone to use this as a jumping point, For instance, what's your stance on the value in placing focus on these different types of testing (or others you have in mind) and do you have ideas on practical steps we can take immediately?

• Unit testing
• Load testing
• Automated UI testing

I believe that unit testing and performance testing absolutely need to be factored into our development and release processes. We should be confident that the basic elements work, what point they scale and this should be proven before the code is considered for inclusion (that relates more to release mgmt). To speak solely to unit tests, we're not going to be able to make this happen overnight but there are ways we can make small steps. Couple ideas . . .

• all new development requires unit tests
• Areas where a regression is introduced require unit tests
• retrofit some of the higher stakes pieces of the software with this (for example, code that deals with grades and assessment)

As for performance testing, we need it to be conducted centrally and prior to release. The scripts should be written in an open source framework so that institutions can not benefit from them locally, but can also easily contribute back. Lastly, I have to say that I'm not sold on automated UI testing. The scripts generated for this type of testing is notoriously frail and I believe that we'll receive much more benefit from focusing on performance and unit testing in the near future.

Roadmap

This is not an initiative that can happen overnight.

References

BOF's

• BOF to coordinate Sakai integration test support- http://bugs.sakaiproject.org/confluence/display/CONF08/Automated+Testing+BOF
• Performance testing BOF -

Thoughts from project teams or institutions

I'd really like to see more folks post out preliminary thoughts on the kind of quality they'd like to see in the foundation releases, how they believe we can do this and if they are willing to commit resources to this.