QA Milestone Boundries
Context
This document defines a suggested structure on how to transition from Milestone, beta, release candidate to production.
Note:This document does not define the relationship between the release process and the community, extra rules of engagement may need to be expanded into this document after debate.
Milestone to Beta
Milestones occur once every two weeks no matter which Jira blockers exist. The transition occurs once the Release Manager accepts that the code base is functionally frozen.
Note: In the future for Sakai 3.x, the transition also does not occur if any unit test during build fails.
During this period QA:
- Organize testing resources.
- Performs a static code review sweep
- Places a QA server under constant base level load and verifies log file errors.
The Release Manager:
On top of their other duties, the Release Manager delegates a sanity checks of Jira, searching for significant issues to feed to the maintenance team and QA.
Beta to Milestone
If the code base becomes significantly functionally unfrozen then the next tag is a milestone. If there is a discussion over significance, the Executive Director has the deciding vote. Any unfrozen code at any level of significance is given extra attention by QA even at the cost to other area's of functionality.
During this period QA:
- Reviews unfrozen code
Beta to Release Candidate
Beta's occur once every two weeks no matter which Jira blockers exist. A release candidate transition occurs after there are no outstanding blockers and all critical issues have been assigned with a clear time to finish. The Executive Director can override this criteria for specific Jira's. Further, the all micro defects from the code reviews needs to have been removed. The release candidate is cut after all significant functionality has been tested and the tests documented on confluence. During this period of time QA performs as many functional tests as possible and signals any logistic issues to the Executive Director. QA plans to automate more tests but that is mostly after the release of 2.7. The Release Manager discuss issues with QA regularly. If the QA director is not satisfied with the level of testing then they should signal this to the community and the Executive Director makes a resource appeal to the community.
Release Candidate to General Availability
Release candidate tagging is driven by the cleaning up of code. As bugs are removed new tags are cut. The longest time between tagging is 3 weeks. If this not achievable due the lack of resources then the Release Manager signals this to the community and the Executive Director and the Executive Director makes an appeal for resources.
The Executive Director accepts that the Sakai final tag based on a memo from the QA director with comments from the Release Manager. The final tag can only be cut if there no blockers. The blockers accepted by the Executive Director when transitioning to release candidate have by this point been removed. A blocker becomes a blocker, but a blocker can be reassigned to critical with a clear TTL by the Executive Director. All significant issues are described in the release documents. The product council is informed of the known issues document and have the right to recommend extra release candidates. No issue is left unassigned.
General Availability
There is a clear relationship between QA and early adopters. QA is actively involved, log files are parsed for new errors, data is collected over usage. Early adopters help form the maintenance teams priorities. The first minor tag is released within a month of the major tag. The criteria for acceptance of a minor tag is defined.