Sakai release guide

Note: The release process changed on release 2.9.0-b03. Please reference these instructions for the old guide: Sakai CLE release guide (pre 2.9)

The current "full" release process (2.9.0-b03+) basically involves these steps:

This guide is based on the release order defined at Indie release order - 2.9. Please refer to that and update that when any new tools are added to indie.

Pre-Release step: Obtaining Credentials

In order to perform a release, you must have credentials on the build server (http://builds.sakaiproject.org:8080/) as well as access to commit to every branch/tag in source.sakaiproject.org.  To just release an individual tool will should only require access to Jenkins. This guide will only provide brief details about this process, if you are seriously interested in getting more involved in the release process, have a CLA on file and are an active member of the Sakai CLE community we'd welcome the participation in the release process.

• You need to have an account on Jenkins. If you click login->login with open id, a temporary account will be created. This needs to be given permission to the system otherwise you'll just get an error.
• All artifacts are uploaded and released from Sonatype. There is a special account for Jenkins to actually upload the releases. All other users need to follow the procedures for registering an account.
  
  • https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide
    
    • The actual process is under "Signup" and it involves signing up for a Jira account and adding a comment to the Sakai CLE project. https://issues.sonatype.org/browse/OSSRH-2835
    • Your request will be approved/denied and you will be able to publish org.sakaiproject artifacts 
  • Only indie artifacts are released to this repository. Core tool artifacts are not currently released.
  • These artifacts are gpg signed by the CLE Release team user on Jenkins server. 
• If you add a task in https://jira.sakaiproject.org/secure/Dashboard.jspa under "Infrastructure" the other appropriate access will be granted.

Release Kernel and Master corresponding to this release "manually"

After all issues are merged to the appropriate branches, the first step in creating a release is releasing Kernel and Master.

All of these 2.9.x projects (the current release) are in Jenkins under the CLE 2x indies tab in the top. Most indies are released semi-automatically (later steps) but these 2 have to be released manually. Kernel and master both have to be released manually under a special process because properties need to be updated in the XML so the process is a little different. Ideally this could be simplified by

• Only having the properties in one pom (moving all of the kernel dependencies and properties into master)
• Have some of the properties protected with profiles
• Using the versions:update-properties goal

But this is how I've been doing it for this release. 

Releasing the Kernel

TODO: Review this kernel master process for making a release branch. This was useful during the Release Candidate Phase but when the real release happens, if you don't remember to go back and change the regular pom's then the build will fail as all of the SNAPSHOT's actually do get changed, so probably this should be altered to just include editing these two 'in-place' and notifying the community that the build will be broken for a few hours. Ideally kernel will just be a regular indie with master on top, making it so only master needs to have special consideration.

Pom changes

The kernel is released from the Jenkins page. All of the projects should be properly minimally configured with the Jenkins M2 Release Plugin. This is similar to doing the maven release:prepare release:preform from your local machine. The server does the branch, tags the new release and releases the artifacts to sonatype (since sonatype is the parent for the master and the kernel). Kernel and master are a little different, because as mentioned some manual work is involved in this process.

What I've been doing is releasing from a separate branch

kernel-1.3.x-release and master-2.9-x-release

svn rm https://source.sakaiproject.org/svn/kernel/branches/kernel-1.3.x-release
svn cp https://source.sakaiproject.org/svn/kernel/branches/kernel-1.3.x https://source.sakaiproject.org/svn/kernel/branches/kernel-1.3.x-release

svn rm  https://source.sakaiproject.org/svn/master/branches/sakai-2.9.x-release
svn cp https://source.sakaiproject.org/svn/master/branches/sakai-2.9.x/ https://source.sakaiproject.org/svn/master/branches/sakai-2.9.x-release

Check out the kernel release somewhere

svn co https://source.sakaiproject.org/svn/kernel/branches/kernel-1.3.x-release 

Then modify the pom.xml file, changing only the property to whatever the next version will be. The release plugin will change the rest.

For example for the 2.9.0-b07 release (and the 1.3.0-b07 kernel) the properties block 

<properties>
  <sakai.kernel.version>1.3-SNAPSHOT</sakai.kernel.version>
  <sakai.jdk.version>1.6</sakai.jdk.version>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

becomes

<properties>
  <sakai.kernel.version>1.3.0-b07</sakai.kernel.version>
  <sakai.jdk.version>1.6</sakai.jdk.version>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

    <scm>
        <connection>
            scm:svn:https://source.sakaiproject.org/svn/kernel/branches/kernel-1.3.x-release
        </connection>
        <developerConnection>
            scm:svn:https://source.sakaiproject.org/svn/kernel/branches/kernel-1.3.x-release
        </developerConnection>
        <url>https://source.sakaiproject.org/svn/kernel/branches/kernel-1.3.x-release</url>
    </scm>

svn commit this file 

Jenkins Changes

To start this process, login into Jenkins at builds.sakaiproject.org. Find the project called "kernel 1.3.x - release" and click on the link "Perform Maven Release". This is already setup to point to this directory and be a release build.

On the page that follows, you need to fill in the release version which you can usually figure out by looking at the tags (https://source.sakaiproject.org/svn/kernel/tags/) for instance "1.3.0-b04". Otherwise ask the release team. This property was defined in the previous step. The development version (which is during most releases is still the current release-SNAPSHOT, for instance "1.3-SNAPSHOT".) You also need to supply your SVN credentials (currently). Hitting release should release the kernel to the repository as well as create the tag.

Note! If any part of the release process fails, you might have to do cleanup (svn rm) manually, deleting broken tags or fixing branches. There is no mvn release:rollback option currently in the maven release plugin. Hopefully the kernel will be released after 15 minutes or so (longer depending on the existing jobs) and appear as a module you can release on sonatype (https://oss.sonatype.org). Before releasing kernel, you should also just release master.

Releasing the Master

After the kernel is released, master can be released. Kernel needs to be released first because master has a dependencyManagement import dependency on the kernel.  SAK-18921 - Getting issue details... STATUS

The properties in master need to be updated similar to kernel, so checkout master

svn co https://source.sakaiproject.org/svn/master/branches/sakai-2.9.x-release 

• Open the pom.xml in a text editor, in the properties section all -SNAPSHOT properties need to be updated. So how do you know what versions to use!?! I've been just keeping them updated in this yaml file for each release. This should be improved. https://source.sakaiproject.org/contrib/cle-release/scripts/jenkinsrelease/

<sakai.version>{this version}</sakai.version>
<sakai.kernel.version>{version of the kernel you defined earlier}</sakai.kernel.version>

• Only update the properties block, making sure to update only the SNAPSHOT properties.
• After this edit is done, commit this back and go to the Jenkins server. A special job is created presently called "master 2.9.x release". 
• Now you can click the button on the side "Perform maven release" similar to kernel. Enter the two versions (in this case 2.9.0-b04, and 2.9-SNAPSHOT) as well as your SCM login/password again. Then click "Schedule Maven Release Build"

Both Kernel and Master should now be in the Repository. Double check that these tags are correct before going any further or you will waste a lot of time! 

NOTE! At this point before you go any further, you need to login to http://oss.sonatype.org and verify that the kernel and master are in the staging repository. It should look like this:

There might be other people that have released for OAE but just look for what you released. If they are 100% correct you need to close and release both kernel and master. Click on the close button, type in a description then the release button.

Release Indie Tools in Order

To perform the release, some ruby scripts are needed. You might be best installing RVM (or maybe jRuby) rather than using the ruby in your distribution. This was tested on ruby 1.8.7.

Check out this directory with svn: https://source.sakaiproject.org/contrib/cle-release/scripts/

The script needs two modules nokogiri and httpparty. cd to the directory "scripts/jenkinsrelease". You can install these two gems with

gem install bundler
bundler install
 
(Note, sometimes you need some libs installed in-order to get these libraries installed. Perhaps it's possible without the libs, but these were avaiable both in Ubuntu and Centos). 
See http://nokogiri.org/tutorials/installing_nokogiri.html
 
For info about that!

In order to get this program to run, you need to create a special configuration file. By default this file is located in ${HOME}/.m2/release.yaml

This is a yaml file with 4 values:

• jenkinsusername: (the name you use to login to Jenkins)
• jenkinspassword: Your API key for Jenkins. Click on your name in the upper right, click configure and click "Show API token". This is the password you want here
• scmusername: The cle-release-team user
• scmpassword: The cle-release-team password provided!

jenkinsusername: jonespm
jenkinspassword: <jenkins apipassword>

scmusername: cle-release-team@collab.sakaiproject.org
scmpassword: <scm password>

The file in this directory "tools.yaml" defines the jobs and the release order. This order is on the top of this page.

It has a format like

tools:
o01:
- svnpath: https://source.sakaiproject.org/svn/entitybroker
  propname: sakai.entitybroker.version
  releaseversion: 1.5.0-b04
  devversion: 1.5-SNAPSHOT
  jobname: entitybroker%201.5.x

You need to edit this file and update all of the release versions to match what you edited in master in the previous step. Ideally this process will also be able to release master. 

Then you can release all of the indies with this script. The order is already good to go.

You run this with

./clebuild.rb -o o01 -m 2.9.3

Doing this will release entitybroker the item in the o01 group. You can specify multiple jobs to deploy, or if you're feeling lucky you can just do it all! The -m parameter is required and it specifies the master version.

./clebuild.rb -o o01,o02,o03,o04,o05,o06,o07,o08,o09,o10 -m 2.9.3

The clebuild script will examine the yaml file, and sort all of the jobs in alphanumeric order. All of the jobs you specify on the command line will be released.

Note! Because of the bug in MRELEASE-449, this script has to checkout the project, run a mvn versions:update-parent on it, and commit it prior to performing the release.

Frequently check the Jenkins page to make sure nothing failed (nothing has a red dot). If something failed, then future groups will not run because each depends on artifacts from a previous group existing, so you'll have a bunch of failures which you probably will need to cleanup. So it's safer to do fewer and keep checking on it.

Jenkins takes about 2-3 hours to process and build all of the indies. If everything completes succesfully, login to http://oss.sonatype.org, change the filter to org.sakaiproject and there is usually 2 projects.

I usually do an extensive verification to make sure everything is there and check it off on this page. Indie release order - 2.9. If you release something and it's wrong, that artifact cannot be used and you have to re-release it and master with a new version.

After everything looks good, click on the checkbox next to the bundle and click Close (Wait a few minutes) then Click Release (Waiting again). Sometimes the page times out, but if you refresh, eventually it works.

Now you need to wait a few more hours for the indies to get to maven central.

Release Non-Indie Tools as a package

<TODO - THIS SHOULD BE MUCH BETTER, THE PROCESS FOR UPDATING VERSIONS IS BAD AND THIS SECTION NEEDS EDITING>

To do the core release:

Run the corerelease.sh script in the cle-release/jenkinsrelease path. This would ideally do everything. I think at this moment it just copies all of the tags for the core. It should be able to do the rest once I get around to it. :/

You run it with this command filling in the appropriate values: ./corerelease.sh https://source.sakaiproject.org/svn/sakai/branches/sakai-2.9.x 2.9.0-b06 2.9-SNAPSHOT SAK-22348. 

That command runs svnmucc and makes all of the branches

• Copy the 2.9.x branch to a tag (svn cp https://source.sakaiproject.org/svn/sakai/branches/sakai-2.9.x https://source.sakaiproject.org/svn/sakai/tags/sakai-2.9.0-b06
• Checkout the tag with the --ignore-externals flag (important)
• Update the version in the pom.xml for the base by editing the file
• Edit the .externals file
  
  • Update all of the externals from branches to tags and update the .x to the correct version  (In vim this is :%s/branches/tags/g  and :%s/\V.x/.0-b06/g ) Replacing b06 with whatever the version is

Note: The master is being copied to the "master-2.9.0-b06" tag, rather than the "sakai" tag. All of these may need to be changed around eventually for the 2.10 release too to match the project name rather than the sakai name like the indies. Note the name for the master is wrong because it is an indie.

Note: You really need to update the externals: svn propset svn:externals . -F .externals .

After you've updated the pom (before committing) you can run "svn up" which will get all of the new externals from their tags, however the versions are wrong.

I probably used pomversionupdater.bash 2.9-SNAPSHOT 2.9.0-b06 /tmp/sakai-2.9.0-b06 . . . updateparents task didn't seem to work.

Commit all of this now.

(Run this command in the root of the directory to generate the command to commit it)

clear ;echo -n "svn ci " ; find ./*/ -maxdepth 0 -type d -print | grep -v ".svn" | xargs -I {} echo -ne  {} "" ; echo -n " pom.xml -m 'Committing sakai core release poms'"

Now run mvn clean install -Ppack

(Deploy all to shared release)

./ssh/config

Hostname source.sakaiproject.org
User shared_release

PWD=`pwd`;THISDIR=`basename $PWD | awk -Fsakai- {'print $2'}`;ssh shared_release mkdir -p /var/www/html/release/$THISDIR/artifacts
PWD=`pwd`;THISDIR=`basename $PWD | awk -Fsakai- {'print $2'}`; scp `find pack -name "*sakai*"` shared_release:/var/www/html/release/$THISDIR/artifacts

Add release java docs (TODO: Clarify this a little more)

This step should be done on the sakai-2.9.x-all branch to get all of the docs (or whatever version)

• Run svn up to update the files for the new release (or checkout a new full branch for the -all

• Run the goals
  
  • mvn clean install javadoc:javadoc 
• This will generate javadocs in the target/site/apidocs directory, tar up this directory and copy it to the shared release /var/www/html/release/$THISDIR/artifacts as above. 
• Also uncompress a copy to the apidocs directory.

Update Jira

Visit the SAK project and release update the release notes. Move all open Jiras to next (tentative) version.

Update release documentation

Ideally this work should already be completed. But in case it's not update the following pages:

1. Release page: http://source.sakaiproject.org/release/ 

• to update page on server, need to ssh to server (managed by Indiana) and find the index.html page in /var/www/html/release/[release id]. For example -  /var/www/html/release/2.9.1
• May need to get the http://source.sakaiproject.org/release/ link to redirect to the latest release directory (also Indiana)

2. Update this page - https://sakaiproject.org/cle-release  for the latest version number, if relevant, and make sure the document links on the right are up to date. For the page update, Drupal access is needed. For the document links, may have to check with the web site maintainers hired by the Foundation.

3. Confluence release documentation: https://confluence.sakaiproject.org/display/DOC/

4. Implement the Release Communications Plan.

Common Problems and Solutions

<TODO>

Extra information about CLEBUILD SCRIPTS

<TODO>

How the Jenkins Jobs are setup

•   prerelease script
•   postrelease script