2.5 Release Planning-8 (20071031)

Meeting Notes

Roughly Paraphrased by Steve G.

Only one agenda item for today

supposed to be released on the 12th. 12 days from now. We are not comfortable with the amount of testing. not sure if pushing the date out is just enough to solve the problem

Number of schools thinking about deploying for new years term, but may not be aware of the ramifications of the lack of testing.

Cape Town is. Somewhat confident.

Stephen/David: As long as the branch in maintained we're pretty confident. Promised users some stuff. June is a difficult time to upgrade. Happy to go with a branch even if the release hasn't happened yet.

<<Seth joins the call>>

Megan: A number of folks have agreed our existing system of releasing isn't working any more. Problems with 2.4.x such as performance, data, etc. Have never felt like we've really done enough testing. Need to sit down and figure out what to do about this. Can't personally endorse releasing on the 12th. Have talked about doing a 2.5 beta. Work on it with some folks in production. Key problems are not enough resources. Not sure if development community will want to support something like that. Some of the biggest problems with 2.4.x were performance testing and we haven't done much of that

Lance: One question to really consider is whether or not we change the process for 2.5 or whether we just grandfather 2.5 into the old way of doing business. No strong feeling either way (IU sticking on 2.4.x)

<<Hannah Reeves joins call>>
<<John and UM folks join call via the polycom technology>>

Michael: I am not comfortable putting out a release knowing it's not testing unless we label with a marker for those grabbing it realize knowing it's not something we recommend putting into production. (unless you're as intense as a school like IU or Capetown).
Ways to approach this issue. When do we start talking about our release management process which most folks agreed needs to change. From conversations with Megan, Peter, we don't feel like there are enough test plans, automated stuff, etc. to test 2.5 a maintenance release of 2.5 and then 2.6 in the Spring. Talked about the possibility of doing a maintenance release of 2.5 after running in schools around January or February. The November release of 2.5 would be a beta release. Problems with that include being on top of the 2.6 release. Worrisome that we'd be able to turn around and test the 2.6 release the way we would want to. There is a resource problem with being able to release a bug-free 2.5 and 2.6 release.

The choices come down to:
1. Focus on getting 2.5 as rock solid as possible between now and next fall. Not worry about the features for 2.6
2.Doing 2.5 as a beta release, never doing an official 2.5 release and moving straight forward to a 2.6.

Those are some options, but we're sure that we don't want a bad release.

Lance: Not sure if we'll be in a fundamentally better position in Spring. Looking at 2.4.x, the improvements have been slow coming and not dramatic impacts. Like the notion of 2.5 beta but not convinced that waiting another 4 months will make 2.5 better.

Michael: Would have to be coupled with more performance tests and putting that effort into 2.5 instead of 2.6.

Lance: Makes more sense.

Megan: Not thinking about these any more as they were before. More of a hybrid.

Michael: Would relax the rules to pull in a few more things into 2.5. Or could be more strict.

Lance: Could bend it as long as an appropriate level of testing went with that chance.

Leasia: Still need a cut off date or else we're back to square one.

Michael: Would have to do this on a feature by feature basis.
Pick 5 or so. It would be hard to make everyone happy.
One requirement is that they would have to have use tests.

We are talking about what would be the foundation release, not what things schools might deploy anyways on top of that release.

What do folks think, assuming that most folks are uncomfortable with the current state of the release.

Lance: Our behaviour needs to indicate that we are more concerned with quality assurance, rather than just releasing things at certain times regardless of quality.

Michael: Would like to have this more deliberately, but as Megan said we are 2 weeks away from the orig release day.

CapeTown: Many of us are unhappy and no one can run the untagged release. Really need to ask what we mean by the release.

Hannah: Agree with most of what was said. We've run up against a lot perceptions on schools that want to use Sakai that the foundation release is ready to go. Want to push releases without denigrating the foundation release or trying to say that it might not be high quality.

<<talking about deployment dates>>

rsmart generally takes the final tag to add the branding and things. most of our local testing is around features we've add, rely on community testing for base things. The releases have gotten more complicated for everyone because of pulling things out of the maintenance branch and pulling things in and out of specific tools. If we don't have enough time between when Sakai goes final and we have to roll out don't have enough time to test the local changes.

Stephen or David: About the Feb date. That's a big problem for the southern hemisphere.

Michael: I think what we're proposing wouldn't leave them in much of a different situation than you're already in. The beta release is for early adopters and they can submit fixes and experiences back.

Hannah: I like the idea of doing a beta release in Nov and then the tight release later in the northern winter.

Michael: Need to bring this into a larger discussion in Newport beach. At the moment narrowly focussing release on 2.5.

Leasia: The Nov Beta, Feb Awesome Release works good for us. Helps solidify confidence for fall release.

<<talk about schools that could test on this beta>>

Jason?: Our interest in going to 2.5 in January goes along with making the jump to MySQL 5. In the past have been too much on the cutting edge. Our intention is to do a lot of performance testing/tuning. We have the environment to do testing on MySQL (not Oracle).

Megan: What's a good beta tag time?

Jason?: Mid November is still reasonable for a beta tag.
Would be cautious about extending the situation with the number of branches.

CapeTown: When you're on a small team switching between branches is time consuming and limits ability to contribute back to the community.

??: What resources would be further committed to testing and development?

Michael: Looking for folks who would pick up the beta release, do some testing or run it in production. (It's no different from where past releases have been). That's on the testing side. On the development side looking for institutions to fix the problems. Looking for a way do to this to wrap up the release. Half of this is just saying to folks not super involved in the community is, "Hey! Don't put this into production yet if you aren't super staffed with Sakai experts". And then after a beta or two, we do have the confidence to release something that folks can actually use.

Jason: Like the beta idea, but unsure of going forward without strong release dates and what's the specific work that has to be done. Would like something like the current qa process but builds spaced out a bit more for performance testing. Basically would like a 2 or 3 month plan for cutting more release builds.

Michael: Excellent points.

Anthony: What's going to prevent schools from ignoring the beta tag and not just going with the 2.5.x branch all the time. Agree a lot with the idea of the schedule so we have more scheduled beta's and release candidates.

Megan: Do we want to rely on a model were folks have to put things into production in order to test our software??

Lance: The notion that somewhere is running it in production does not assume full coverage.

CapeTown: In a status where people come to the list and ask what we're running in production? Are we changing the label and/or the actual process.

<<discussion about 2.4.0 vs 2.4.1 vs 2.4.x and what we tell people what they should actually download if they are new to the community>>

<<discussion about various release processes for different open source projects.>>

Seth: One of the problems is that we proposed the .x branches were not called the latest releases, and that it wasn't fully qa'd etc. We can have a long discussion about how stable each part is. For 2.5 compared to 2.4 and I think it's very hard to make the comparison.

There are numerous numerous changes in 2.5. If you compare it to 2.4.x compared to 2.4.0 or 2.4.1. Those may not be runnable out of the box, you have to upgrade to 2.4.x. The state of 2.5 compared to ... <<it's hard to paraphrase Seth's wisdom and lyrical prowess>>

It's hard to just keep up with the 2.4.0 to 2.4.x changes. The big upgrades are a monster.

How do you make a decision about which one? We need to have a very serious discussion about that. But no...

<<Ack! I ran out of credit on my Skype account>>

...

<<Purchased more Skype credit>>

Anthony: Let Megan do her analysis of where she is now before announcing anything special like when we can cut this first beta tag. She should be able to really look closely at this.

Michael: I think that's a good answer, with one proviso, if we are calling it a beta we should get it out there sooner or later so people can start testing it.

Anthony: Get some schools lined up ahead of time to see if they will test it.

Peter: Chicken and Egg.

Michael: We need to let Megan research and make a recommendation. There's no right answer to that equation.

Megan: We are just talking about doing a beta tag. Beta to me says we are not making any guarantees, and we are going to be making our best to move it along, but the foundation is not saying that this code is a release ready thing.

Jason: We need to get a glossary for this. I know in a lot of projects the difference between something like a milestone and a beta can be vast. Need definitions.

Michael: Objectives: Making sure the 2.5 final release is very solid. The second objective is to signal appropriately things are putting out that are not production ready (especially if you're new to the community).

<<Someone is giving the Apache definition of a beta release)

Hannah: Outside the community I would want to know what the message is. It doesn't matter what we call it if that commitment isn't there on a regularly scheduled basis. If we're pushing out that release, regardless of what it's called, it should have some guarantee from us.

Michael: Broadly speaking, no one wants to cut the 2.5.0 tag while everyone feels there hasn't been enough testing. And that the idea of going from beta, to release candidate, to release, seems to be good. IF, we can document the procedure and define the outcomes so that the community can understand them.

Jason: By biggest worry is cutting a beta tag and not having a clear plan for going forward. Do the people doing QA currently have a stop date of testing assuming the previous schedule.

Megan: I do know of some people we are going to lose. However there are a number of people are just joining the effort. I will be looking at this data and putting it together over the next week.

Jason: My feeling that, is that we should take every step to ensure a better release, however if from a sheer scope of resource effort, we need to draw a line to say we're going to move forward and change things next time.

Megan: As it stands, we agree 2.5 is ready, but we don't all seem to know how to fix the problem eventually.

Peter: 2.4 got better because people starting running it in production.

Jason: We strategically planned going into production about 30 days after Cape Town.

Michael: Not going to solve all problems before 2.5 but need to start a broader discussion of how to improve these issues going forward. Personally, liking the plan of calling this a beta, and putting together some guidelines of moving forward to the real release. Would like to have 2.5 process wrapped up by the end of January. Would be nice if northern hemisphere schools were running the same tag in the summer.

And that was documented on the foundation website as something we encourage people to take.

Dave: Who is going to maintain this newer branch (we can)

Peter: We have you pencilled in.

Jason: We can help.

<<talk about people who can contribute resources>>

<<hands tired, just listening>>

Michael: Will need to, over the next week, to wrap things up so we can make an official decision. Please send your concerns, suggestions, etc. to Megan over the next release.

Megan: Need to find out who will be available for testing. Will talk to CapeTown to work on appropriate times to cut the beta since they are pioneering it. Will write this up in a document. Then need to let people know ASAP about this.

Would be good to let people know things in the next 4 days or so.

Mostly need to flesh out when we are going to do this tag and the goals we want to see happen between the beta tag and the final release.

Michael: Makes sense, that's what needs to happen before making a broad communication.