Product council review: PC members Review

State of community product management functions

What do you think of the product development lifecycle?

Most Product Council members are strongly supportive of the product development lifecycle as a necessary part of a mature community process and a useful framework within which to set expectations, particularly for 2.x. It has the virtue of calling more attention to how projects get started, with the aim of raising issues before it gets too late. One member notes that it's fairly aspirational, however, and it hasn't been a comfortable fit for how development teams have gotten things done. It also seems to have as many interpretations as interpreters.

However, other processes were offered as options:

• The process encourages "one big release", neglecting a rich ecosystem of optional components. Another model is to have only the kernel supported by the community teams, and tools to be on an independent cycle.
• The community might provide some guides to help adopters evaluate ancillary projects. Shepherding boundary definitions and stewarding boundary crossings should be part of the Product Council's charge.

And refinements suggested

• Perhaps the Product Development phase may need a couple of flavors, because a dynamic system will have relatively mature parts still gaining new capabilities.
• The product development lifecycle currently described for Sakai seems better tailored to Sakai 2 development than Sakai 3. If we are not going to make large interventions in the way Sakai 2 is developed, maintained, and released, than the lifecycle is probably adequate for Sakai 2. My hope is that Sakai 3 will follow a different path.
• I also think we should change the name of the "maintenance" phase to something that conveys its true nature, the "production phase". Differentiating  "maintenance" from "production ready" is the right distinction.
• Incubation needs to be reconsidered.

What are the critical functions that the community needs in the context of managing the product?

The Product Council members identified a comprehensive set of processes and areas of focus: a clear vision, clear standards, common tools, reachable goals and timelines, broad community support and tangible participation/contribution, straightforward setup, evidence of scale and load testing, concise marketing, and outreach conducted by many community members as well as by Foundation staff. Alignment with users needs is a priority.  
Processes

• Makes sure that the product being developed is intended to meet needs that users have prioritized as important, will meet those needs, and will be consistent with other projects released by Sakai.
• Development project has sufficiently broad community sponsorship to ensure that there is strong enough need to drive support commitments
• Ensuring that each project is meeting community-defined standards for things like coding, QA planning, documentation, etc.
• Facilitating community conversations around the coherence of Sakai (2.x or 3.x) as a whole, beyond the basic standards for each discrete tools.
• Strong project management that can cope with unstructured projects and product management to maintain a roadmap and overall vision
• Sakai should have open, transparent, and effective governance processes.

Attributes

• Coherence: Sakai should demonstrate overall coherence, including coherence across the Sakai 2 > 3 boundary.
• Roadmap: Sakai should develop and work towards a roadmap that reflects aspirations as well as deeds.
• Quality/standards: Sakai should publish and adhere to clear standards and standard practices for a wide variety of characteristics, including accessibility, documentation, internationalization, maintenance, security, technologies, and user experience.
• Communication: Sakai as a community should clearly and regularly communicate on open channels about its qualities, activities, processes, and plans.
• Distribution: Sakai should provide distribution mechanisms that make it easy to find and consume our products.
• Security: Sakai should provide vigilant, credible, obvious processes to collect, review, address, and disseminate security issues.
• Quality Assurance: Thorough and repeatable testing
• Release Management
• Code Maintenance
• Easily installable and configurable 'basic' install. Availability of a highly customizable 'enterprise' install.
• Documentation: thorough and current documentation, both user facing and technical

For which of those critical functions, if any, do you think a product council is needed?

The Product Council members uniformly believe that these functions are best accomplished by having a coordinating body. Having a group do this oversight rather than an individual aligns with the Sakai community's values. It is less clear how specific the Council's work should be, and how much authority the Council should have. Several proposed models highlight the differences of approach:

• The PC is a mechanism for facilitating the development and uniform implementation of community-generated goals and standards.
• For none of them directly. For all of them indirectly, by codifying community standards in those areas, and advising projects in their maturation and planning
• The Product Council should participate in shepherding the development and stewarding the maintenance of all the activities listed above. Yet the Council should understand itself to be most highly engaged in product governance, where it should be the most visible and active group. The Council should also be highly engaged in the stewardship of Sakai's roadmap and the product's coherence and quality as it travels along that path.
• A body, charged with reviewing practice against standards, can be very effective in holding a group to its own standards.
• The PC's role is best suited as a broad surveyor and facilitator. That is, discussing coherence, incompatibilities, risks and opportunities, and communicating them to project teams and institutional leaders.

How does the product council relate to other groups working on these critical functions?

The Product Council members responded that coordination with other groups has not been done well. They note that intergroup boundaries are unclear between many groups, and offered ideas for how to improve the Product Council's working relationships in the future:
Product management should have a lot to say and some authority early in the life cycle of any project that has started on the journey to be included in "core", and less o say, mostly in an advisory capacity, later in the development and life cycles.

• We should make the jobs of QA and maintenance easier by, for example, making sure the development teams start with good QA plans, think about test automation when they design their code, write adequate documentation, and so on. By the time actual QA and maintenance decisions are being made, those should by and large be made by the experts on the QA and Maintenance teams, with occasional advice from the PC.
• When considering the retirement of some bit of functionality for the PC to advise the maintenance team on whether we think doing so will leave an important functional hole or create some inconsistencies in the Sakai release as a whole.
• Continued integration with the Teaching and Learning Group.
• Facilitate the relationship between development teams and those groups, particularly the community expectations that emerge in the practices of these groups.
• Part of the Sakai Product Council charge was to judge work against community standards. The charge to establish these criteria was inconsistently interpreted (or perhaps just inconsistently adhered to). It is unclear whether the Sakai Product Council is tasked with documenting the community standards, or only with weighing projects presented against standards documented by other groups in the community.
• The PC should provide broad institutional memory and seek to make groups aware of things that seem to be missed. It should not be a mediator.
• The Council should not direct the activities of these other key groups, but should be more aware of and engaged in the work of these other teams, perhaps even assigning its own members to serve tours of duty with these other groups, or incorporating representatives from other groups into itself.  Ideally, the Council will assist these other groups by doing its own job and helping them resolve issues that call for product governance.

Which of those functions are missing from current activity or processes?

All of them

The mission and charter of the Product Council

When you heard about the product council, what did you hope the product council might achieve?

Product Council members describe a role that provides planning, consistency, transparence and predictability to the software. They describe the role of the Product Council to develop a clear roadmap for the software, and to uniformly enforce community developed standards for tools. It is envisioned as a locus for community discussion about issues related to creating coherent and cohesive products, and as an advising body to help assure this happens. The diverse Product Council membership would be brought to bear for the benefit of the product. One member described the role as getting out ahead of problems while projects are in R&D, rather than having the problems surface during production.

How close to your hopes did the product council charter come?

Most members responded that the Council has made a good start, but hesitancy has caused slowness in its work. There is a need to clarify what the Council will and will not do.

What would you change about the charter?

In general, Product Council members think the charter is fine as a mission statement. As one reviewer commented, "The Council should crystallize community scrutiny of the Sakai roadmap, and the coherence and quality of the product. Secondarily, the Council should foster the formation of community standards and practices, and help guide projects toward meeting those goals for coherence and quality. In essence, the Council should act and appear as if it were a microcosm of the larger community, but with the ability to make final decisions where community consensus has not brought closure."

Several pointed out the need for clarity in expected actions, purview and outcomes.

Regarding the membership of the Council, this is generally perceived as right, and a few suggested that additional points of view, e.g., from the Maintenance Team, might be included. Several mentioned the need to appoint the Product Manager as chair.

What impression, if any, do you think the product council has on people looking at Sakai from outside the community?

This is widely seen as providing a positive impression of Sakai because potential adoptees on the outside want to be able to see that there is some reasonably clear process by which the community defines direction and enforces standards. The PC is part a symbol of that and part an embodiment of it. One of the fundamental differences between Sakai and alternative platforms is Sakai's transparent, structured governance. The Council is the clearest expression of Sakai's different model, and reinforces the community values for software development and delivery.

One reviewer commented that this is likely a false assurance.

Membership

Have we got the membership right? What constitutes the right mix of people on the Council? How should members be selected?

The members are generally perceived as the right mix of experts, though cross representation with other working groups should be considered.

There are mixed opinions about whether to appoint members to assure the right mix, or to elect members to assure community representation.

Should Board members be on the Council?

Product Council members unanimously comment that it doesn't cause a conflict of interest to have cross membership between the Board and the Council. The responsibilities and actions of the Board and Council do not overlap. Several cited concerns about individual bandwidth to participate in both groups.

What expectations should there be on council members?

In addition to citing the ability to meet commitments made to the Council and regularly participate, they need to be actively engaged in Sakai communities that are collectively working to collect or produce insights, standards and practices that inform the product. Members should facilitate the work of the Product Council in the community.  Most responders noted the time commitment is significant, and that establishing priorities for the Council work would help prevent the members from bottlenecking production work in the community through slow response. A need for active leadership was noted.

What the Product Council has done

What do you think the product has focused its attention and energy on?

• The main effort was evaluate submissions for the core in release in 2.7.
• Getting its footing (e.g., establishing it's identity and interaction style; dealing with community ambivalence; not over-stepping or expanding its charter);
• Establishing requirements for incubation; the 2.7 release;
• Imagining the implications of moving Sakai 3 from R&D to production.
• Understanding what projects would ideally supply to the community in terms of documentation for an almost-mature project.

What do you see as the successes of the product council so far? What are the disappointments?

Successes

• The Sakai 2.7 Technical Elements and Interoperability Worksheet. The "Dry Run" sets of answers for incubation projects available as child pages to the 2.7 Exercise page permitted frank and focused discussion on these tools.
• Intelligent review of the 2.7 release. Although incomplete and tardy, this process means that for the first time, the projects in the release were held accountable to formal product governance by the community.
• The initial steps in (a) ensuring quality and cohesiveness, (b) determining which projects go into a release, and (c) informally advising projects.
• The Product Council was able to come up with a rather good way of doing technical reviews.
• The Product Council has also provided a forum for discussions that have not been able to get off the ground before. I think the very productive work of the T&L group in recent months is owed at least in part to the fact that the PC exists.
• The product council has provided (and will get better at providing) some structure that helps make sense of the Sakai 3 work and how it can be ushered from a small R&D project toward an acceptable community release.
• The clarity around what the PC would like to see from projects offers some consistently formed information to be perused, not only for PC evaluation, but by community members making adoption decisions. This is a success in that adopters can understand what's coming in a way that was more difficult before.

• Many of the conversations about criteria served to codify community expectations around quality and coherence. This process is working toward a shared vision of how product success is defined.

Disappointments

• Slow to get traction
• The Product Council has been fairly passive - a place to report to and talk about issues, and only more rarely a constructive force.
• The Product Council has made very little progress toward codifying or clarifying objective standards for its decisions, apart from the technical review mentioned above.
• Stepping on the toes of teams who were already successfully shepherding work.
• Not having written, commonly-recognized gating criteria for moving from R&D to incubation, thence to production/maintenance and finally to managed end-of-life.
• The PC has not yet engaged with a full product lifecycle as the Sakai 2 tools were largely already developed, and the Sakai 3 lifecycle is still in incubation. So the strengths and weaknesses of the model are still largely to be seen.
• Work was less complete in the "Documentation and Community Support" and the "Interaction, Visual Design, i18n and Accessibility" set of criteria. A project wishing to move to production does not have clear guidelines for meeting the community expectations on Documentation, Community Support, Interaction, Visual Design, il8n and Accessibility.

Misunderstandings

Some felt the criticism of the product council to be unfair. One reviewer stated, "Product management has to be done in the context of a release. We have had only one release in the last year. We are gearing up for a second and third. I don't believe that the PC should be doing terribly much more than it has done so far later in the release cycle. For 90% of the decisions where the PC should play a role, if you have to ask for our input during QA, then we have probably already failed. Our main job is to make sure the homework is done coming into a release. Our job is more to prevent problems in development, QA and maintenance than it is to fix them once they arise."

• The forum that the Product Council provided has been something of a catch-all for product issues that weren't otherwise being addressed. Most particularly, there was an expectation that the PC would be an operational help with the release cycle.
• Many community members were confused about the role of the PC. This caused a number of negative effects, from misplaced hopes, to some paralysis amongst teams (RM/QA/MT) that were confused about what the PC would be doing in their area. This role clarity issue must be resolved.
• There has been a fair amount of misunderstanding and at times sniping about the PC. The PC has been regarded as both not doing enough (people seemed to think it would both do things that had historically been done by other groups, and that it would or should solve other existing unsolved issues), and as doing too much or having too much authority (where it's typically perceived to have more formal authority than it actually does have).

   

What has been the net pay-off of the product council thus far? Is the Sakai 2 product better off? Sakai 3?

The Product Council members unanimously responded that the 2.7 release is improved because of its work. The improvement is in the form of better, clear process, a set of questions that allow developers to check if they are on track for community expectations, and of an evolving governance model that will provide community direction for the product in the longer term.

Most comment that it is too early for there to be impact on Sakai 3 as a product, though there is expectation that the Council work will be of benefit to Sakai 3 in the future.