T&L Capability Review Process

Summary

The idea behind this review process is to leverage the collective knowledge of the Sakai Teaching and Learning community by having them review Sakai Tools and provide feedback, in the form of a brief reports, on both the tools capability's with regards to supporting teaching and learning activities as well as the tool's usability (ease of use of its interface). Those involved in this review process would adhere to the following guiding principles:

To the extent possible, recommendations for capability or user interface enhancements will be based on user research, feedback and testing.
Thoughts and ideas will be surfaced often and openly as means to gather broad input from the community.
Those leading technical development work as well as the larger technical community within Sakai would be engaged throughout the process as collaborative partners on the review.
Those involved in the review process will commit to helping to seek the resources needed to develop any new capabilities or other enhancements with the understanding that requests for improvements without additional resources represents an "unfunded mandate".

The goal will be to provide a feedback loop into the development process so that as new work is planned the ideas from the larger teaching and learning community can be considered. At the same time, we hope to use the final report as means to engage institutional decision makers and work to secure the resources needed to support the development of the recommended changes. We would generate data/information for the report those a range of collaborative activities including:

Teaching and Learning User Research - the goal of this work would be to identify new capabilities that, if introduced, could either allow the tool to be used for a broader set of "traditional" teaching and learning use cases OR support new and innovative approaches to teaching and learning. One approach that we may take for this work is a series of online sharing exercises, which would be recorded for future reference, in which folks with more experience using the tool would share examples of their work, best practices, challenges, etc. with the larger group as means to stimulate discussion of new capabilities that would be beneficial. We would also observe and research instructor and learner behavior as means to better understand their needs.
Light Weight Usability Testing- We are a bit less sure of how to proceed with this type of review but have discussed conducting lightweight usability studies on our campuses and then working to pool the outcomes from these into a collective summary of usability issues. If designer were available to help create wireframes the group could also provide feedback on such concept materials. We should also consider reviewing "accessibility" and internationalization issues in this type of review.

Once the review process was completed we envision bringing the reports we might generate to others in the community, particularly the TCC, as means to get their input and help in identifying technical strategies and the time/effort that these would require. This will help us prioritize the plans while also giving us a sense of the resources needed to not just implement the enhancements but also maintain them over time. This information could then be used to help secure the resource needed for the work.

Phases of Capability Review Process

Start Up Phase - This would be an initial period when we would meet with those who are leading the development effort on a tool or capability to have an initial discussion around their thoughts and areas of interest in terms of getting feedback from the T&L group. We would also use it as a time to communicate out broadly regarding our plans to engage in a capability review as means to get others involved.
Phase One: Preliminary Capability Enhancement Ideas - This would be an early stage brainstorming activity that would span a relatively short period (2-3 weeks maybe) that would try and generate as many ideas as possible for both enhancements to existing capabilities, entirely new capabilities and usability (UI/UX) improvements. We would encourage participation from as many people as possible with the idea that the T&L Group might discuss these as means to develop initial thoughts around prioritization.
Phase One: Community Check-In - This would be a formal period of time when we could pause in adding to our initial list and focus on getting preliminary feedback from those leading development, from the TCC/PMC and institutional decision makers. In particular, we would be looking for technical folks to surface concerns, issues or challenges that might help us decide which ideas to focus in on.
Phase Two: Usability Testing and Capability Reviews - This would be a period when we would engage in the types of work outlined above which might include lightweight usability testing and identify capability enhancements. We would work to remain in direct contact with tool developers on questions that surface but would also make sure the process was open to all and transparent.
Phase Two: Community Check-In - This would be a similar check-in as during phase one but would be more specifically focused on assessing technical issues and scoping the time, effort and resources needed to implement the proposed enhancements.
Phase Three: Draft Capability Review Report - We would develop and release a report that would include recommendations for future development work and the resources needed to do this work.
Phase Three: Community Check-In - This would be a final check-in period during which people could question and comment on the draft report
Phase Three: Release Final Report and Seek Resources - We would release the report broadly and work both at our local institutions but also within the community to engage institutional decision makers as means to secure the resources needed to implement the enhancements.

Draft Report Outline

Executive Summary

Capability Overview - What is the tool intended to be used for and the basics of how it works

Capability Use Cases, Best Practices and Tips - This would be a series of short screen movies that would be used to demonstrate different use cases based on real world application of the tool. Workarounds that users are developing could be highlighted as means to show where new development effort might be useful.

Suggestions for Capability Enhancements - This would be a list, possibility also placed in JIRA, of enhancements that the T&L has reached consensus on as being important for future development effort. We might also prioritize these to provide feedback on which were the most important from our perspective.

Suggestions for UI Enhancements - We would like to produce wireframes that provide visual examples of interface changes that we would recommend based on direct feedback from users as well as light weight usability studies that we would run on several campuses.

Technical Assessment and Resource Requirements - This is just a rough idea at this point but it would be very useful if more technically savvy Sakai folks could review the report and provide an assessment of what the best technical approaches would be to address the different suggestions and what types of resources (people time) would be needed. This information would allow the T&L to engage with their local institutional decision makers as means to help find resources needed to support the development effort.

The current Sakai Teaching and Learning group has started to engage in this type of work which is being documented on the Lessons Tool: Draft Capability Review Process wiki page.