Page Comparison

...

1. Matrix/wizard administrator sets required number of evaluators per cell/page to 1.
2. Matrix/wizard admin selects evaluators for cell/page X.
3. Student submits matrix/wizard cell/page X for evaluation
4. Evaluators who have opted in for individual or digest notifications, receive an email notification.
5. Evaluator 1 logs in to evaluator dashboard and selects the cell/page X. 
6. Cell/page X disappears from the dashboard of all approved evaluators because quota will be met when Evaluator 1's work is complete.
7. At some point (the evaluation may be in-progress or resubmitted), Evaluator 1 determines that s/he is unable to complete the cell/page X evaluation or has a question for the matrix/wizard admin.
8. The Evaluator forwards cell/page X to the matrix/wizard admin.
9. A notification is sent to the matrix/wizard admin.
10. The matrix/wizard admin logs in a locates the forwarded item in his Forwarded Evaluations queue.
11. The matrix/wizard admin either answers question and returns cell/page X to Evaluator 1 for completion or (if Evalluator 1 cannot complete), releases cell/page X back into the Itmes Pending Evaluation queue. 

User Story 15:  Multiple Evaluators Use Evaluation Form with Clickable Descriptors to Rate Goals and Calculate Average Ratings - rSmart (

...

Feb 18, 2008)

1. An evaluation form providing a scoring rubric in the form of matrix with clickable descriptors is associated with a matrix cell, wizard page, or portfolio. (The basic functionality for this requirement already exists in OSP 2.5.)
2. The evaluation form identifies a specified set of evaluators (by role or username) to rate the cell, page, or portfolio.
3. The set of evaluators may be organized to require any number of ratings per cell, page, or portfolio.
4. The set of evaluators can include one or more peer advisers whose ratings are advisory only and do not enter into the calculation.
5. If multiple goals or standards have been associated with the cell, page, or portfolio, evaluators rate each goal/standard.
6. The evaluation forms for each evaluator feed into an evaluation results form associated with the cell, page, or portfolio.
7. The evaluation results form includes a calculation field to provide the mean rating for each goal/standard across all evaluators. This field provides a cumulative total across evaluators as ratings are added.
8. The evaluation form provides a comment area for evaluators to include public and private comments. The evaluation results form displays the public comments.
9. The calculation field identifies an acceptable range of inter-rater reliability for each rating and flags ratings that require a third rater.
10. A flagged evaluation form requires an additional rater before the cell, page, or portfolio can be complete. The additional rating may be preset to either substitute for the rating farthest from the mean or combined with existing ratings.
11. The number and identity of the evaluators can be preset such that when all evaluators have completed their ratings, the evaluation results form completes the cell, page, or portfolio and is displayed to the owner of the cell, page, or portfolio.
12. The owner of the matrix cell, wizard page, or portfolio can view combined ratings and public comments of all evaluators (with or without owners identified) according the process specified for the evaluation form.

...