Page Comparison

...

User Story 9: Different evaluators with different evaluation devices/forms - UMich (Feb 2, 2008)

This user story is for the Undergraduate Research Opportunities Program. In this program, students are evaluated by both their faculty mentor and the peer advisor who runs a weekly seminar.

1. Student submits matrix cell for evaluation
2. Evaluator 1 (peer advisor roleadviser) fills out peer-advisor-form, which asks about things like the student's attendance and performance in the peer-advisor-led seminar
3. Evaluator 2 (faculty mentor ) fills out faculty-mentor-form
   remaining details tbd

User Story 10:  Two or More Evaluations Required- No Routing - IU (Feb 15, 2008)

1. role) reads peer advisor evaluation and takes it into account when filling out faculty-mentor-form. The faculty mentor's form covers all aspects of the program, both participation in the seminars (which the faculty mentor knows about through the peer advisor evaluation) and research work done directly under the faculty mentor.
2. Peer adviser meets with student to discuss both evaluations
3. After this meeting the peer adviser unlocks the cell, making the two evaluations available to the student.

User Story 10:  Two or More Evaluations Required- No Routing - IU (Feb 15, 2008)

1. Matrix/wizard administrator sets required number of evaluators per cell/page to 2.
2. Matrix/wizard admin selects evaluators (number of evaluators equals or exceeds number of required evaluations)
3. Student submits matrix/wizard cell/page X for evaluation
4. Evaluators who have opted in for individual or digest notifications, receive an email notification.
5. Evaluator 1 logs in to evaluator dashboard and selects the cell/page X.
6. Evaluator 1 writes and submits evaluation for cell/page X.
7. Cell/page X status is not set to complete because evaluation count has not met requirement in step 1.
8. Evaluator 2 logs in to evaluator dashboard. 
9. Evaluator 2 select cell/page X for evaluation.
10. Cell/page X disappears from the dashboard of all approved evaluators for this cell/page because quota will be met when Evaluator 2's work is complete.
11. Evaluator 2 writes and submits evaluation cell/page X.
12. Status of cell/page X is set to complete.
13. Student receives email notification.

...

1. Matrix/wizard administrator sets required number of evaluators per cell/page to 1.
2. Matrix/wizard admin selects evaluators for cell/page X.
3. Student submits matrix/wizard cell/page X for evaluation
4. Evaluators who have opted in for individual or digest notifications, receive an email notification.
5. Evaluator 1 logs in to evaluator dashboard and selects the cell/page X. 
6. Cell/page X disappears from the dashboard of all approved evaluators because quota will be met when Evaluator 1's work is complete.
7. At some point (the evaluation may be in-progress or resubmitted), Evaluator 1 determines that s/he is unable to complete the cell/page X evaluation or has a question for the matrix/wizard admin.
8. The Evaluator forwards cell/page X to the matrix/wizard admin.
9. A notification is sent to the matrix/wizard admin.
10. The matrix/wizard admin logs in a locates the forwarded item in his Forwarded Evaluations queue.
11. The matrix/wizard admin either answers question and returns cell/page X to Evaluator 1 for completion or (if Evalluator 1 cannot complete), releases cell/page X back into the Itmes Pending Evaluation queue. 

User Story 15:  Multiple Evaluators Use Evaluation Form with Clickable Descriptors to Rate Criteria and Goals and Calculate Average Ratings 

...

- rSmart (March 24, 2008)

A) Functionality that already exists in 2.5 (with customization of forms).

1. An evaluation form that uses .xslt to provide a scoring rubric in the form of matrix with clickable descriptors is associated with a matrix cell, wizard page. The evaluation form provides rows for each criterion (standard, outcome, etc.) and columns for each scoring level.
2. The cells in the matrix provide descriptors for student performance for each criterion at each scoring level. Evaluators click on one matrix cell per row to indicate the score they award to each participant for each criterion. The cell that has been selected as the score for each row is highlighted for visual confirmation that it has been selected.
3. Each time an evaluation form is submitted by an evaluator, the mean score across all criteria is calculated and displayed on the form for viewing by any user with permission.
4. The evaluation form may provide a comment area for evaluators to include comments.

B) Functionality desired for OSP 2.6 or beyond (with customization of forms).

1. Evaluation forms are also used to rate goals linked to a matrix cell or wizard page via the goal management tool.
2. Evaluation forms, specified evaluators, and the goal management process may also be used with portfolios via the association of evaluation forms with portfolio templates.
3. When evaluators add comments to an evaluation form instance, they may designate them as public or private comments.
4. When each evaluator submits an instance of an evaluation form for a matrix cell, wizard page, or portfolio, the calculations for each criterion or goal in that form instance, as well as the mean score for the form instance, are combined with calculations from all other evaluation form instances for the owner of that cell, page, or portfolio to provide a mean score for each criterion or goal and a total mean score across all evaluators.
5. The set of evaluators associated with a matrix cell, wizard page, or portfolio template can include one or more peer advisers whose ratings are advisory only and do not enter into the calculation of mean scores.
6. The calculation of the mean score for each criterion or goal identifies an acceptable range of inter-rater reliability for each rating and flags ratings that require a third raterevaluator.
7. A flagged evaluation form instance requires the submission of an additional rater evaluation form instance by a third evaluator before the cell, page, or portfolio can be complete. The additional rating may be preset to either substitute for the rating Ratings from the additional evaluator replace the scores from the evaluator farthest from the calculated mean or combined with existing ratings.The number and identity of the evaluators can be preset such that when score for the cell, page, or portfolio.
8. When all evaluators have completed their ratings, the evaluation results form completes the cell, page, or portfolio and is displayed to the owner of the cell, page, or portfoliobecomes complete and its contents and score cannot be changed.
9. The owner of the a completed matrix cell, wizard page, or portfolio can view combined ratings and public comments of all evaluators (with or without owners identified) according the process specified for the evaluation form.

Functional Details (may be added after community demo) (1)

See: Multiple Evaluator Workflow - Functional design

Describe any functionality not fully captured in the User Stories.

...

Include any ERDs, flowcharts, sketches, mockups, etc.Image Removed !Matrix cell - Two evaluator workflow.png !

Community Acceptance (4)

Indicate how this feature has been discussed by the larger community (e.g., list discussion, specific meetings, etc.). Provide specific records of community acceptance (e.g., list institutions and contacts who also identify this feature as a requirement).

...