Page Comparison

...

Please see the Administrator Guide for information on System Settings, Creating Scales and Setting up Permissions.

A. Create template and add/remove items

1. Create template.
2. Add several items from "My Items" bank of questions.
3. Remove one item.
4. Exit edit mode and re-enter template and confirm the item was removed.

B. Test drag and drop re-ordering

1. Try to reorder the items by using the drag-and-drop UI.
2. Click "Save Order."
3. Exit edit mode and re-enter template and determine if order was saved.

C. Test drop-down menu re-ordering

1. Go back and edit template.
2. Try to reorder using the drop-down menu box for the numbered items.
3. Exit edit mode and re-enter template and determine if order was saved.

D. Add new questions and test drag and drop re-ordering

1. Try to add new items (versus selecting existing items in "My Items" bank of questions)
2. Try to reorder them by using the drag-and-drop UI.
3. Click "Save Order."
4. Exit edit mode and re-enter template and determine if order was saved.

E. Add new questions and test re-ordering within a group using UI drag-and-drop

1. Go back and edit template.
2. Try to reorder using the drop-down menu box for the new items.
3. Exit edit mode and re-enter template and determine if order was saved.
4. Create question group and test ordering
5. Click the checkbox next to several items using the same evaluation scale and then click the "Create Question Group" button.
6. Preview template to be sure blocked items are displayed correctly.
7. Try to reorder the items inside a bock of items by using the drag-and-drop UI.
8. (There is no need to click the "Save Order" button, because the changes are automatically saved; the "Save Order" button is disabled.)
9. Exit edit mode and re-enter template and determine if order was saved.

F. Test Instructor items on evaluation

1. Edit a survey question and click the "Evaluatee/Instructor" radio button in the "Category" area.
2. Save the question.
3. Preview the survey and confirm that the question for the "Instructor" is showing up for each of the preview instructors.

G. Test Creating a Group with a blank item header text

1. Select several items with the same rating scale.
2. Click "Create Group".
3. Leave Item Header Text blank and see if system lets you enter it with a blank.
4. Should be presented with the screen again where the Item Header Text box is still blank (expecting to be filled in).
5. Test "Cancel" button.
6. Type in header text.
7. Click Save item.

H. Test hierarchy template features

1. Go to Administrate.
2. Check the "Use Hierarchy menus, options, and data in the system" checkbox and then click "Save Settings."
3. Click on "Control Hierarchy".
4. Click on the "Add" link next to the "Root" hierarchy level.
5. Name the node "Department 1" and click "Save Item."
6. Repeat those steps to create another node immediately under "Root" called "Department 2".
7. Go back to edit template view. 
8. Add node assignments to two different items.  Edit the item and choose "Department 1" from the "Hierarchy Assignment" menu for one item.  Choose "Department 2" for a different item.
9. Preview template to be sure the order of items is displayed correctly and in the correct nodes.

I. Set up courses, instructors, and students

1. Set up two Sakai course sites, eval_test_1 and eval_test_2.  (additional course sites may be added now to fulfill requirements for sections N - P below)
2. Add an instructor level account (ex. instructor1) to eval_test_1 as Instructor; add two instructors (ex. instructor1 and instructor2) to eval_test_2.
3. Add at least one student account (ex. student1) to each course. (Multiple students should be added to the courses to fulfill requirements for sections N - P below)

J. Set up evaluation

...

General Information

The Evaluation System is used to conduct formative and summative (End of Term) evaluations and review the results. An evaluation consists of a series of scaled and/or free text questions, which are delivered to students over a fixed time period (typically a few days). This guide will explain how to create items (questions), templates, and evaluations, deliver the evaluation and view the results. Permissions have been assigned for the "maintain" and "access" roles.  Instructors with the maintain role can create a template, assign an evaluation to a template, and be evaluated in an evaluation.  Students with the access role can complete and submit an evaluation.  Test IDs should be created for an Instructor and Students to assist the tester with this effort.

 The test plan is setup to record results from multiple testers.  It's advisable that you enter your Name, Date, Tag, and Browser information in the heading.  There is also a column to enter JIRA ticket numbers for any issues you create, the Operating System (eg, Windows 7, Mac OS 10.6) and Browser (eg, Internet Explorer 9.0, Firefox 10.0.2) you used to test with and any other pertinent information you have into the Notes section.

Overview of Evaluation Process

The process of using the Evaluation Tool involves the following general steps:
1. Create your items (this is optional as you can also create items while creating a template)
2. Create a template
3. Add items to your template
4. Create an evaluation
a. Set the parameters for the evaluation
5. Run the evaluation
6. View the results

Each of these steps are explained in the following sections.

Preparation

If you are new to QA testing for Sakai, there are some things that you do before you begin:

Creating test accounts and sites on the Sakai QA servers

• The QA5 server has been used in the past by UM for testing Release candidates.  It can be found here:
  Quick link: http://qa5-us.sakaiproject.org/portal
• Login to the Qa5 server: 
  • UserID = admin
  • Password = (traditional Sakai login)  contact Cathy Crouch if you need additional information (crouch@umich.edu)
• On the Sakai QA servers, each user can create his or her own test accounts and sites.
• Test accounts have been setup for the following IDs:  Instructor01, Student01, and Student02.  See the Information page of the test plan for details.
• Navigation for Teaching Evaluations for the 1.4.x Release:  Click on "More Sites" at the top of the page > "TQ-test" site > Evaluation System tool
• Set up 2 Sakai course and/or project sites, i.e. EvalTest01 and EvalTest02
• Add an instructor to each site
• Add at least 1 student to each site.  Multiple students should be added to at least 1 site to fulfill the requirements for testing as a Teaching Assistant and to simplify the process of testing many student test conditions.

Evaluation Dashboard

The Evaluations Dashboard provides a summary of group information and the current status of your evaluations. Links at the top of the page when selected will display the following:

• My Evaluations
• My Templates
• My Items
• My Scales
• My Email Templates
  • + Add Template
  • + Add Evaluation

You'll also see two boxes on the screen which are described below:

• Site/Group Listing: displays the groups that the instructor can be evaluated in and the groups they can evaluate.
• Evaluations I am creating or administering: displays the Title, Status and Date of the instructor's evaluations
  • Clicking on the Title of the evaluation will bring up a preview of the evaluation.
  • The Status column will indicate whether an evaluation is active or if the results are viewable. You can click on the View Report link to gain direct access to the report for that particular evaluation.
  • The Date column will indicate the due date of the evaluation or the date of the when results become viewable.

Test Plan

All of the functional testing features are listed in the Google workbook named TQ 1.4 Release - Test Plan*.  Each worksheet identifies a feature to test and contains one more sets of tasks as described below. Click on a feature's link to view test conditions for each step.

• Instructions- Instructions for completing the test plan and preliminary steps that are necessary to complete before testing features in the test plan.
• Create Items - Test scripts that focus on creating, modifying and removing items.  These are the questions or statements which the evaluator responds to.
• Create Templates - Templates store items in a layout and are used to create evaluations.  This page contains test scripts for creating and editing templates.
• Create Evaluations - Test scripts that focus on authoring assessments. The conditions will focus on creating new and updating previously creating evaluations.

K. Check evaluation

1. Log into Sakai (or super-user in) as "student1".
2. Click into the evaluations for each course.
3. Confirm that questions are in correct order.
4. In eval_test_1 evaluation, confirm that only "instructor1" item block appears.
5. In eval_test_2 evaluation, confirm that both instructors' item blocks appear.
6. In eval_test_1 evaluation, confirm that "Department 1" items appear and "Department 2" items do not appear.
7. In eval_test_2 evaluation, confirm that "Department 2" items appear and "Department 1" items do not appear.
8. Confirm that reminder e-mails are sent on schedule.
9. Confirm that reminder messages are not sent after a student completes an evaluation.

...