Evaluation Requirements

Information

This is the list of requirements for the evaluation system Sakai version 1.0.
The requirements are listed in criticality and dependency order.
All interface layouts and linkages are defined in the wireframe.
The requirements are marked by criticality:

• C - Critical (for version 1.0)
• E - Essential (for next version)
• D - Desirable (for future versions)
• X - Not Needed

Requirements list

Administrative Features/Requirements:

Create, edit and modify templates (C)

Administrators with appropriate permissions can create, edit, and modify templates beneath their hierarchy level. Any level admin (including instructors) can create a template. Templates can be shared based on the rules in the template sharing requirement.

Create, edit and modify evaluations (C)

Administrators with appropriate permissions can create, edit, and modify evaluations.
Any level admin (including instructors) can create an evaluation.
Only administrators can create official evaluations (where results are used for tenure or evaluation).

Set evaluation start and end dates (C)

Administrators may adjust the opening and closing dates of any evaluation and send automated reminder emails to non respondent students (beneath their hierarchy level).

Assign evaluations to worksites (C)

Administrators may assign an evaluation to any group or worksite depending on permissions.

Admin roles and access (C)

The following levels of access are built in the system:

1. System Administrator: Maintains the system, controls site settings, controls users/courses/scales/hierarchy
2. Institution Administrator: Top hierarchy level, administers institution wide evaluations
3. School Administrator: Mid hierarchy level, administers school wide evaluations
4. Department Administrator: Low hierarchy level, administers department wide evaluations
5. Instructor: Contributes to the process, creates templates and evaluations
6. Student: Fills out the evaluation
   Each level of access can be set to readonly or full access.

Admin read and write (C)

Administrators can be set to have read only permission to a level of the hierarchy or to have full read/write permission. This should be controllable from the administrative portion of the system.

Hierarchy control (C)

System administrators can create and customize their own hierarchy through the system admin interface. (This will hopefully be taken over by Sakai at some point)

Admin hierarchy of a configurable number of levels (System, Institution, School, Dept, Instructor, etc...) (C)

The admin hierarchy controls access to evaluation results and the ability to create evaluations in their domain. Samples below show a 5 level hierarchy.
System - Can view all evaluations in the system and has the ability to create/edit/delete any template. Controls users, courses, hierarchy, and scales. Sets system settings. The system admin may also generate any type of report on any evaluation data in the system.
Institution - Create evaluations which can be assigned to any school, dept. or course; view results of any school, dept. or course, edit any template owned by the Institution level, view results for questions created at the Institution level for any evaluation created at the Institution level, view official evaluation results for any school, dept. and/or course, ability to create official evaluations
School - Create evaluations which can be assigned to any dept. or course within this school, view results of any dept. or course within this school, add questions to Institution level evaluations which include this school, view results for questions created at the school level for any evaluation created by this school, view official evaluation results for any dept. and/or course in this school
Dept. - Create evaluations which can be assigned to any course within this dept., view results of any course within this dept., add questions to Institution or school level evaluations which include this dept., view official evaluation results for any course in this dept.
Instructor - Create an evaluation which can be assigned to any course this instructor is teaching, add questions to any higher level evaluations assigned to any course this instructor is teaching, view official evaluation results for courses this instructor is teaching

Assign evaluations to worksites based on hierarchy (C)

The evaluation may be assigned to students by the admin in their own domain. For example the admin with subject code type access may only assign the evaluation to the courses of that subject area while the admin with school code level access may assign the evaluation to all courses of that school (several subject codes).
The evaluations can be assigned to any courses below the hierarchy level of the admin creating the evaluation (owner) within the current term.

Control ah-hoc groups

There should be an admin insterface to control (create, edit, remove) ad-hoc groups. Ad-hoc groups can be created by anyone. Groups should be treated the same as a course/worksite and should be able to have group admins who can control the group. The owner (creator) of the group should always have the ability to control it. Groups should be able to be placed into the hierarchy.

Assign evaluations to groups

Evaluations should be able to be assigned to groups the same way they are assigned to courses. Groups should NOT be assigned to an evaluation which is assigned to their parent container, this only works for worksites (maybe make this an option when assigning).

View evaluation results (reporting)

Administrators can view, print, and download (in pdf, excel and csv) results of any evaluations after they are closed. Super admins may also view results while the evaluation is in progress.

Optionally require all non-text questions to be answered

Administrators may choose to require all non-text questions to be answered before the evaluation is submitted. Students are always warned about non-text questions that they have left blank. This option may be turned on for the entire system or for a single evaluation.

Allow instructor added questions (system option)

Administrators can optionally allow instructors to add their own course specific questions to the bottom of the evaluation.
Any hierarchy level below the hierarchy level creating the evaluation can add questions to the evaluation. These questions will only be presented to the students in the courses beneath this level. Questions may only be added up to the start date of the evaluation. A configurable number of questions may be added. The final evaluation that is taken by the student is constructed from all questions added at all levels of the hierarchy. Only questions added within the appropriate school/dept/course are visible to students in courses within the college or dept.

Users control (C)

Users come from the Sakai framework. User settings and permissions related to evaluation are controlled via this interface. User settings not stored in the standard Sakai user are stored in the local Sakai User object properties.

Courses control (C)

Worksites come from the Sakai framework. Worksite settings and permissions related to evaluation are controlled via this interface. Worksite settings not stored in the standard Sakai worksite are stored in the local Sakai Site object.

Scales control (C)

Scales can be created, modified, or removed via the scales interface. Scales may not be removed or modified if they have been used. Anyone can create a scale but only administrators can create scales which are visible system wide. System/Institutional administrators can control expert scales as well.

Terms control (C)

The terms (semesters) come from the Sakai framework. Settings (like the current term) come from the Sakai framework.

Ability to control whether all instructors must be evaluated

On a course by course basis, an admin can control how the instructors in a course are evaluated. The choices are: all instructors required to be evaluated, only primary instructor evaluated (defines the primary when this choice is made), and a per instructor setting (each instructor is marked as required, optional, or no evaluation).

System specific settings

There are a number of system wide settings which are configurable including: Allow instructors to create evaluations, Allow instructors to view results, Control the number of questions instructors may add, Require students to answer all non-text questions, Allow students to modify responses up to the due date, Allow use of Not Available in questions

Template/Evaluation Features/Requirements:

Four question types (Scaled/Survey, Block of questions, Short Answer/Essay, Text Header) (C)

Evaluations allow for 4 questions types:
Scaled/Survey - Some scales available by default, scales can be added by System admins via scale control or individual users for personal use
Block of questions - This allows a set of questions to be defined which have a common scale and are displayed using a configurable set of indicators
Short Answer/Essay - Provides a text space for the student to enter an answer, the size of the space is configurable on a per question basis
Text Header - Special type, this is not actually a question, it is simply text that is placed in the evaluation as a header for a set of questions or additional instructions

Templates can contain any number of questions (C)

Templates have no limit to the number of questions they can contain. Question blocks which share a scale can also be created within templates.

Items can be identified as course or instructor related (C)

During the template creation process, the owner marks questions as instructor related or course related. Course related questions are grouped at the top when the evaluation is presented to the student, instructor related questions are automatically grouped, labeled, and duplicated for each instructor in the course. Instructor sections may display an image of the instructor if there is an image of that instructor defined in the system.

Items can be categorized for a configurable set of types

During the template creation process, the owner marks question items as related to a category that is customizable per institution. These cateogorized items would be grouped and duplicated based on data from the hierarchy (like the instructor/course items).

Templates can be created by any level of the admin hierarchy (C)

Any administrator can create an evaluation template. These templates are then used to create the evaluations themselves. Templates may be kept private, shared globally, or shared with the level of the hierarchy that user is in.

Items may be displayed using a variety of presentation options

Question display options are controlled on a per template basis. For scaled questions the template owner may choose from compact, compact colored, full, full colored, stepped, stepped colored, vertical, and vertical colored. Colored items use ideal coloring which indicates where an ideal answer is using coloring. A Not Available choice may be added to any scaled question. For essay items, the owner may choose to provide one to five lines of text (the user may still enter as much text as they like, but the box the text appears in will be limited in size.

Existing templates can be combined to create new templates

A user may use any owned or shared template to create a new template or simply insert that template content into an existing template. This allows a department to create a standard set of questions and then insert those questions into the end of term evaluation each term without recreating them. It also allows instructors to create a set of questions and insert those into the end of term evaluation without recreating them.

Evaluations can be assigned to any courses below the hierarchy level of the admin creating the evaluation (owner) within the current term (C)

When assigning an evaluation, the hierarchy determines what courses the current admin can assign that evaluation to. An admin may assign an evaluation to any course under them in the hierarchy.
For Example:
The admin for the school of engineering could assign an evaluation to all courses in CS and EE (departments within engineering). They could also assign the evaluation to a selection of courses from the OE dept. (also in engineering) at the same time. They could not assign the evaluation to any courses in the Math dept. because it is part of the school of Arts and Sciences. They also could not assign it to courses from the previous term.

The evaluation owner sets the active period and assigns the evaluation to courses (C)

The creator of the evaluation (also the owner) sets the active period (between the start and end date) for the evaluation. Up to the start date, the evaluation may be modified (and the courses it is assigned to may be adjusted). Once the start date has passed students may begin taking the evaluation and no further changes can be made to it. Once the end date has passed, students may no longer take the evaluation and reports can then be run against the results.

Evaluations are assigned to hierarchy levels or courses using a tree view (C)

A tree view (similar to the view used in most file browsers) is used when assigning courses to an evaluation. This same tree view is used to modify the course assignments. Portions of the tree are visible based on the hierarchy level of the administrator viewing it.
For Example:
The admin for the school of engineering would see the depts. in that school and the courses in each dept. that the view was expanded for. This admin would not see the portion of the tree containing other schools or the depts. and courses in those schools.

Users may create custom scales for use in their own questions

By default, users choose their scales from a set of scales defined by experts. They may also create their own scales for questions they create but these scales will only be visible/usable by the owner/creator (also visible to any admin through scales control).

Owned and shared items may be inserted into templates (C)

Users may insert any question items they own or that have been shared with them into a template they are working with. The items will be listed by owner/sharing method and by category (if categorized). These questions are accessible through a simple interface and searchable. Users may also choose a template and insert a group of items from an existing template.

Expert question items are available to users

Users may insert expert created questions into any template they have control over. These expert questions could be entered and maintained by an administrator. They would ideally come from experts in the field of evaluation. When users go to insert an expert question they would be presented with a step by step process which leads them to an appropriate question depending on what they are trying to accomplish. They would choose a Category first, and then an Objective and then they would choose the question from the list of expert questions related to those categories and objectives. They would be able to see a tip with each question that helps indicate what results they could expect when using that question.

Assign evaluations to worksite sections

Evaluations can be assigned to worksite sections as defined in the Sakai worksite.

Instructor Features/Requirements:

Instructors access all evaluations which affect them through a common interface (C)

All evaluations (including past evaluations) are visible through a common interface which is broken up by semester with the current semester appearing at the top.

View evaluation results for worksites they belong to (C)

View and print evaluation results (as allowed by higher level administrators).

Ability to create evaluations for owned courses

Instructors have the ability to create their own evaluations and assign them to their own courses only. This allows ad-hoc and midterm evaluations as desired. This is configurable by the system administrator.

Ability to keep results of owned evaluations private or share them with others

Instructors can choose to share the results of their personal evaluations with the administrators above them in the hierarchy or keep the data private. This is accomplished with a simple privacy flag which affects all associated results in the evaluation. Administrators can configure results of an evaluation to always be visible to them (override the flag) but instructors will be notified on screen when this is the case (they will not be able to use the flag).

Ability to add questions to evaluations from above in the hierarchy that only their students will see

Instructors can add questions to evaluations assigned to their courses by higher level administrators. Questions are added on a course by course basis and will only be visible to the students in the courses taught by this instructor.
Instructors may add up to a system configurable number of questions to an evaluation.
These questions may come from a template and instructors should be able to save those questions to a template.

Instructor evaluation email notification

Instructors should receive email notices when:

1. An evaluation is created above their level and assigned to at least 1 course they are teaching
2. An evaluation they have enabled/created starts
3. An evaluation ends and results are visible (i.e. the view results date and not the end date unless they are the same)

Instructors can choose whether to use evaluations created at higher levels in their own courses

Instructors have the option to allow their students to see and take the evaluations created at a hierarchy level above them. Evaluations created above them can be turned off by default, turned on by default, or required. If disabled by default, the instructor must choose to enable the evaluation or students will not receive email notifications or be allowed to take the evaluation. Instructors should get a receipt/confirmation screen that they can print out when they enable/disable an evaluation.

Student Features/Requirements:

Evaluations confidential (C)

Evaluations submitted by students are confidential. No instructor can link the evaluation results to any particular student. System admins can link results to a particular student in order to run statistical comparisons. Instructors cannot tell which students have taken an evaluation.

Common interface for accessing evaluations (C)

A common interface is also provided which shows evaluations for the current and previous semesters. Active evaluations will have a link which allows the student to take the evaluation or view the answers they submitted. Students may modify the evaluation up until the end date (controlled via a system option).

Students may skip questions as desired or be required to complete all non-text questions (owner controlled option) (C)

Students can be required to fill out all the questions in an evaluation. Students will be warned when leaving non-text questions unanswered regardless of this setting, however, they can choose to continue if the requirement is not set. Students may skip any and all questions in the evaluation as desired if the requirement is disabled.

Emails include direct link to evaluation (C)

Students are provided with a link which goes directly to the evaluation referred to in the email. Separate emails are sent out for each course being evaluated. After clicking the link in the email, students are asked to authenticate. Once they successfully authenticate, the evaluation is displayed. This provides students with the least number of steps when taking evaluations and reduces the possibility of confusion.

Controlled access to results (system option)

Students may view the results of evaluations they have taken as allowed by the admin. The owner of the evaluation may choose to allow students to view the results of the evaluation (by course only). This setting is off by default.

Students are automatically sent emails about the evaluation (controlled by the owner) (C)

Students are sent emails automatically if the creator of the evaluation decides to use this feature. Automatic email notification is on by default.

Emails are sent to students who have not completed the evaluation within the active period (between the start and end dates) at an interval controlled by the owner of the evaluation (C)

Students are continuously notified about evaluations during the active period (between the start and end dates) of the evaluation. This notification is sent out daily by default but can be configured for multiple days or disabled. Emails are only sent out to students who have not completed the evaluation. Emails appear to come from a centralized helpdesk. This helps to ensure students that the results of their evaluations will be anonymous and provides a point of contact for technical issues.

Reporting Mechanism Features/Requirements:

Report data only viewable after evaluation view date (C)

Report data cannot be viewed until after the end date of the evaluation has passed. The view date may be same as the end date by cannot be before it. This allows evaluation results to be hidden until after grades are turned in. This is part of the privacy protection that is enforced by the system.

Fine control over release of results (C)

The results can be shared with other instructors in the course, students, and/or just the owner. Dates for releasing the results to instructors and students can be specified separately. The system may be set to release results only if there are enough responses (the number is defined by an administrator).

Report data exportable (C)

Evaluation results are downloadable by admins and instructors in excel and CSV formats. Reports are also available in PDF format.

Reports can be generated across semesters if common templates are used

Evaluation results reports can be generated which span multiple semesters as long as the same template was used in creating the evaluation each time. This allows for analysis of long term trends and other statistical analysis.

Reports can include any number of schools, depts., or courses

Reports can be generated by administrators which include anywhere from a single course to multiple colleges (for a single evaluation). This allows complete flexibility in analyzing the results of evaluations and creating reports.

Reports are available based on the hierarchy structure

(e.g. results of dept. A are visible to a dept. A admin or college B admin where college B contains dept A): The hierarchy defines the access of administrators to report data. This allows the security and privacy of data to be ensured.
For Example:
The admin for the college of engineering can view the results for evaluation questions created at the college level in the college of engineering. They can also view official results for any evaluated course beneath them in the hierarchy. They cannot view the results for an evaluation given in any other college unless the results are explicitly shared with them. They cannot view the official results for any evaluated course in any other college.

Reports summary available by template and semester and includes response rates

A reports summary is visible to any administrator. The evaluations they can see are limited by the hierarchy. This report summary screen displays evaluations by semester on a single screen with response rates.

Dynamic reporting system which can accommodate many report types

Reports of any format created by the system administrator can be generated since all reports are generated dynamically from data in the database. This allows customized report formats to be created after the evaluation cycle which can include data from previous semesters.

Terminology/Definitions