University of Maryland System Requirements

Implementation Plan

Summer I Pilot:  Summer School courses - unsure if we will do all of them or a subset of them.  (Start survey July 4, end July 11)
**Expectations for this pilot is to have Email notification, LDAP connectivity, secure server, ability to take survey

Summer II Pilot:  All summer school courses (Start survey August 15, end August 22)
**Expectations for this pilot is to have as much of the final version (per our system requirements below) in place as possible.  This means we want hierarchy structure and reporting in place.

Fall 2007 Pilot:  Campus-wide pilot with university questions only (Start survey 11/26/07, end 12/12/07)
**Experienced major performance problems.  Evaluation opened on 11/30/07 instead of 11/26/07.

Spring 2008:  Campus-wide pilot with university questions and college level questions for 3 colleges.  Also implementing multiple instructor functionality
**Key project dates:
    3/28/08 Freeze all code - Sakai and Evaluation System code
    3/31/08 - 4/11/08 Load testing in QA environment
    4/14/08 - 4/18/08 Load testing in production environment
    4/29/08  Survey begins
    5/14/08  Survey ends

System Requirements (required)

System level

1. System must support four (4) levels of questions.  This list also defines the hierarchy in terms of level of access with item 1. being the top level when referring to access in the requirements defined below.
   1. University-wide
   2. College-wide
   3. Department/Program-wide (course prefix?)
   4. Instructor
2. System must cap total number of questions at a set level. 
3. Must integrate with LDAP for authentication
4. Must integrate with UMEG for course enrollments and grade distributions
5. Ability to define the cut-off enrollment for courses to be included into evaluation process.
6. Student responses will be confidential, but not anonymous (information that can be traced back to an individual will not be reported).
7. Ability to set start and end date and times for evaluations to be available.
   1. Default start date is 2 weeks prior to end date recorded in system
   2. Ability to override start and end dates for non-standard course lengths
8. Ability to tag questions by type (student public questions, APT questions, etc.
9. Ability to store system-wide question sets that are available to all levels of access for creating questions (i.e. CORE questions)

Front-End Question Input

1. Ability to assign different roles to individual users to include:
   1. System Administrator - can manage (read/write/edit) all question sets across the institution at all levels of the hierarchy.  This includes the ability to review and edit college-wide, department/program-wide, or instructor level questions, and the ability to set access roles for levels below in the hierarchy.
   2. College Administrator - can manage (read/write/edit) all question sets at the college-wide level and below in the hierarchy, and has the ability to set access roles for levels below in the hierarchy.
   3. Department/Program Administrator - can manage (read/write/edit) all question sets at the department/program level and below in the hierarchy, and has the ability to set access roles for levels below in the hierarchy.
   4. Instructor - can manage (read/write/edit) question sets for their individual courses.
      1. Default access to instructor of record, ability for individuals above in hierarchy to override
      2. Instructors can select existing items from a bank of closed-ended questions or write open-ended items of their own
2. Ability to define question sets within each level below University-wide.
3. Ability to define which courses or programs have access for each question set
4. Ability to create multiple templates for college or department level questions for groups of courses, such as lectures, labs, studio, etc.
5. Ability for templates and question selections to "roll over" to the next semester
   1. At University, College, and Department/Program level, previous semester's selections automatically roll over if no response
   2. At the instructor level, individuals must indicate they would like to keep the questions or add/change questions, or else no questions will be included at the instructor level
6. Ability to automatically send email reminders to administrator roles about deadlines for finalizing questions.
7. Ability to save and reuse question sets across semesters.

Delivery of evaluations

1. Ability to track and determine if a student has filled in all of their assigned course evaluations.
2. Students must fully complete an evaluation in order to submit
   1. All closed-ended items will not have a default answer selected upon opening a new evaluation
   2. All closed-ended items must be answered, with "N/A" as a response option
   3. Open-ended items may be left blank
   4. If a student does not fully complete an evaluation, a message must pop up indicating the reason why the evaluation could not be submitted
3. Students may wave their ability to complete course evaluation(s)
   1. Students will receive a message indicating that waving the evaluation does not fulfill the requirements necessary to view public results the following semester, and they will have the option to return to their evaluation(s)
4. Ability to display to the student upon login the list of evaluations they are responsible for and their status (completed or not).
5. Ability to email all individual students when evaluations begin (not an individual email per course).
6. Ability to send reminder email messages to students who have not filled in all of their evaluations.
7. Ability to send reminder email messages to students that indicate which evaluations have not been completed yet.
8. Ability to set up automatic email reminders based on a period of time.
9. Ability to manually send out an email reminder immediately.
10. Ability to check response rates during the evaluation period.

Back-End Access and Reporting

1. Ability to assign different roles to individual users to include:
   1. System Administrator - can access all raw data across the institution and all levels of the hierarchy and has the ability to set access roles for levels below in the hierarchy
   2. College Administrator - can access reports (not raw data) for APT-designated university-wide questions and/or at college level and below in the hierarchy
   3. Department/Program Administrator - can access reports (not raw data) for APT-designated university-wide questions and/or at college level and below in the hierarchy. Can access aggregate results for a specific instructor across courses and/or semesters
   4. Instructor - can access reports for their individual courses.
2. Ability to track and determine the return rates for all courses.
3. Ability to display results from a subset of all questions at all hierarchy levels.
4. Ability to display results from courses where there is a 70% or above return rate.
   1. Students waving the evaluation will not count towards the response rate
   2. Response rate calculated student-by-student, not item-by-item
5. Ability to restrict access to results if a student has not completed all their evaluations or waved some or all of their evaluations.
6. Ability to determine which semesters are included in calculation of eligibility of students to see public results.  Only fall and spring semester will count towards process of determining if a student is eligible to see results.
7. Ability to set different access rights for non-public question results.
   1. APT committees cannot view results to instructor-level items
8. Ability to set different access rights for public question results.
   1. APT committees cannot view results to public questions at any level of the hierarchy
9. Public results should be available to be displayed in an appropriate context; that is, with class average, in relation to results within the college in that same level (100-level courses, 200-level courses).
10. Ability to export data to a delimited file that can be used in alternative analysis programs.
11. Ability to export data at each level of the hierarchy to approved representatives without student ID information.
12. Ability to set access for an approved representative to download data with (IRPA uses only) or without student ID information.
13. Ability to mark questions at college or department level that are public to students.

System Requirements (desired)

1. Ability to include a link to the evaluation inside the email messages sent to students.
2. Provide students the ability to cut the public results by question 14 (reported effort) and 16 (reason for taking course)
3. Ability for instructors to pick questions from an existing database of questions.
4. Ability to evaluate courses outside the regular semester schedule
5. Ability to set the end date for program-wide courses that are not full semester