Email Notifications

Owned by Former user (Deleted)
Last updated: Jan 18, 2011 by Former user (Deleted)Version comment
7 min read

Information

Documents the design related to email notifications and reminders.

Email Notification Design

Here are some details about the original process of designing the email notifications which will hopefully help clarify why the system works the way it currently does.

We did initial research to see how other systems did the notifications. The results of that research was that all systems we reviewed have an option to send out an email to indicate that a survey is available. Most had automatic reminders which went out either daily or at a controlled interval to all non-respondents. Some had the ability to also manually trigger reminder emails. A few only had the ability to send out reminders manually or not send reminders at all.

There was a requirement that the notification emails included a direct link to the evaluation so that the student would do the following:

  1. open the email
  2. click the link
  3. login to the system
  4. take the survey

By sending out one email for each course evaluation we met this requirement. This requirement was based on feedback from the Cold Fusion system at VT in which students mentioned the one click access to the survey as being convenient and contributing to their responding to the evaluation by not introducing extra steps. It was also based on a desire from the faculty to have the minimum number of steps to reach the evaluation.

In general, the notification emails are meant to meet the following goals:

  1. Tell users that there are evaluations to take
  2. Tell users where to go to take active evaluations
  3. Increase response rates to evaluations

Current functionality

Currently emails are generated under the following conditions:

1. Created: This email, using the Created template,  is sent to all instructors after an evaluation is created when the Instructors may add this number of questions to evaluations from above in the hierarchy before the release date is greater than 0. This notifies instructors that they have until the evaluation start date to add questions to the template for their courses. This setting is found on the Administrate page under Instructor Settings.

2. Available:  This email, using the Available template, is sent to everyone after an evaluation becomes Active. This notifies the students and instructors that an evaluation is ready to be completed.  This occurs when the status is changed when the scheduler recognizes the current time is greater then the start date and time.

  Recent modifications allow this email to be skipped. By selecting Allow users to choose whether to send an initial email notification when creating an evaluation (otherwise an email notification will always be sent when the evaluation opens) on the Administrate -> Control Email Notification -> Email Options allows an evaluation to control whether or not to send available email.

 When creating an evaluation, selecting Send notification email the day the evaluation goes live. then the Available email will be set.  Do not check and no available email is sent.

One email is sent to each user in each group associated with the evaluation. This means that a student user would get one email for each course in the active evaluation.

After the available email has been processed (note that the Sakai email service does not indicate if the email was sent or not, we can only know that it was processed without failure) successfully, the system queues up reminder emails if they are enabled for an evaluation. If they are disabled then nothing happens until the evaluation ends.

3. Reminders: This email, using the Reminder template, is sent to everyone who has an outstanding group to be evaluated. From the Administrate -> Control Email Notification -> Email Options, the default reminder frequency to use for new evaluations can be set.

Then, when creating an evaluation, select the Email reminder to non-respondent students. for this evaluation.  Reminder can be sent never, daily, every 3,4,5,6 days, or every week, or the last 24 hours before closing.  The frequency can be changed anytime during an active evaluation.  If the next reminder is past the due date, it will not be scheduled.

When reminders run, one email is sent to each user in each group associated with the evaluation who has not yet responded to the evaluation. This means that if a student has responded for 3 out of 5 group evaluations then they will receive 2 email reminders.
This process repeats base on the reminder frequency as long as the evaluation is active. Once the due date is reached then the reminder email processing stops for an evaluation.

4. Single Email per Student: This email using the ConsolidatedAvailable and ConsolidatedReminder templates, sends a single email to a student listing all groups to complete, regardless of the number of active evaluations.  On the Administrate -> Control Email Notification ->Single Email Per Student, select   Enable single email per student, default is to send one email for each group in each evaluation. The single email per student implementation (UM) uses a daily Quartz job to process email. This job must be running when evaluations begin.

There is no override capability for this option when setting up an evaluation.

5. Results: This email using the Results template, sends email to either the instructors or students when and the view date is reached. An email is sent based upon the settings on the Administrate page.  By setting the Instructors allowed to view results or Students allowed to view results to yes the appropriate population will receive the email.

Possible Reminder Email designs

From conversations over the last few days I am seeing the following possible ways of handling email reminders. Please feel free to add notes to each one and to add your own methods. Please use the comments or the email list to discuss the pros and cons or each.

As written here, evaluation refers to a single evaluation which must be for one or many courses. This is different from course evaluation which is the evaluation as accessed for a single course/group. An evaluation by itself cannot really be taken as there is no context for it. A course/group evaluation can be taken because now we know who is be being evaluated and who may do the evaluating.
num of emails is the number of emails that the system will send per reminder period with each method.

  1. One email per course evaluation method - configurable interval per evaluation with one reminder sent per course evaluation per user and the email template is configurable per evaluation, includes direct link to the evaluation, after taking the evaluation the user is directed to the main page which lists all current evaluations
    • num of emails = num of assigned courses * num of evalaution * num of takers
    • (this is the method we are currently using)
  2. Landing page method - one email per user taking any evaluation (and per global reminder period) which directs the user to a landing page (main summary page) which includes a list of all their current evaluations, email is generic and uses a global template with no direct links
    • num of emails = num of takers
    • (this has the advantage of being pretty simple to code and execute with the only tricky bit being how to handle the case where an individual evaluation has a shorter reminder interval than the system interval)
  3. Fully aggregated email method - Aggregate all reminders into a single email which includes the list of all current evaluations for the user, this would be one email per user per global reminder period, the template would be a global template and individual evaluation templates would not be supported
    • num of emails = num of takers
    • (this is the most complicated to develop of the 4 options)
  4. One email per evaluation - one email per evaluation per user which lists a set of all course/groups for that evaluation and includes a direct link for each course evaluation, email template is configurable per evaluation and reminder interval is configured for each evaluation
    • num of emails = num of evaluation * num of takers
    • (this is the easiest one to develop of the non-current methods)
  5. Michigan method - one email per [email template of] student having one or more evaluations for which no response has been submitted. E.g., if a student has seven active evaluations that use two email templates, send two emails to that student. Email includes 1) a direct link to the summary page of the student's My Workspace site tool placement, and 2) the earliest due date among the active evaluations for which the student has not submitted a response. Earliest due date appears in the subject and body of available and reminder notifications. There are available and reminder consolidated-email [changing "consolidated" to "single email"] templates. Global reminder period is expressed in number of days.
    • num of emails = num of students with at least one evaluation response that hasn't been submitted * number of email templates used for those evaluations
    • sets of evaluations using the same email template will be handled as part of the data import, creating email templates required and then setting email template ids when evaluations are saved.
    • exception handling in which an email template has to change will also be handled as part of the data import
    • this is relatively simple as it was implemented, a Quartz cron job firing once daily under the Job Scheduler
    • the ScheduledInvocationManager continues to manage evaluation state changes through the SCHEDULER_DELAYED_INVOCATION table, but email is handled by this cron job
    • Overlapping sets of evaluations (e.g., Pharmacy and seven-week courses might have overlapping evaluation dates) are not tagged or differentiated in terms of email templates
    • Email settings were added to sakai.properties and set as part of the build process based on target server, for safer email testing (options of "no email", "log email", "send email"), to limit by throttling the impact of email processing in production, and as a check to log email recipients. These settings would move to the administrate page, and appropriate values would be pre-loaded. The email notification settings were:
      • waitInterval@org.sakaiproject.evaluation.logic.EvalNotificationSettings
      • batchSize@org.sakaiproject.evaluation.logic.EvalNotificationSettings
      • logRecipents@org.sakaiproject.evaluation.logic.EvalNotificationSettings
      • deliveryOption@org.sakaiproject.evaluation.logic.EvalNotificationSettings
    • Performance required to fit in the maintenance window this fall after increasing courses 6-10x relative to pilots is a current subject of attention.

Proposed Changes