Information
Documents the design related to email notifications and reminders.
Email Notification Design
Here are some details about the original process of designing the email notifications which will hopefully help clarify why the system works the way it currently does.
We did initial research to see how other systems did the notifications. The results of that research was that all systems we reviewed have an option to send out an email to indicate that a survey is available. Most had automatic reminders which went out either daily or at a controlled interval to all non-respondents. Some had the ability to also manually trigger reminder emails. A few only had the ability to send out reminders manually or not send reminders at all.
There was a requirement that the notification emails included a direct link to the evaluation so that the student would do the following:
- open the email
- click the link
- login to the system
- take the survey
By sending out one email for each course evaluation we met this requirement. This requirement was based on feedback from the Cold Fusion system at VT in which students mentioned the one click access to the survey as being convenient and contributing to their responding to the evaluation by not introducing extra steps. It was also based on a desire from the faculty to have the minimum number of steps to reach the evaluation.
In general, the notification emails are meant to meet the following goals:
- Tell users that there are evaluations to take
- Tell users where to go to take active evaluations
- Increase response rates to evaluations
Current functionality
Currently emails are generated under the following conditions:
1. Created: This email, using the Created template, is sent to all instructors after an evaluation is created when the Instructors may add this number of questions to evaluations from above in the hierarchy before the release date is greater than 0. This notifies instructors that they have until the evaluation start date to add questions to the template for their courses. This setting is found on the Administrate page under Instructor Settings.
2. Available: This email, using the Available template, is sent to everyone after an evaluation becomes Active. This occurs when the status is changed when the scheduler recognizes the current time is greater then the start date and time.
Recent modifications, allow this email to be skipped. By selecting Allow users to choose whether to send an initial email notification when creating an evaluation (otherwise an email notification will always be sent when the evaluation opens) on the Administrate -> Control Email Notification -> Email Options. This allow an evaluation to control whether or not to send available email.
By selecting Send notification email the day the evaluation goes live. then the Available email will be set, do not check and no available email is sent.
The current system sends out an available notification when an evaluation becomes available to take. This email is based on a template and goes out 5 minutes after an evaluation was created. This allows the user to remove that evaluation in case they made a mistake without having sent out a large number of emails with now invalid links. It also allows them to effectively notify the takers of the evaluation rapidly that there is an evaluation available. This email would normally include a direct link to the evaluation but the evaluation owner can control this via the template.
One email is sent to each user in each group associated with the evaluation. This means that a student user would probably get one email for each course in an evaluation that is available.
After the available email has been processed (note that the Sakai email service does not indicate if the email was sent or not, we can only know that it was processed without failure) succcessfully, the system queues up reminder emails if they are enabled for an evaluation. If they are disabled then nothing happens until the evaluation ends.
3. Reminders:
If reminders are turned on then the scheduler will send a reminder in the same fashion as the available email using either the default template which includes a direct link to the evaluation or a custom template if one is set up for the evaluation. One email is sent to each user in each group associated with the evaluation who has not yet responded to the evaluation. This means that if a student has responded for 3 out of 5 group evaluations then they will receive 2 email reminders.
This process repeats as long as the evaluation is active. Once the due date is reached then the reminder email processing stops for an evaluation.
4. Results:
After an evaluation is over and the view date is reached an email is sent to the owner indicating that results are now available. This email uses a system template and includes a direct link to the results of the evaluation.
Possible Reminder Email designs
From conversations over the last few days I am seeing the following possible ways of handling email reminders. Please feel free to add notes to each one and to add your own methods. Please use the comments or the email list to discuss the pros and cons or each.
As written here, evaluation refers to a single evaluation which must be for one or many courses. This is different from course evaluation which is the evaluation as accessed for a single course/group. An evaluation by itself cannot really be taken as there is no context for it. A course/group evaluation can be taken because now we know who is be being evaluated and who may do the evaluating.
num of emails is the number of emails that the system will send per reminder period with each method.
- One email per course evaluation method - configurable interval per evaluation with one reminder sent per course evaluation per user and the email template is configurable per evaluation, includes direct link to the evaluation, after taking the evaluation the user is directed to the main page which lists all current evaluations
- num of emails = num of assigned courses * num of evalaution * num of takers
- (this is the method we are currently using)
- Landing page method - one email per user taking any evaluation (and per global reminder period) which directs the user to a landing page (main summary page) which includes a list of all their current evaluations, email is generic and uses a global template with no direct links
- num of emails = num of takers
- (this has the advantage of being pretty simple to code and execute with the only tricky bit being how to handle the case where an individual evaluation has a shorter reminder interval than the system interval)
- Fully aggregated email method - Aggregate all reminders into a single email which includes the list of all current evaluations for the user, this would be one email per user per global reminder period, the template would be a global template and individual evaluation templates would not be supported
- num of emails = num of takers
- (this is the most complicated to develop of the 4 options)
- One email per evaluation - one email per evaluation per user which lists a set of all course/groups for that evaluation and includes a direct link for each course evaluation, email template is configurable per evaluation and reminder interval is configured for each evaluation
- num of emails = num of evaluation * num of takers
- (this is the easiest one to develop of the non-current methods)
- Michigan method - one email per [email template of] student having one or more evaluations for which no response has been submitted. E.g., if a student has seven active evaluations that use two email templates, send two emails to that student. Email includes 1) a direct link to the summary page of the student's My Workspace site tool placement, and 2) the earliest due date among the active evaluations for which the student has not submitted a response. Earliest due date appears in the subject and body of available and reminder notifications. There are available and reminder consolidated-email [changing "consolidated" to "single email"] templates. Global reminder period is expressed in number of days.
- num of emails = num of students with at least one evaluation response that hasn't been submitted * number of email templates used for those evaluations
- sets of evaluations using the same email template will be handled as part of the data import, creating email templates required and then setting email template ids when evaluations are saved.
- exception handling in which an email template has to change will also be handled as part of the data import
- this is relatively simple as it was implemented, a Quartz cron job firing once daily under the Job Scheduler
- the ScheduledInvocationManager continues to manage evaluation state changes through the SCHEDULER_DELAYED_INVOCATION table, but email is handled by this cron job
- Overlapping sets of evaluations (e.g., Pharmacy and seven-week courses might have overlapping evaluation dates) are not tagged or differentiated in terms of email templates
- Email settings were added to sakai.properties and set as part of the build process based on target server, for safer email testing (options of "no email", "log email", "send email"), to limit by throttling the impact of email processing in production, and as a check to log email recipients. These settings would move to the administrate page, and appropriate values would be pre-loaded. The email notification settings were:
- waitInterval@org.sakaiproject.evaluation.logic.EvalNotificationSettings
- batchSize@org.sakaiproject.evaluation.logic.EvalNotificationSettings
- logRecipents@org.sakaiproject.evaluation.logic.EvalNotificationSettings
- deliveryOption@org.sakaiproject.evaluation.logic.EvalNotificationSettings
- Performance required to fit in the maintenance window this fall after increasing courses 6-10x relative to pilots is a current subject of attention.