Refactor autosubmit to enclose each update in a separate transaction and notify on failure
Description
depends on
is depended on by
Activity
Matthew Jones March 14, 2018 at 5:44 PM
Merged this to 11.x as depends on it.
Paul Lukasewych September 7, 2017 at 3:32 PM
It has been a while since I wrote this, but the idea was that the PROPAGATION_REQUIRES_NEW would make every method call create a new transaction. So for each iteration of the loop, the call to PersistenceService.getInstance().getAutoSubmitQueries().autoSubmitSingleAssessment() is done in its own transaction so that if it errors out it does not affect the rest of the submissions. Originally the entire batch of submissions was in a single transaction that rolled back if anything failed, so if you have an unrecoverable error with one of the submissions it just keeps rolling back each time, building up more and more queued submissions to process, yet never actually completing any of them.
When I was working on this job recently for our 11.3 upgrade, I realized that this implementation does not perfectly match the logic of the autosubmit job. It really should be collecting all the attempts for a single user/quiz pairing into one transaction, in case there is something wrong with an individual submission, as the logic relies on seeing all the attempts and deciding which one to autosubmit. However, that will likely require either introducing manual transaction management or collecting up the attempts and calling a new method to autosubmit them in batches. But when I was writing this originally, the problem we had encountered was related to the quiz itself (multiple gradebook entries for the same quiz), so it wouldn't have mattered in that case, none of the attempts for that quiz would have been processed.
Sam Ottenhoff September 7, 2017 at 2:16 PM
can you explain the changes made in this JIRA a bit? I'm having a tough time with this code while testing for Sakai 12. Specifically, can you explain your transaction changes especially PROPAGATION_REQUIRES_NEW?
Paul Lukasewych September 7, 2017 at 2:15 PM
We're still running with this modification on 11.3, and I'm not seeing these exceptions in our logs and our Autosubmit job seems to be running fine at the moment. We're on Oracle though.
Sam Ottenhoff September 7, 2017 at 1:58 PM
I receive deadlocks when testing on trunk/master when running auto-submit job
https://gist.github.com/ottenhoff/df187c705e1c5aa14cac7856d5037e2a
Details
Details
Priority
MajorFix versions
Components
Assignee
SAMIGO TEAMReporter
Paul Lukasewych
The entire autosubmit loop happens in the same transaction, so any failure during the loop iteration causes all previous work to be rolled back, effectively silently killing the job. To improve this:
Each loop iteration gets its own transaction
In the event of an iteration failing (an exception is caught), rather than just logging and continuing, note the failure and send an email to a configurable address when the job completes.
This patch introduces two new sakai.properties:
samigo.email.autoSubmit.errorNotification.enabled=true
samigo.email.autoSubmit.errorNotification.toAddress=notifyus@example.com