Assignments: Not all released student grades are recorded in Gradebook
Description
is blocked by
is related to
Activity
Sean Horner March 16, 2023 at 3:22 PM
- Thank you again for taking the time on Tuesday to walk me through some nuances for how Hibernate handles objects it persists. Per the approach we discussed, I proceeded to analyze the “Save and Release to Students” code, looking for subsequent writes to the AssignmentSubmission object after a session.merge. Tracing the code for both 21.x and master, I couldn’t find evidence of such writes.
That said, I will soon force push a newer version of my code branch which is more inline with what we discussed. Because I couldn’t find subsequent writes that might explain the bug, I am skeptical that the newer branch as-is will fix the issue that our institution (and at least one other) experienced. Regardless, I’m including in the branch two new debug log statements that will log the AssignmentSubmission object’s hashcode and the state of its grade property.
When you’ve had the opportunity of running the detection script for this bug on other instances of Sakai, I’d be interested to know (in broad terms) what you find. Thanks.
Sean Horner March 2, 2023 at 7:57 PMEdited
Hello ,
Thank you for your suggestions. In our Sakai 21 instance, we iteratively added extensive logging to troubleshoot and resolve both this issue and SAK-48553. I’ll limit my comments here to SAK-48551 and use an example instance of this bug’s occurrence from Feb. 10, back when this bug was still active for us.
We logged an instance where an instructor scored a student submission (id: 2c8061e5-2743-49d9-a6ee-6bf397b346df) and clicked “Save & Release”.
10-Feb-2023 19:50:30.985 INFO [ajp-nio-10.1.8.97-8009-exec-9] org.sakaiproject.assignment.tool.AssignmentAction.grade_submission_option Legacy grading interface grading submission /assignment/s/PSYC330_Spring_2023/9be9c07b-afce-4054-892a-d6a879835d83/2c8061e5-2743-49d9-a6ee-6bf397b346df with grade [1000] and grading option: return
We also captured the the new grade (1000) for the submission and its date modified timestamp right before AssignmentServiceImpl.updateSubmission calls AssignmentRepositoryImpl.updateSubmission. The latter revises the submission’s date modified and delegates to Hibernate saving the submission.
10-Feb-2023 19:50:30.988 INFO [ajp-nio-10.1.8.97-8009-exec-9] org.sakaiproject.assignment.impl.AssignmentServiceImpl.updateSubmission Before updating repo - submission.getGrade: (1000); submission.getDateModified: (2023-02-09T22:59:54Z)
We also added a log in AssignmentServiceImpl.updateSubmission immediately after the call to AssignmentRepositoryImpl.updateSubmission, which confirms that the submission’s date modified was updated to “now”.
10-Feb-2023 19:50:30.991 INFO [ajp-nio-10.1.8.97-8009-exec-9] org.sakaiproject.assignment.impl.AssignmentServiceImpl.updateSubmission After updating repo - submission.getGrade: (1000); submission.getDateModified: (2023-02-11T03:50:30.989Z)
Further downstream, AssignmentToolUtils.integrateGradebook executes. The following line shows that the submission object is fetched from the AssignmentService (which wraps the AssignmentRepository, which in turn wraps Hibernate).
A few lines below this call, we log information about the submission object (id: 2c8061e5-2743-49d9-a6ee-6bf397b346df) retrieved from Hibernate.
10-Feb-2023 19:50:31.041 INFO [ajp-nio-10.1.8.97-8009-exec-9] org.sakaiproject.assignment.tool.AssignmentToolUtils.integrateGradebook gradeStringToUse: (); submission.getGrade: (null); submission.getDateModified: (2023-02-09T22:59:54Z); gradeString: ()
10-Feb-2023 19:50:31.042 INFO [ajp-nio-10.1.8.97-8009-exec-9] org.sakaiproject.assignment.tool.AssignmentToolUtils.integrateGradebook Update GB score to () for submission (/assignment/s/PSYC330_Spring_2023/9be9c07b-afce-4054-892a-d6a879835d83/2c8061e5-2743-49d9-a6ee-6bf397b346df) and submitter (e9163df7-02d3-4977-9f56-605ed4a6bac6)
The submission’s grade is null and its date modified matches the timestamp we logged prior to the execution of AssignmentRepositoryImpl.updateSubmission. The submission object retrieved is therefore stale.
From this I concluded that the retrieval of the submission object needs a Hibernate refresh from the database. Upstream, the Hibernate session flush that we introduced when Hibernate saves revisions to the submission object ensures that the object is stored in the database prior to its subsequent retrieval in this “Save & Release” runtime logic.
I’m open to differing conclusions or approaches to fixing this (though I think I’ll want to retain our current local Sakai 21 fix that we have in production since Feb. 17 to avoid further disturbing instructors with this bug).
Regarding the alternate code path proposed in my current SAK-48551 branch, I’m definitely open to refactoring that. For AssignmentRepository and AssignmentService, I intentionally added new methods for retrieving the submission from the database so as to only induce a performance cost (of fetching the submission object from the database) only where the evidence of this bug called for it.
I do have a question for clarification regarding your prompt to refactor the code. I can refactor the code to retain the new interface I proposed while removing redundancy (e.g., between AssignmentServiceImpl’s getSubmission and getSumissionFromDatabase, etc.) Is that what you are asking for, or were you preferring to see the interface for AssignmentRepository and AssignmentService not include new methods (e.g., getSubmissionFromDatabase) such that extant methods would include a session refresh call? Would the latter be desirable from a performance perspective?
Earle Nietzel March 2, 2023 at 5:57 PM
Hi Sean,
Thanks for the great detective work on this, I suspect that this is related to an old reference making it’s way back into a hibernate session.
Here is what I would suggest you do once the user clicks save to update the submission scores. In a debugger notice the Submissions object id that contains the correct score and follow it back into the service AssignmentService and AssignmentRepository to where it will get saved always checkin the object id to make sure it is the same and verifying it’s GRADE is correct. If the AssignmentSubmission Object ID changes you should verify in the DB that it looks correct with the new Object ID, in doing this it will lead you to the place that is causing the issue.
If you would like me to look into it for you let me know?
Automation for Jira March 2, 2023 at 2:10 AM
A pull request has been created, "SAK-48551 - Assignments: Not all released grades recorded in GB", you can see it at https://github.com/sakaiproject/sakai/pull/11307
Details
Details
Priority
CriticalAffects versions
23 Status
22 Status
21 Status
Components
Assignee
Sean HornerReporter
Sean Horner
Our institution detected over 50 cases of a bug across various course sites in Fall 2022 whereby point scores released by an instructor for assignments with gradebook associations were not recorded for one or more students in Gradebook. Many such cases, if not caught, would adversely impact the calculated course grades for affected students.
Most if not all these cases occur when an instructor grades and releases student work one-by-one-- using the “Save & Release to Student” button-- as opposed to releasing all student grades at once with the Release Grade link. These cases occur regardless of which grading interface (legacy vs. Grader) is used and regardless of whether the associated gradebook item originates from Assignments or Gradebook. (None of the cases we identified were instances of group assignments or peer evaluation assignments.)
In the runtime logic that is triggered when an instructor clicks the ‘Save & Release’ button to release a score to a student and to Gradebook, an inconsistency in the submission object’s state is occasionally induced due to the state of the object in the Hibernate cache. At the point of saving a new score value for the submission to the database, the submission contains the intended score as reported by its “getGrade” method. However, further downstream-- i.e., in the AssignmentToolUtils method “integrateGradebook”--. the submission object is obtained through a Hibernate intermediary which occasionally returns an older version of the object whose “getGrade” method returns null.
Because this bug is not happening for every “Save & Release” user event, I surmised that some kind of race condition and/or a caching sync problem was occurring. Introducing Hibernate “flush” and “refresh” calls within key points in the logic above prevents this bug’s recurrence.