BACKGROUND:
Intro to Psychology proceeded to give an exam to 300+ student simultantaneously using Respondus Lockdown Browser (integrated with Sakai), and Samigo in the Fall of 2012.
After initial failure, restructuring, adding access points, etc, the testing was still failing due to performance issues/lag times in page loading in Sakai. Network Engineers confirmed these were not bandwidth related.
Their insistence in pursuing testing this way led to efforts to validate that Sakai 2.9 + SAM-1695 would indeed resolve their issues for testing in Fall 2013.
We ran pre-patch and post-patch scenarios.
We rested on 4 separate 2013 dates: July 2, July 15, July 22 and July 23.
Results pre-patch were so bad we could not complete a test sequence and therefore began to test only the patched software to attempt to validate, not the % of the performance increase, but whether the Instructors would be able to successfully test at all.
The test itself involved:
1) Creating 300 fake user accounts
2) Scripting exact clicks and points and exit processes
3) Ramping up users (as agreed upon at runtime) in pre-designated time periods and intervals
4) We duplicated one of the real exams, a complex exam involving:
- 64 Question Pools out of which no more than 3 questions were drawn at a time
- 75 questions delivered in the exam using all of the 64 pools.
- Question answers themselves were randomized.
- Questions were delivered 1 per page (logging could not be collected per question if this weren’t so)
In the end, we could not recommend more than 80 students taking an exam simultaneously in the same course site with this degree of randomization.
The final application configuration (July 22nd and 23rd testing) was sized identically to our production environment, 4 app servers. Specs are included in the report.
Report Available upon request: lgekeler@nd.edu