BACKGROUND:

Intro to Psychology proceeded to give an exam to 300+ student simultantaneously using Respondus Lockdown Browser (integrated with Sakai), and Samigo in the Fall of 2012.

After initial failure, restructuring, adding access points, etc, the testing was still failing due to performance issues/lag times in page loading in Sakai. Network Engineers confirmed these were not bandwidth related.

The Instructors insistence in pursuing giving exams this way led to efforts to validate that Sakai 2.9 + SAM-1695 would indeed resolve their issues for testing in Fall 2013.

We ran pre-patch and post-patch scenarios.The distribution we used wasANI's Sakai CLE 2.9.1 release (based on the community Sakai CLE 2.9.1) with selected patches from community 2.9.2 and 2.9.3. Included were SAM-2107, SAM-1695 and SAM-2102.

We rested on 5 separate 2013 dates: July 2, July 15, July 22, July 23 and Aug. 2nd

RESULTS:

Results pre-patch were so bad we could not complete a test sequence and therefore began to test only the updated Sakai CLE (with patches) to attempt to validate, not the % of the performance increase with patches, as we'd intended, but simply whether the patches would enable the Instructors to successfully test at all.

The testing itself involved:

1) Creating 300 fake user accounts

2) Scripting exact clicks and points and exit processes

3) Ramping up users (as agreed upon at runtime) in pre-designated time periods and intervals

4) We duplicated one of the real exams, a complex exam involving:

64 Question Pools out of which no more than 3 questions were drawn per Part, but usually only 1 or 2.
75 questions delivered in the exam using all of the 64 pools.
Question answers themselves were randomized.
Questions were delivered 1 per page (logging could not be collected per question if this weren’t so)

In the end, we could not recommend more than 80 students taking an exam simultaneously in the same course site in an exam with this degree of randomization.

The final application configuration (July 22^nd and Aug 2nd testing) was sized identically to our production environment, 4 app servers. Specs are included in the report.

Report and associated logs available at the link below. Questions can be directed to: Laura Gekeler, University of Notre Dame, lgekeler@nd.edu

https://notredame.box.com/s/5lxrxpzgs2ylrxe3iqfv

Browser not supported