Sakai 3 Load Testing

Owned by Former user (Deleted)
Last updated: Apr 12, 2012 by Former user (Deleted)Version comment
8 min read

Overview - Goal of Load Testing

Load Testing of Sakai 3 on a regular schedule will permit:

  • identification of memory leaks
  • identification of bottle necks
  • recommendations for production deployment sizing

This page attempts to summarize the dead-simplest implementation of a repeatable set of load tests which can easily be run at any institution, and the results collected for valid comparison.

The QA Workgroup has started an effort with many of the same goals.

Tools & Scripts

JMeter

Jmeter can be run from any commodity laptop or desktop.

Download JMeter 2.4 at http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi

Expand it to some reasonable directory, and start it with ./bin/jmeter on unix/mac or ./bin/jmeter.bat on windows

If you have never run JMeter before, whip through JMeter proxy Step-by-Step to confirm your system works against a public internet site before you get caught up in the specifics of Sakai 3 load testing.

Managed Project JMeter Test Plans

The Sakai 3 JMeter Test Plans are part of the standard github repository: http://github.com/sakaiproject/nakamura/tree/master/testscripts/jmeter/

The files users.pl and users.csv at the root of this sub-tree populate test users and provide test credentials, respectively. A users.csv or a netids.csv specific to individual test plans are also in some subdirectories.

WITH_DASH: Configure and Do a Test Run

These instructions are accurate as of October 14, but will change to require less configuration

Populate Test Accounts
Before running any load test, clean out the sling directory and populate the system with test users. This assumes your system is only being used for load testing. Don't go wiping out real users.

From the nakamura directory
rm -rf ./sling
./testscripts/jmeter/users.pl
The users.pl script creates 2,000 test users and will generate output of the form

100,17
200,15
...
1800,32
1900,36

Configure the JMeter test plan WITH_DASH.jmx
In JMeter, open nakamura/testscripts/jmeter/WITH_DASH.jmx.
In the HTTP Request Defaults screen, change

  • Server Name or IP to your load test environment, for example localhost
  • port Number to your environment's port, for example 8080, which is the default Sakai port.
    In the {{Users} screen, change
  • Filename from /home/aberg/Desktop/Stress_S3/users.csv to the local copy of the users.csv in your testscripts/jmeter directory.

Run it
Set the Test Plan to run a single loop, to confirm your configuration is correct:
Click the on "FAIL LOGIN / LOGIN / DASHBOARD / LOGOUT" Test Plan, replace 100,000 in LOOPS with 1.
Click on the View Results Tree Listener to monitor the test as it runs.
Select Run -> Start

Test Plans in Detail

KERN-1306 - Generalized Load Simulation

Preconfiguration - Short Form

Call one_script_to_rule_them.sh in testscripts/jmeter/kern-1306. C'est tout.

Preconfiguration - Content File Acquisition

Assuming you did not call one_script_to_rule_them.sh or you just want to know what that script does:

  • Call prepare-load-files.sh in testscripts/jmeter/kern-1306. No parameters are needed.

This will pull down approximately 500 gigs of content files from an NYU server, as well as the csv with associated metadata for the content files. This is pulled down separately in order to keep test data out of the git repository – there's no reason to track a half a gig of sample data.

Preconfiguration - Source files for Users, Messages, Tags and Content Files

  • Call prepare-data.sh with number of user to create and location of the test content in testscripts/jmeter/kern-1306.
    e.g. prepare-data.sh 300 TestContent

You will get output akin to

creating 300 netIDs in ../netids01.csv
creating 30000 message recipients in recipients.csv
creating 900 user profile tags in user-tags.csv
creating list of content from TestContent/ in content.txt
creating ID,FILEPATH,MIMETYPE for file uploads in file-uploads.csv

This overwrites the netids01.csv and any file-uploads.csv you downloaded, recreating with the given number of users and associating those users with the available content files. The prepare-data.sh file creates 100 messages per user and 3 tags per user.

The generate-recipient-list.rb ruby script may create recipient = self. Is that a valid data pair?

Preconfiguration - Separate scripts for groups and group members

These will be folded into the one_script_to_rule.sh and prepare-data.sh tomorrow.

  • Call ruby generate-groups.rb with number of groups to create and output the contents to groups.csv
    e.g. ruby generate-groups.rb 75 > groups.csv
    This creates group owner, group name and generates a list of 20 members for each group

Preconfiguration - Load It Up

  • Call load-data.sh with hostname and port in testscripts/jmeter/kern-1306.
    e.g. load-data.sh localhost 8080
    This assumes your jmeter exists in ./jakarta-jmeter-2.4. Edit the script to provide a different directory if necessary.

Preconfiguration - Separate load for Groups and Group Members

These will be folded into the one_script_to_rule.sh and load-data.sh tomorrow.

Preconfiguration - very optional - Clean it up and Reload

  1. Shut the system down - uh, how? No, really, how do I shut it down cleanly? kill 9 the pid?
  2. Remove the sling directory - rm -rf nakamura/sling
  3. Start the system up - tools/run_production.sh &

Preconfiguration - Load Just the Users

Use the standard kern-1306/netids01.csv file if you want to compare test results with others.
To load users:

cd testscripts/jmeter
./usersfromcsv.pl ./kern-1306/netids01.csv

The usersfromcsv.pl load script assumes a file with fields in format username,password with no quotes. The load outputs duration in seconds for creating every 100 users. The default netids01.csv in github provides 6000 user accounts, all with password set to test. Calling prepare-data.sh overwrites the default version with the number of users you provide

Preconfiguration - Load Just the Messages

The load-messages.jmx test plan reads source usernames from netids01.csv and destination usernames from recipients.csv then generates messages between them based on the /usr/share/dict/words unix word file. The prepare-data.sh script provides 100 messages per user. Message content will vary based on /usr/share/dict/words contents.
To load messages:

cd /testscripts/jmeter/kern-1306
jmeter --nongui --testfile load-messages.jmx

Preconfiguration - Load Just the Content Files

Some magical person alanmarks out there generated a content pool, producing a directory with 1/2 GB of content files. The test content pool is stored at NYU. We anticipate this data collection will grow to 2GB, and no one wants that in github. Content is in TestContent.zip and the corresponding csv file-uploads.csv assumes the default netids01.csv. If you call prepara-data.sh both the netids01.csv and the file-uploads.csv will be generated to keep the usernames in sync.

Call the script which grabs and unzips TestContent and file-uploads.csv:

cd testscripts/jmeter/kern-1306
./prepare-load-files.sh

Now that you have the content files locally, load them in

jmeter --nongui --testfile load-files.jmx

Note that the variable "PASSWORD" is set in the test plan itself. ID, FILE, and MIMETYPE are taken from the file upload list file_uploads.csv. To call the JMeter test plan you must be in the kern-1306 directory in order to get access to the associated content files.

The test plan load-files.jmx reads every line of file_uploads.csv, connects as the user, uploads a file and logs out. The test plan runs (forever) until the end of file is reached. A single thread is used, but See http://jakarta.apache.org/jmeter/usermanual/component_reference.html#config_elements for details on the settings for CSV data file loads. IMPROVEMENTS to load time can be made by increasing the number of threads or disabling the Gaussian timers on each step.

Preconfiguration - Load Just the Tags

The load-tags.jmx test plan reads username-tag pairs from user-tags.csv and generates a tag, then applies it to the same user. The default user-tags.csv provides 10,000 pairs, resulting in 18,000 tags. If you call the prepare-data.sh script the user-tags.csv file is regenerated with 3 tags per user in order to keep the usernames in sync. Tags are randomly selected from /usr/share/dict/words.
To load tags:

cd /testscripts/jmeter/kern-1306
jmeter --nongui --testfile load-tags.jmx

Want to make your own users, messages and content files?

Think a bit before you decide to go off with your own data set. Multiple tests on the same data set will give us maximal comparability across hardware setups. Variety of data sets will give us maximal coverage of the code itself.

Make users Users are hashed by username, so users must be distributed over a likely spectrum. To generate user names of the form XXNNN, that is two alpha characters followed by four numeric characters, use the script testscripts/jmeter/netidusers.pl. This takes a count on the command line. Just in case it makes some duplicates, generate extra names and take the head. Note that this produces slightly fewer-than-random accounts at the tail of the alphabet. Take the tail to produce slightly fewer-than-random account at the start of the alphabet.

netidusers.pl 6100 | sort | uniq | head -n 6000 > someusers.csv

Make list of users for message creation
To populate messages, the messages must pass between two users. The Ruby script testcripts/jmeter/kern-1306/generate-recipient-list.rb takes a count N, and picks N usernames from the file netids01.csv. It appends the string ',test' to the username because jmeter turns out to cough on a csv with no 'c' in it. This script should be run from within kern-1306 in order to access the kern-1306 associated netids01.csv file.

cd testscripts/jmeter/kern-1306
ruby generate-recipient-list.rb 10000 >> some-message-recipients.csv

Messages are created by the same load-messages.jmx file as in the non-customized version. The load-messages.jmx script relies on /usr/share/dict/words to randomly generate the content of the messages. _load-messages.jmx terminates when all lines of message-pairs.csv have been read *? still true? *

Make control file for content file upload
Two step process for creating a file-uploads.csv file appropriate for use with load-files.jmx

find $CONTENTDIR -type f > content.txt
ruby generate-file-upload-list.rb > file-uploads.csv

File-uploads.csv will be generated with fields in format username,filename,filetype.

  • Make user-tag pairs for tag creation* 
    generate-user-tags.sh

Standard User

Basedon the bug bash instructions, the test plan performs the following actions:

raydavis
:
Anything is fair game, but try to do the following items:
25p raydavis:
1. Find contacts and communicate with them.
25p raydavis:
2. Create groups and pages, add content, and share with contacts.
25p raydavis:
3. Add widgets/goodies to your pages.
25p raydavis:
4. Update your profile information.
25p raydavis:
5. Find a buddy and test permissions on content.

What test plan you ask? Oh the one Max is going to write <someday>.

PublicViews - Unauthenticated pages

The PublicView.jmx test plan follows a simple unauthenticated use case. The typical landing page (/dev/index.html) is retrieved, with each of its constituent elements pulled as well. A loop retrieves a handful of public user profiles. The thread ends with a simple get of the Acknowledgements page.

The test plan runs 5 threads, each for 1 loop. The test assumes the existence of 2000 test accounts with username format 'tester*N*' as created by the users.pl script.

To run the Public View test plan from the command line:

./runPublicViews.sh

Results are written to results/PublicViews.jtl. Multiple runs append results to the same file.

To view the results, open the PublicViews.jmx file in the JMeter GUI, click Summary Report, and enter the PublicViews.jtl location in the Filename box.

Customizing PublicViews.jmx
This test plan is configured with intelligent defaults. It can be modified either in the JMeter GUI or in a text editor.

  • Load test host and port - defaults are localhost and 8080** JMeter GUI: Click HTTP Request Defaults. Under Web Server enter your values for Server Name or IP and Port Number.
    • Edit PublicViews.jmx: Change <stringProp name="HTTPSampler.domain"> from localhost to your host name and <stringProp name="HTTPSampler.port"> from 8080 to your port.
  • Threads and loops - defaults are 5 threads and 1 loop.
    • JMeter GUI: Select the Test Plan title Call Series of Public Pages. Modify the values of the LOOPS and THREADS User Variables.
    • Edit PublicViews.jmx: In the <collectionProp name="Arguments.arguments"> element, change the Argument.value for the LOOPS and THREADS elements.
  • Number of Profiles Retrieved on each thread loop - default is 5** JMeter GUI: Select the Loop Controller title Retrieve 5 User Profiles. Modify the value of the Loop Count. Remember to update the title as well.
    • Edit PublicViews.jmx: In the <LoopController> element, modify the LoopController.loops element value. Remember to change the testname to reflect the changed count.
  • Test user profiles - default is ~tester1 to ~tester2000** JMeter GUI: Expand the Loop Controller title Retrieve 5 User Profiles and select Get Random Profile. In the {{Path} field, the value ~tester${__Random(1,2000)} appends a random number between 1 and 2000 to the string '~tester', for example '~tester1246'.
    • Yah, do this is the GUI. Don't muck with the PublicViews.jmx to change the loop value generator.

Contributing Load Test results

How do you register your results so they'll be useful to others?

Contributing Test Plans

How do you add test plans to github, so the load test base expands?