performance problem related with user session checking

During our load testing for Sakai 2.9 release (with kernel 1.3.0), we've noticed increased app CPU compared to 2.7.2 codebase, whereas the response time, GC, db CPU remains comparable. This finding remains the same even when we reduced the load test scenarios down to user login/logout activities.

With yourkit profiler, we found ClusterEventService.notifyObservers() call consumes noticeable big amount of app cpu, which is new in 2.9 code. The stacktrace leads to the update() function inside newly added ActivityServiceImpl.java with KNL-498, where a db query for session table is needed whenever an event is generated with empty userId.

Even though userId is defined in Event API, none of the BaseEvent constructor actually assigns value to the userId field. So this means for all the events generated, the userId field will always be empty, and hence the ActivityService would execute a db query to SAKAI_SESSION table to find out the valid user id based on the session id. Furthermore, this query is not cached.

This is consistent with the Oracle AWR report:

CPU CPU per Elapsed
Time (s) Executions Exec (s) %Total Time (s) %CPU %IO SQL Id
---------- ------------ ---------- ------ ---------- ------ ------ -------------
....

246.0 3,426,660 0.00 4.0 256.0 96.1 .0 8807mqz8wgagc
Module: JDBC Thin Client
select SESSION_ID,SESSION_SERVER,SESSION_USER,SESSION_IP,SESSION_HOSTNAME,SESSIO
N_USER_AGENT,SESSION_START,SESSION_END,SESSION_ACTIVE from SAKAI_SESSION where S
ESSION_ID = :1
....

where this session table query was not costly by itself, it was however executed too many times, and led high app server CPU spending on db read for the returned data.

Here is a quick fix to this problem: always assign the user id (based on current session user) in post() calls.