Users locked out of Sakai

Description

Note:

This condition occurred after our Oracle database crashed.

I am recording it as an incident only, with a cataloging of symptoms in the UI and database and our course of resolution.

===

users locked out of sakai - when they try to log in they are met with the Velocity template error.

typical user emai:

from (USER):

I have been trying to access coursework.stanford.edu since 8:00 PM
when the website finally started functioning. Everytime I attempt to
login, I am redirected to the portal as I should be. Then, when I
click on the portal link, I am redirected to an Error page. In other
words, when I enter my SUNet ID and password, I am authenticated but
redirected to an error page with bug report. Please get back to me as
soon as possible!

It doesn't seem to be a problem with my computer since I have tried to
access the site through my roomate's computer (my roomate has been
able to access the site with her laptop).

My SUNet ID is USER

Please get back to me as soon as possible!

====

support gets dozens and dozens of these in short period...and it starts to escalate.

====

------- Additional Comment #12 From Christine Doherty 2007-09-26 20:35 [reply] -------
How is this for some more data points: List of SUNet IDs who put in HelpSU
tickets or emailed today. Many of them put in multiple tickets. This is brutal.
There are way more tickets for this issue than when the whole system was down.
There is nothing we can do for these students when there are assignments they
have to take online.

(...long list removed)

======

debugging found that the users had lost permissions to visit their own sites, if the
site still existed. A number of users lost references to their home sites, including
our part-time dba:

from lydia, near the end of the crisis:

====

Just to clarify, we ran the retroactive script this morning and that
appeared to fix bug 1114. Julian and I tried SU to a dozen of those
users, and didn't get the template error.

Now the error Sam got when trying to login earlier was different from
bug 1114, although it also seemed that his myworkspace records were
corrupted. (There was no record for his home site in sakai_site
table, even though he's logged in in the past few days) Ran the
script for his My Workspace fixed his problem too.

How these people's myworkspace sites got corrupted at the first place
is still a mystery. So, keep in mind that it's possible that these
bugs might happen again.

====

The 'retroactive' script lydia referred to is the now-standard Delete
My Workspace script used by various schools to change the set of
tools on individual sites by, well, mass deleting the sites.

The solution to this corrupted MyWorkspace Site/Realm is to
delete the users MyWorkspace

Activity

Filter Maintainer August 8, 2008 at 6:03 AM

2.5.0 QA has been completed. Unfortunately, there were not resources available to specifically test this issue. If you find this to still be a problem in the latest release (at this point 2.5.2), please reopen the JIRA and add the latest affects version.

Ian Boston October 19, 2007 at 5:35 AM

I have fixed the VM errors, but I am not certain that this was the bug reported here.

If it was, can you test and close

Thanks.
Ian

Casey Dunn October 16, 2007 at 1:15 PM

it was maskign the real error.
if someone is running w/out the template patch, they will get the 'masked' condition.
that's why I recorded it. I believe it is important to log symptoms esp in those situations where
there are layers of symptoms. This is also why I wrote that preamble, where I was recording this as an incident. in our triage we first record incidents, and then we see if we can find bugs - they are bugs when they are reproducible. (of course sometimes they are immediately reproducable)

when I single stepped through one of the accounts logging in I found that it was never given a grant to their MyWorkspace.

in some spot checking some folks were missing their my workspace. others were missing the visit grant to their workspace. they were generally fouled up.

When lydia pulled the big trigger, deleting ALL the users my workspaces, new one were created and we plowed on to the next crisis.

Stephen Marquard October 16, 2007 at 1:08 PM

I just can't see an obvious connection between velocity template issues and database inconsistencies in realms / sites, unless the vm template is masking the real error.

Casey Dunn October 16, 2007 at 1:03 PM

here stephen, perhaps this will bring back some memories from September.:

I'm reading through the thousand or so Jiras now, but
in the meantime has anyone experience with
these exceptions - we're getting lots of them
now that the Students have shown up.

this happens for both authenticated users and unauthenticated
users, multiple paths.

— snippet —

request-path: /portal/site/F07-PWR-1-06
time: Sep 23, 2007 03:59:22

stack trace:

org.sakaiproject.portal.api.PortalHandlerException: java.lang.RuntimeException: Failed to render template
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:718)
caused by: java.lang.RuntimeException: Failed to render template
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doError(SkinnableCharonPortal.java:221)
caused by: org.apache.velocity.exception.ParseErrorException: Encountered "s" at line 3, column 7.
Was expecting one of:
"[" ...
"(" ...
<STRING_LITERAL> ...
"true" ...
"false" ...
<NUMBER_LITERAL> ...
<IDENTIFIER> ...
"{" ...
"!" ...

---------------------- This automatic notification message was sent by Sakai Collab (https://collab.sakaiproject.org/portal) from the DG: Development (a.k.a. sakai-dev) site.
You can modify how you receive notifications at My Workspace > Preferences.

Re: Charon Parse Error - common? no bad?

by Stephen Marquard Sep 23, 2007; 11:12am :: Rate this Message:

Reply | Reply to Author | View Threaded | Show Only this Message
caseyd1 wrote:
> I'm reading through the thousand or so Jiras now, but
> in the meantime has anyone experience with
> these exceptions - we're getting lots of them
> now that the Students have shown up.
>
> this happens for both authenticated users and unauthenticated
> users, multiple paths.

We've never seen this on our system. It looks like a .vm template perhaps has a
syntax error in it. If you've modified any of the .vm files in portal, I'd start
looking there.

Cheers
Stephen

> –
> Stephen Marquard
> Learning Technologies Co-ordinator, Centre for Educational Technology
> Centre for Higher Education Development, University of Cape Town
> Email and XMPP (Jabber/GTalk): stephen.marquard@...
> <http://www.nabble.com/user/SendEmail.jtp?type=post&post=12848987&i=0>
> Cell: +27-83-500-5290
I've found a 2.5 patch to error.vm from Ian and Antranig (r 34511 for
sure, and)
which has edits matching this error message
(line 3, column 7, "s" found expecting blah blah)

This velocity error as masking some rampant MyWorkspace corruption
issues, but I'm sure it's masking more.

=----

hmm?

anyway I suspect the underlaying issue is like

Fixed

Details

Priority

Affects versions

Fix versions

Components

Assignee

Reporter

Environment

Sakai 2.4.x Oracle 10/Solaris8

Created October 12, 2007 at 3:46 PM
Updated October 23, 2008 at 6:29 AM
Resolved October 19, 2007 at 5:35 AM