Note:

this condition occurred after our Oracle database went into heavy swapping and became unstable. The database did not crash.

I am recording it as an incident only, with a cataloging of symptoms in the UI and database and our course of resolution.

This was discussed in the sakai-dev list in an email thread titled "users suddenly missing all their sites/tabs." that unfortunate title was due to the first bug reports coming in from the Instructor and TAs, for whom this was the only Site they were using. A fine introduction to Sakai, btw.

=====

A Sakai site became unavailable to members.

The user symptoms were:

1) disappearance from all members Tabs
2) disappearance from all members Membership My Sites list.
3) was not present on members 'hidden Tabs" list

Additional UI symptoms:

1) the Site is available to Administrators via WorksiteSetup and Sites tools
2) therein the Site was listed as Published, Joinable
3) All the Site members and their Roles were visible
4) the Site Realm was available
5 ) the Site was listed in the Joinable Sites list. Joining the site had no effect other than listing the new member in the above tools - the Site was not visible to the new members as described in the first set of symptoms.

Database symptoms:

SAKAI_SITE has an entry for the site
SAKAI_REALM has a Realm for the site
SAKAI_SITE_USER is empty for this site
SAKAI_REALM_RL_GR has each users entries for the Realm associated with the site.
SAKAI_REALM_RL_FN is empty for the Realm.

evidence of lack of transactional integrity.

Solution:

we used an earlier DB snapshot in a QA context to recreate the DB symptoms, and then confirmed the user observable error conditions.

We deleted and recreated the Site via the UI. This restored access to all members in QA.

We repeated the deletion and recreation procedure in Production.