How to Terracotta Cluster Enable a Sakai Tool

Introduction

This page will help a developer understand the basic ideas to Terracotta cluster enabling a Sakai tool. This page rely on the code changes documented in the High Level Design and Jira Task Description pages. The steps and ideas presented here are a guideline and will most likely not cover all situation that will come up when trying to cluster enable a tool.

Turn Terracotta On and Run Sakai

Follow the instructions in the System Administrator Guide to build and deploy a Terracotta Enabled Sakai. Once the Terracotta server is running, start the Sakai Tomcat server with Terracotta enabled. With Sakai running in Terracotta enabled mode, try to login and use your tool. If you have not updated the whitelist property, your tool should work normally. Now stop Sakai and update the tool whitelist property (see the System Administrator Guide for more information on this property) and add the tool id for the tool you wish to cluster enable.

clusterableTools@org.sakaiproject.tool.api.SessionManager=sakai.login,sakai.membership,
sakai.resources,sakai.dropbox,sakai.filepicker,sakai.resource.type.helper,
sakai.announcements,sakai.synoptic.announcement

Note: this should all be on one line, but has been wrapped for legibility

Start Sakai again with Terracotta enabled. Login and exercise your tool. You will most likely receive a Terracotta Portability exception in the catalina.out log file. These exceptions will help you know which classes must be instrumented for Terracotta.

Create a Terracotta Integration Module

Use one of the existing Sakai TIM modules as a reference to copy and create a version for the tool you are interested in porting. For example if I was interested in trying to cluster-enable the announcement tool, I might do something like this:

svn export https://source.sakaiproject.org/svn/unicon/content/branches/session-clustering-2-5-x/content-tim announcement/announcement-tim

After you have copied the previous tim, you should edit the pom.xml file and update it for your module name. It should be as simple as changing content to announcement. After you have updated the pom.xml file, you should edit the terracotta.xml file. It is recommended you start from an empty terracotta.xml file that looks like this:

<?xml version="1.0" encoding="UTF-8" ?>
<xml-fragment>
      <instrumented-classes>
      </instrumented-classes>
      <transient-fields>
      </transient-fields>
      <additional-boot-jar-classes>
      </additional-boot-jar-classes>
</xml-fragment>

After the terracotta.xml file is updated, you can try to build the announcement-tim module with the standard maven commands (e.g., {{mvn clean install}).

Next, add the announcement-tim module to the parent pom.xml.

Finally, you will need to modify the terracotta-config module to include your TIM as part of it's build process. You will need to modify the pom.xml and the tc-config.xml file in the terracotta-config module. You can copy/past/edit the content line as an example. In the case of the anouncements example, I would add the following line:

<copy
file="${settings.localRepository}/org/sakaiproject/tim-announcement/${sakai.version/tim-announcement-${sakai.version}.jar" 
tofile="${terracotta.config.tims}/org/sakaiproject/tim-announcement/${sakai.version/tim-announcement-${sakai.version}.jar"/>

to the pom.xml file and the following line:

<module name="tim-announcement" version="<at:var at:name="SAKAI_VERSION" />" group-id="org.sakaiproject"/>

to the tc-config.xml file. AFter these changes are all complete, the maven command referenced in the System Administrator Guide should be run within the terracotta-config directory:

code
mvn -Dterracotta.enable=true clean install
code

If you've successfully copied and modified all the right files in all the right places, you should see output in your build that includes something like this:

[copy] Copying 1 file to /home/holdorph/projects/sakai/apache-tomcat-5.5.26/sakai/tcotta/tims/org/sakaiproject/tim-announcement/M2

Add Classes for Instrumentation

After you add your tool to the whitelist and try to use that tool in a Terracotta enabled Sakai, you should start seeing Terracotta portability exceptions for anything that is added to the Session. The beginning of the stacktrace might look like this.

com.tc.exception.TCNonPortableObjectError: 
*******************************************************************************
Attempt to share an instance of a non-portable class by passing it as an argument to a method of a
logically-managed class. This unshareable class has not been included for sharing in the
configuration.

For more information on this issue, please visit our Troubleshooting Guide at:
http://terracotta.org/kit/troubleshooting

Thread                      : http-8080-Processor23
JVM ID                      : VM(5)
Logically-managed class name: java.util.concurrent.ConcurrentHashMap
Logical method name         : put(Object,Object)
Non-included classes        : org.sakaiproject.announcement.tool.AnnouncementActionState, org.sakaiproject.cheftool.ControllerState

Action to take:

1) Reconfigure to include the unshareable classes
   * edit your tc-config.xml file
   * locate the <dso> element
   * add this snippet inside the <dso> element

       <instrumented-classes>
         <include>
           <class-expression>org.sakaiproject.announcement.tool.AnnouncementActionState</class-expression>
         </include>
         <include>
           <class-expression>org.sakaiproject.cheftool.ControllerState</class-expression>
         </include>
       </instrumented-classes>

   * if there is already an <instrumented-classes> element present, simply add
     the new includes inside it

It is possible that some or all of the classes above are truly non-portable, the solution
is then to mark the referring field as transient.


*******************************************************************************

	at com.tc.object.ClientObjectManagerImpl.throwNonPortableException(ClientObjectManagerImpl.java:826)
	at com.tc.object.ClientObjectManagerImpl.checkPortabilityOfLogicalAction(ClientObjectManagerImpl.java:800)
	at com.tc.object.tx.ClientTransactionManagerImpl.logicalInvoke(ClientTransactionManagerImpl.java:740)
	at com.tc.object.TCObjectLogical.logicalInvoke(TCObjectLogical.java:20)
	at com.tc.object.bytecode.ManagerImpl.logicalInvoke(ManagerImpl.java:235)
	at com.tc.object.bytecode.ManagerUtil.logicalInvoke(ManagerUtil.java:287)
	at java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:430)
	at java.util.concurrent.ConcurrentHashMap.put(Unknown Source)
	at org.sakaiproject.tool.impl.MyLittleSession.setAttribute(MyLittleSession.java:278)

As you can see, Terracotta tries to be extremely helpful providing you with the specific information you'll need for the next steps.

Add class to terracotta.xml file

As the stacktrace points out, you will need to add a new section to the terracotta.xml file you created above (as part of creating the TIM module). Given the stacktrace above, my terracotta.xml would now look like this:

<?xml version="1.0" encoding="UTF-8" ?>
<xml-fragment>
      <instrumented-classes>
         <include>
           <class-expression>org.sakaiproject.announcement.tool.AnnouncementActionState</class-expression>
         </include>
         <include>
           <class-expression>org.sakaiproject.cheftool.ControllerState</class-expression>
         </include>
      </instrumented-classes>
      <transient-fields>
      </transient-fields>
      <additional-boot-jar-classes>
      </additional-boot-jar-classes>
</xml-fragment>

Determine if class has any transient fields

Some classes that need to be instrumented and stored in the Terracotta cluster might contain fields that can not be shared. An example of a non-shareable field could be a database connection, a socket, a file handle or just simply a reference to a service or component. Any fields that contain values that should not be shared must be marked as transient. There are two ways to accomplish declaring a field as transient. A field can use the Java transient keyword and use the Terracotta <honor-transients> designation in the terracotta.xml file. Or a field can be simply explicitly declared as transient in the terracotta.xml file. Here are examples of both ways.

Using the Java transient keyword

public class Foo
{
    private transient Connection conn;
...

      <instrumented-classes>
         <include>
           <class-expression>org.sample.Foo</class-expression>
           <honor-transient>true</honor-transient>
         </include>
      </instrumented-classes>

Telling Terracotta Directly that a Field is Transient

public class Foo
{
    private Connection conn;
...

<instrumented-classes>
    <include>
        <class-expression>org.sample.Foo</class-expression>
    </include>
</instrumented-classes>
<transient-fields>
    <field-name>org.sample.Foo.conn</field-name>
</transient-fields>

Create a mechanism for resolving transient fields if necessary

coming soon

Promote inner classes to top level classes

coming soon

Retest and Repeat

After you have added the class for instrumentation, checked for transient fields, created a mechanism for resolving transient fields and promoted an inner class to top level class (if necessary), then it is time to retest. Rerun the build process to create your tool's TIM and rerun the terracotta-config build to deploy the tc-config.xml and TIMs. Test your tool again. If you still see exceptions repeat the process. This is one iterative way to work your way through cluster enabling a Sakai tool.