How to Terracotta Cluster Enable a Sakai Tool

Introduction

This page will help a developer understand the basic ideas to Terracotta cluster enabling a Sakai tool. This page rely on the code changes documented in the High Level Design and Jira Task Description pages. The steps and ideas presented here are a guideline and will most likely not cover all situation that will come up when trying to cluster enable a tool.

Turn Terracotta On and Run Sakai

Follow the instructions in the System Administrator Guide to build and deploy a Terracotta Enabled Sakai. Once the Terracotta server is running, start the Sakai Tomcat server with Terracotta enabled. With Sakai running in Terracotta enabled mode, try to login and use your tool. If you have not updated the whitelist property, your tool should work normally. Now stop Sakai and update the tool whitelist property (see the System Administrator Guide for more information on this property) and add the tool id for the tool you wish to cluster enable.

clusterableTools@org.sakaiproject.tool.api.SessionManager=sakai.login,sakai.membership,
sakai.resources,sakai.dropbox,sakai.filepicker,sakai.resource.type.helper,
sakai.announcements,sakai.synoptic.announcement

Note: this should all be on one line, but has been wrapped for legibility

Start Sakai again with Terracotta enabled. Login and exercise your tool. You will most likely receive a Terracotta Portability exception in the catalina.out log file. These exceptions will help you know which classes must be instrumented for Terracotta.

Create a Terracotta Integration Module

Use one of the existing Sakai TIM modules as a reference to copy and create a version for the tool you are interested in porting. For example if I was interested in trying to cluster-enable the announcement tool, I might do something like this:

svn export
https://source.sakaiproject.org/svn/msub/unicon.net/content/branches/session-clustering-2-5-x/content-tim
announcement/announcement-tim

After you have copied the previous tim, you should edit the pom.xml file and update it for your module name. It should be as simple as changing content to announcement. After you have updated the pom.xml file, you should edit the terracotta.xml file. It is recommended you start from an empty terracotta.xml file that looks like this:

<?xml version="1.0" encoding="UTF-8" ?>
<xml-fragment>
      <instrumented-classes>
      </instrumented-classes>
      <transient-fields>
      </transient-fields>
      <additional-boot-jar-classes>
      </additional-boot-jar-classes>
</xml-fragment>

After the terracotta.xml file is updated, you can try to build the announcement-tim module with the standard maven commands (e.g., {{mvn clean install}).

Next, add the announcement-tim module to the parent pom.xml.

Finally, you will need to modify the terracotta-config module to include your TIM as part of it's build process. You will need to modify the pom.xml and the tc-config.xml file in the terracotta-config module. You can copy/past/edit the content line as an example. In the case of the anouncements example, I would add the following line:

<copy
file="${settings.localRepository}/org/sakaiproject/tim-announcement/${sakai.version/tim-announcement-${sakai.version}.jar" 
tofile="${terracotta.config.tims}/org/sakaiproject/tim-announcement/${sakai.version/tim-announcement-${sakai.version}.jar"/>

to the pom.xml file and the following line:

<module name="tim-announcement" version="@SAKAI_VERSION@" group-id="org.sakaiproject"/>

to the tc-config.xml file. AFter these changes are all complete, the maven command referenced in the System Administrator Guide should be run within the terracotta-config directory:

mvn -Dterracotta.enable=true clean install

If you've successfully copied and modified all the right files in all the right places, you should see output in your build that includes something like this:

[copy] Copying 1 file to /home/holdorph/projects/sakai/apache-tomcat-5.5.26/sakai/tcotta/tims/org/sakaiproject/tim-announcement/M2

Add Classes for Instrumentation

After you add your tool to the whitelist and try to use that tool in a Terracotta enabled Sakai, you should start seeing Terracotta portability exceptions for anything that is added to the Session. The beginning of the stacktrace might look like this.

com.tc.exception.TCNonPortableObjectError: 
*******************************************************************************
Attempt to share an instance of a non-portable class by passing it as an argument to a method of a
logically-managed class. This unshareable class has not been included for sharing in the
configuration.

For more information on this issue, please visit our Troubleshooting Guide at:
http://terracotta.org/kit/troubleshooting

Thread                      : http-8080-Processor23
JVM ID                      : VM(5)
Logically-managed class name: java.util.concurrent.ConcurrentHashMap
Logical method name         : put(Object,Object)
Non-included classes        : org.sakaiproject.announcement.tool.AnnouncementActionState, org.sakaiproject.cheftool.ControllerState

Action to take:

1) Reconfigure to include the unshareable classes
   * edit your tc-config.xml file
   * locate the <dso> element
   * add this snippet inside the <dso> element

       <instrumented-classes>
         <include>
           <class-expression>org.sakaiproject.announcement.tool.AnnouncementActionState</class-expression>
         </include>
         <include>
           <class-expression>org.sakaiproject.cheftool.ControllerState</class-expression>
         </include>
       </instrumented-classes>

   * if there is already an <instrumented-classes> element present, simply add
     the new includes inside it

It is possible that some or all of the classes above are truly non-portable, the solution
is then to mark the referring field as transient.


*******************************************************************************

	at com.tc.object.ClientObjectManagerImpl.throwNonPortableException(ClientObjectManagerImpl.java:826)
	at com.tc.object.ClientObjectManagerImpl.checkPortabilityOfLogicalAction(ClientObjectManagerImpl.java:800)
	at com.tc.object.tx.ClientTransactionManagerImpl.logicalInvoke(ClientTransactionManagerImpl.java:740)
	at com.tc.object.TCObjectLogical.logicalInvoke(TCObjectLogical.java:20)
	at com.tc.object.bytecode.ManagerImpl.logicalInvoke(ManagerImpl.java:235)
	at com.tc.object.bytecode.ManagerUtil.logicalInvoke(ManagerUtil.java:287)
	at java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:430)
	at java.util.concurrent.ConcurrentHashMap.put(Unknown Source)
	at org.sakaiproject.tool.impl.MyLittleSession.setAttribute(MyLittleSession.java:278)

As you can see, Terracotta tries to be extremely helpful providing you with the specific information you'll need for the next steps.

Add class to terracotta.xml file

As the stacktrace points out, you will need to add a new section to the terracotta.xml file you created above (as part of creating the TIM module). Given the stacktrace above, my terracotta.xml would now look like this:

<?xml version="1.0" encoding="UTF-8" ?>
<xml-fragment>
      <instrumented-classes>
         <include>
           <class-expression>org.sakaiproject.announcement.tool.AnnouncementActionState</class-expression>
         </include>
         <include>
           <class-expression>org.sakaiproject.cheftool.ControllerState</class-expression>
         </include>
      </instrumented-classes>
      <transient-fields>
      </transient-fields>
      <additional-boot-jar-classes>
      </additional-boot-jar-classes>
</xml-fragment>

Determine if class has any transient fields

Some classes that need to be instrumented and stored in the Terracotta cluster might contain fields that can not be shared. An example of a non-shareable field could be a database connection, a socket, a file handle or just simply a reference to a service or component. Any fields that contain values that should not be shared must be marked as transient. There are two ways to accomplish declaring a field as transient. A field can use the Java transient keyword and use the Terracotta <honor-transients> designation in the terracotta.xml file. Or a field can be simply explicitly declared as transient in the terracotta.xml file. Here are examples of both ways.

Using the Java transient keyword

public class Foo
{
    private transient Connection conn;
...

<instrumented-classes>
    <include>
        <class-expression>org.sample.Foo</class-expression>
        <honor-transient>true</honor-transient>
    </include>
</instrumented-classes>

Telling Terracotta Directly that a Field is Transient

public class Foo
{
    private Connection conn;
...

<instrumented-classes>
    <include>
        <class-expression>org.sample.Foo</class-expression>
    </include>
</instrumented-classes>
<transient-fields>
    <field-name>org.sample.Foo.conn</field-name>
</transient-fields>

Create a mechanism for resolving transient fields if necessary

Once a field has been marked as transient, the field will no longer be shared to other nodes in the cluster. In many cases this will cause a problem. Many objects will need to have their transient fields repopulated on any JVM the object is moved to. There are a couple of common ways to resolve transient fields with Terracotta.

Write a method
You can write a separate method in the class that contains the transient field. You then specify that method for being run in the terrracotta.xml file whenever the object is moved to a new JVM. The following example demonstrates the change you would make to the terracotta.xml file to call a method to resolve transient fields. In the example below resolveTransientFields would be the name of a no argument method in the Foo class.

<instrumented-classes>
    <include>
        <class-expression>org.sample.Foo</class-expression>
        <on-load><method>resolveTransientFields</method></on-load>
    </include>
</instrumented-classes>
<transient-fields>
    <field-name>org.sample.Foo.conn</field-name>
</transient-fields>

Inline Java or Beanshell Code
An alternative to modifying the .java code directly, is to instead write the code directly in the terracotta.xml file. This has the advantage of avoiding a change to the .java file. It has a disadvantage to not being checked by a Java compiler during your system build. The following example demonstrates modifying the terracotta.xml file to write Java code in the XML file to populate a transient field.

<instrumented-classes>
    <include>
        <class-expression>org.sample.Foo</class-expression>
        <on-load><execute><![CDATA[self.conn = new Connection("example");]]></execute></on-load>
    </include>
</instrumented-classes>
<transient-fields>
    <field-name>org.sample.Foo.conn</field-name>
</transient-fields>

Promote inner classes to top level classes

The Sakai source code has many Java classes which contain inner classes. Inner classes present a problem for Terracotta clustering, when the desire is to cluster/share the inner class, while NOT cluster/sharing the outer class. The reason this problem exists, is that inner classes contain an implied reference to their outer class. This implied reference is just like a member field reference from the Terracotta point of view. Yet, there does not exist any Java syntax to mark this implied reference as transient.

In order to support this code, one of two solutions must be chosen. Either the inner class must be made static, therefore eliminating the implied reference to the other class, or the inner class must be refactored to a top level class. It is recommended the inner class be refactored to a top level class. If the class is left as a static inner class, the outer class can directly reference the inner class fields, without using a mutator method. This is a common Terracotta problem (see this problem documented on the Terracotta web site here). To avoid this pitfall, the simplest solution is to make the fields private and only provide accessor methods. However, in the case of inner classes, private does not mean anything to the outer class. Therefore there is no Java syntax to prevent you from doing the wrong thing accidentally. However, making the inner class a top level class, will provide you with a Java compile time check against accidentally accessing a private field directly.

It is strongly recommended to promote any cluster-enabled inner class to a top level class, if the existing outer class is not going to be cluster enabled.

Retest and Repeat

After you have added the class for instrumentation, checked for transient fields, created a mechanism for resolving transient fields and promoted an inner class to top level class (if necessary), then it is time to retest. Rerun the build process to create your tool's TIM and rerun the terracotta-config build to deploy the tc-config.xml and TIMs. Test your tool again. If you still see exceptions repeat the process. This is one iterative way to work your way through cluster enabling a Sakai tool.