A Tour of Sakai Import

The first thing to understand about this code is the motivation behind it. For a long time (maybe since the beginning), Sakai has shipped with a feature in the Site Info tool called "Import from file." The idea is that the maintainer of the site will upload a specially-formatted zip file that contains an archive of all her course materials. The tool works like a wizard, with four screens:

1. Asks for a file to upload.
   
2. Shows you a list of categories of content in your archive and allows you to choose the items that should be pulled in.
   
3. Displays a confirmation screen with a summary of what is about to happen and a "Finish" button.
   
4. Upon completion, it displays an "Import Complete" message and an "Ok" button.
   

The problem with this feature from the get-go was that the only archive files this tool understood were those produced by Sakai's archival functionality and Indiana University's legacy Oncourse system. The initial motivation then was to find a way to substitute other archive file formats for this one, namely Blackboard 5.5 archives in use at Texas State University.

All the code for reading and parsing an archive file was in the Site Info tool code itself, in a 12,000+ line file called SiteAction.java. The new design needed to factor the archive parsing and processing out into an independent module and provide the flexibility to support new archive types without breaking the existing support for Indiana's files.

During integration week for Sakai 2.3, the import code was moved into the Sakai trunk from contrib, where it had been under development. It was decided that the appropriate place for it to live would be within the existing archive module. The existing archive code includes the ArchiveService which has been used for importing and exporting in Sakai, including handling the archives in "Import from file."

The long-range plan is for Sakai to support the IMS Common Cartridge specification natively for both importing and exporting. The archive module will gradually acquire and consolidate all these capabilities, but for now it contains two distinct pieces: the original archive service, and the new import architecture. This document exclusively deals with the latter portion of this module.

The import side of things consists of five directories: import-api, import-impl, import-parsers, import-handlers, and import-pack. We will discuss each of these in turn, but first let us address a high-level overview of the approach to archive importing.

One of the important facets of the archive import problem is that archives can come from learning management systems other than Sakai. This means that there is not a clear-cut mapping between resources in an archive and the tools they will inhabit in Sakai. Also, since Sakai represents a large and growing ecosystem of tools, the design for parsing archives should be decoupled from the design for pushing archive resources into the various appropriate tools.

Thus the key architectural feature of the import code is that it divides the problem into two parts: reading an archive file and extracting the pieces of content out is done by exactly one parser. Taking the content pieces and stuffing them into Sakai tools is done by multiple handlers, one handler for each Sakai tool.

A parser is written to accommodate all the specific features of a particular archive format. For example, there is a parser for Blackboard 5.5 archives, and another parser for Blackboard 6.0 archives, because the formats are different. There is yet another parser for IMS Common Cartridge 1.0. Note that only the Common Cartridge and the original Sakai format parsers are included in the Sakai release; Other parsers are available in contrib/migration/import.

The key to decoupling the parsing from the handling is that the parser must produce a collection of vendor-neutral content objects. This means that by the time they get to the handler, they are generic, and don't have any vendor-specific formatting. This ensures that the handlers don't have to have any code in them to deal with the particulars of any single format, which in turn means that the same handlers can be used again and again for any archive format, present or future.

These generic content objects still have types which identify what kind of content they are. Some examples of these content object types are: WebLink, Assessment, Announcement, etc. Each content class must implement an interface called Importable. As a group, these content objects are referred to as importables. An object called an ImportDataSource acts as a container for the group of importables. We can ask an ImportDataSource for the categories for that archive, and ask it for the Importables that match a given list of categories.

Sitting between the parsers and the handlers is the ImportService. The default implementation of the ImportService, BasicImportService, keeps a list of available parsers and a list of available handlers, both of which are Spring-injected and configurable in a components.xml file. Here is how a hypothetical client of the ImportService might use it:

ImportService importService = org.sakaiproject.importer.cover.ImportService.getInstance();
byte[] fileData = fileFromUpload.get();
if (importService.isValidArchive(fileData)) {
    ImportDataSource importDataSource = importService.parseFromFile(fileData);
    List lst = importDataSource.getItemCategories();
    importService.doImportItems(lst, siteId);
    }

Here are two simple data flow diagrams illustrating the two steps of the import process:

1. parse an archive file and get back an ImportDataSource
   
2. pass a collection of importables to the handlers, which then push content into Sakai
   

Edit import-pack/src/webapps/WEB-INF/components.xml file

Configure parsers

By default, the only parser that is configured with Sakai is the original Sakai archive parser. The common cartridge parser is commented out. You can add common cartridge support to import by uncommenting the common cartridge bean.

	<!-- File parsers -->
	<bean id="org.sakaiproject.importer.api.ImportFileParser-Sakai"
			class="org.sakaiproject.importer.impl.SakaiArchiveFileParser"
			singleton="false">
	</bean>

<!--
    <bean id="org.sakaiproject.importer.api.ImportFileParser-CommonCartridge"
			class="org.sakaiproject.importer.impl.CommonCartridgeFileParser"
			singleton="false">
	</bean>
-->

You also have to add your chosen parsers as properties of the ImportService bean. In this case, again, the Sakai archive parser is enabled by default, and the common cartridge must be uncommented:

<property name="parsers">
    <list>
      <ref bean="org.sakaiproject.importer.api.ImportFileParser-Sakai"/>
 <!-- <ref bean="org.sakaiproject.importer.api.ImportFileParser-CommonCartridge"/> -->
    </list>
</property>

Configure handlers

A handler is code that works on behalf of a Sakai tool and contributes import content to that tool. You must uncomment the handlers that you want available to the import. The original Sakai input format doesn't use any handlers, because it still uses the ArchiveService to push things into Sakai. The handlers are used by common cartridge as well as any other parsers you configure.

	<!-- Handlers -->
<!--
	<bean id="org.sakaiproject.importer.impl.handlers.AnnouncementHandler"
			class="org.sakaiproject.importer.impl.handlers.AnnouncementHandler"
			singleton="true">
		<property name="announcementService">
			<ref bean="org.sakaiproject.announcement.api.AnnouncementService" /> 
		</property>
	</bean>
-->
<!--
	<bean id="org.sakaiproject.importer.impl.handlers.ResourcesHandler"
			class="org.sakaiproject.importer.impl.handlers.ResourcesHandler"
			singleton="true">
	</bean>
-->
<!--
	<bean id="org.sakaiproject.importer.impl.handlers.SamigoHandler"
			class="org.sakaiproject.importer.impl.handlers.SamigoHandler"
			singleton="true">
	</bean>
-->
<!--
	<bean id="org.sakaiproject.importer.impl.handlers.MessageCenterHandler"
			class="org.sakaiproject.importer.impl.handlers.MessageCenterHandler"
			singleton="true">
	</bean>
-->

And just like the parsers, the handlers must be set as properties of the ImportService bean:

<property name="resourceHandlers">
    <list>
      <!-- <ref bean="org.sakaiproject.importer.impl.handlers.AnnouncementHandler" /> -->
      <!-- <ref bean="org.sakaiproject.importer.impl.handlers.ResourcesHandler"/> -->
      <!-- <ref bean="org.sakaiproject.importer.impl.handlers.SamigoHandler"/> -->
      <!-- <ref bean="org.sakaiproject.importer.impl.handlers.MessageCenterHandler"/> -->
    </list>
</property>

Edit import-pack/project.xml

There are two dependencies in import-pack/project.xml that need to be uncommented if you decide to use the Samigo handler. Again, the handlers don't do anything for the original Sakai archive format, so don't bother uncommenting the dependencies for the Samigo handler unless you are using the common cartridge parser or one of the other parsers available at https://source.sakaiproject.org/contrib/migration/trunk/import-parsers