CC App Devel Notes - P1

May 15, 2008

Met with Brandon Muramatsu of Utah State and the COSL project (OCW) and Jeff Merriman of MIT and the OKI project. Together we finalized details of the Common Cartridge Installer application. The intent is to use OKI OSID 3.0 technology to install OCW content packaged as IMS-CC into Sakai. Later, we're going to try to make the application install into other environments including:

• Pure web server environments (Apache)
• Moodle (via web service)
• BlackBoard (via building blocks)

Funding has been approved to build the prototype application as a Sakai tool. Money comes from a Hewlett Foundation grant to USU. This project will be presented at Open-i-World in Lyon, France at the end of June, and at the Sakai conference in Paris - start of July.

May 17. 2008

Portability of software is one of the primary goals we're trying to demonstrate with this project. For that reason, I'm thinking of using JSP as the UI technology. Aaron Zeckoski kindly provided the following pointer to a simple JSP app that he worte that runs in Sakai:

https://source.sakaiproject.org/contrib/programmerscafe/trunk/samples/helloWorldUser/

Downloaded the code. In part, it leverages a JSP adapator servlet. I was aware that this existing, but not quite what it was called or where it lived. This hello world app simplifies the startup time considerably. My experience with JSP and JSTL is very limited, but many of the same concepts come up in other PT approaches like Struts and JSF. It will be a good learning experience for me, but if it proves too much in the short term, I'll fall back onto RSF and code it that way. I have code that I can re-use for a prototype app that would allow me to focus more on CC parsing and OKI OSID implementations. Still, JSP is a more general approach.

I'm planning on calling the application cc-install (Common Cartridge Installer). COSL will be writing an OCW CC packager, so I don't have to worry about that at this time, though it would be interesting to port it into Sakai for general use. I tmight encourage the use of both tools in the Sakai community.

May 19, 2008

Got out my books on JSP. I'm thinking that figuring out JSP may take more time than I have available to me to get something up and running. Using RSF would be a lot simpler, especially considering that I have code lying around that can be re-used to allow a CHS collection to be selected and support for file upload for installation.
That will allow me to concentrate on the OSID implementation and parsing the IMS-CC manifest.

May 28, 2008

The cc-install module was cloned from the Sousa page tool, since it already has most of the support structure needed, in terms of selecting a target collection. However, there is a lot of stuff that isn't needed. Stripped Sousa code out of PageObjectProducer. Resources will be shown as generic files. Drop down collection navigation will be retained. A single "Install" link on the right will allow common cartridges to be installed. The link will go to a POST form page that will collect the name of the cartridge to be installed and the ZIP file to upload.

Removed links on resources. Changed resources to a generic icon. Collections and resource view works.
Pruned out unneeded sosua files and removed them from the applicationContext. Removed unused templates and graphics.\

Stripped Sousa code out of EditItemProducer. Removed all parameters. Removed description line. Edited field labels. Brought in file input field from one of the content handlers (image). Page is displaying.

Code is ready collect the zip file to be uploaded. Need to create a results producer. Also change package names away from Sousa.

May 29, 2008

Renamed package path from org.sakaiproject.sousa.tool to org.sakaiproject.cc.tool.
Renamed PageObjectProducer to CCToolProducer.
Renamed PageObjectParameters to CCToolParameters.
Renamed EditItemProducer to UploadProducer.
Renamed EditItemParameters to UploadParameters.
Fixed requestContext references.

Works (after some debugging).

Deleted unneeded files. Stripped out sousa code from UploadItemBean. Created a ResultsProducer. All of this works now.

I'm to the point of doing something with a real zip file. I think I might start on a simple zip and see how that goes.
Created a very simple zip file with two images and a text file (no manifest yet) called "test_ims_cc.zip".
Modified UploadItemBean to save file name and input stream into session state. Modified ResultsProducer to display uploaded file name. Works.

Jun. 5, 2008

Got the initial Filing OSID specification as a PDF of the Excel file. These are the abstract definitions. Worked up a preliminary set of Java bindings. There are many dependencies on other OSIDs (authentication, type, id, etc.).

Jun. 6, 2008

Tom Coppeto has sent the initial set of Java bindings for Filing. It's still missing some dependencies, but its a good start. Created a Filing OSID document in confluence to document details of the implementation.

I didn't really expect 26 interfaces for filing and that doesn't include the dependent OSIDs, most of which will need to be implemented as well. Fortunately, not all of the File OSIDs will be needed for the project. I can focus on FilingSession, DirectoryEntry, Directory, and File, I think.

Moved the new sources into a Sakai project. Tried to compile it. There was a couple of syntax errors in AuthenticationValidationSession. Once those were fixed, 31 unresolved symbol references were left, mostly dependencies not present in the generated interface sources.

Jun. 7, 2008

Created a minimum version of OSID v3 based on the sources sent by Tom yesterday. All unneeded filing interfaces were removed. Other unresolved interface references were commented out in unused services (authentication, etc.). I had to create org.osid.transport.DataInputStream and org.osid.transport.DataOutputStream by hand, since they were not included in Tom's initial sources, but required by the FilingOsid. They are stubbed to return the java equivalents. There might be a more clever way to finesse this - perhaps by extending the appropriate java.io interfaces and classes, but this will work for now. The api-v3-min compponent compiles without errors now.

Coded the chsFilingProfile implementation. Compiles without error. Lots of things are stubbed to return null or false.
Created a stubbed version of the FilingManager in chsFilingManager. Many are commented out, rest throw Unimplemented.

Jun. 10, 2008

Created a CC Test application. This will be something that I can call out to the Filing OSID and make sure that it works.

Broke out v3 osid components into their own artifact id's, since I was getting a conflict in the Presentation Tool looking for a v2 OSID.

Jun. 11, 2008

Registered the FilingManager as a Sakai component. This should give me access to the singleton, but it's not working so far.
The error is:

This is a bit odd. I verified that chsFilingManager is present in the pack'ed component. Spring should find it.

Another hour spent looking for the problem hasn't fixed it. Even though the OSID implementation is being bundled into bot the pack file and the CC-Test app, the class loader can't find it. I don't have time to fool around with this, in spite of being sure that it is a simple configuration problem - somewhere. Meanwhile, I'm just going to move the implementations into the API JAR and get on with it.

Copied all of the osid directory into xosid. Stripped out EVERYTHING except a single FilingManager API and an implementation called xFilingManager. Replace reference to this in the cc-test app. App is able to find the FilingManager bean. It figures. I think something got corrupted. Too many versions of things, etc.

Well, now it's just a matter of migrating the work done previously into this new cluster of Sakai components. It's really only the sources, so they should be easy to bulk move.

Naturally, I as soon as I moved the sources over, it got corrupted again. I just don't have the time to fool around with this.

Some of the problem seems to be that UnimplementedException is, itself, unimplemented. When I construct chsFilingManager, I get a class undefined error. Implemented the exception class, but needs to be bundled into the WAR.

Can now create a FilingManager using it's constructor. NOT the preferred approach, but I'll come back and fix it later.
Implemented DirectoryAdminSession, FileSession, DirectoryEntry, and File.
Implemented several exceptions.

Ready to test file creation.

Jun. 12, 2008

Flesh out code in chsFile. CHS entries are now being created.

Jun. 13, 2008

Fixed a number of bugs. Restructured the test tool to take a step by step approach. Process is now working clear through writing and reading data to/from a content resource. I'm also going to need the ability to create new directories. So I'll add that to the test app.

Added support to create directories.

At this point I have a working, minimal implementation of the v3 Filing OSID.

Did some work on file queries (lising files in a directory, etc.). Incomplete.

Jun. 16, 2008

Time is running out. Started integration of Filing OSID into the CC Installer app.
Checked in this API and impl as rev. 50316 into contrib/mnorton/osid3.

Basic file structures are working, including directories. Just need to copy data into files and it will work.

Commits always hang at this point.

Jun 20, 2008

More debugging into the CHS. I seem to be using the database storage method. Jim suggests:

Jun. 21, 2008

Added an out.write() to cc-test-tool and it worked completely (including read). Somehow, I'm having problems with IO blocking due to buffers being filled, or not be closed properly. Updated cc-install-tool, but still fails. I think it's because I am writting to the output stream and filling up it's buffer. Since nothing is consuming on the other side of this pipe (until commitI()), it just hangs. I could read the entire file into memory and write it out in one batch, but that will severely limit the size of the media included in the CC.

Jan. 22, 2008

Added an out.close() to simple test case and it worked. This lends evidence to the theory that data is not moving down the stream correctly. In the simple case of writing the entire data to the output stream provided by the FileSession, it works. However, it fails on the ZipStream case.

Jun. 23, 2008

I don't think that Piped I/O is quite the right approach for the FileSession implementation. Piped stream are blocking I/O designed to communicate between threads. Since I don't want to force applications to be a separate thread from the OSID implementation, I need to find a way to couple the data provided from the application to the CHS. I can think of two ways:

• Create an out of band agreement that enables the original data source to be connected to it's ultimate sink.
• Create a set of new I/O objects that allow data to be chained (piped, but that word is taken).

One of these is bound to work.

Created ChainedInputStream and ChainedOutputStream implemented by extending java.io.InputStream and java.io.OutputStream.
Streams are joined using a connect() method or an alternative constructor. Bi-directional connection is enforced.

Problems with this approach. java.io.OutputStream.write(int) is abstract and must be overridden. I tried to write it terms of one of the other write methods, but this is the base method called by the others leading to recursion and stack overflow. So much for layering on a bit of new functionality. The Piped I/O does work in some cases (short data, fully written to the pipe). I'm going to roll back to the Piped I/O for the simple case and create an out of band agreement to directly couple the zip input stream to the content input stream required by CHS.commit().

It just gets worse and worse. The above method SHOULD have worked - and it does for the first entry in the Zip archive. Sadly, CHS closes the nput stream, thus closing the zip stream, prevening subsequent entries from being read.

I think the only approach left to me is to unpack the zip archive into temporary files on the server, then save them as ContentResources one by one.

Sept. 2, 2008

Not much has happened on the OCW Installer project since presenting the initial work in France. I had one meeting with Jeff to discuss further work. We agree that there was lots to be done, but the work is awaiting funding approval.

I have some time available to me now so I'm cleaning things up a bit. Updated this page with notes from the conference. I note that the initial work was never checked into contrib. That needs to be done ASAP.

The v3 OSIDs are already checked into contrib/mnorton. Added cc-install and checked it in as rev. 52669.
Close investigation shows that the osid3 check-in happened before leaving for France. Substantial work was done on the trip and not checked in. Unfortunately, the work wasn't done in an SVN workspace. Perhaps the simplest thing to do is to check it in under a different name (osid3mjn).

All of the OSID-3 work done in France is now checked in under /contrib/mnorton/osid3mjn as rev 52672.
Deleted the old osid3 project in the same directory as rev 52673.

New Work
Created a clean copy of sakai_2-5-x and built it.

New work on CC-Install and the OSIDs going forward should be based on the new Java API assembled and published by Tom Copetto. It's very likely that this will break everything in sight.

Things To Do

• Install new OSID v3 API from Tom and get it to build.
• Create mvn2 POMs
• Update existing Filing and Transport impls to new API.
• Make sure it still works with cc-install