Phase 2 of the OCW Common Cartridge Installer Project

Jan. 7, 2009

The second phase of the OCW-CC app is designed to accomplish the following goals:

1. Demonstrate application portability by moving the existing app to an apache environment.
2. Create a second implementation of the Filing OSID against a server file system.
3. Finish work done on the Sakai version of the project - especially in handling the CC manifest.

Ideally, I'd like to be able to change the build/deployment of the existing OCW-CC app such that it runs in a pure Tomcat container with no dependencies on Sakai. I know that RSF supports targeting a pure Tomcat environment. It should be a matter of adjusting the maven file and creating a new version.

The current cc-install application ran against Sakai 2.4.x, which implies that it doesn't even have maven 2 projects.

It might be good to start with a hello-world experiment. I will attempt to build a copy of RsfExample-1 (hello world) such that it runs purely from Tomcat.
Copied RsfExamples/Ex1-Content to RsfTomcat/Ex1-Content and created a POM file for it. Seems to build without error, but this is using the Sakai RSF dependencies.

used mvn insall rsf:deploy, but that doesn't create an invocable application.

Jan. 8, 2009

Had a look at RsfHelloWorld located at https://saffron.caret.cam.ac.uk/svn/projects/RSFHelloWorld. Both the maven POM and the web.xml file are simplified. I adjusted my files and rebuilt it using this command:

mvn insall rsf:deploy

Started up tomcat. Didn't seem to work for the following URL

http://localhost:8080/RsfEx1/

working from the RSF example is a good idea, but rather than trashing around trying to figure out how my Sakai examples are different, I'm going to start with a clean Tomcat environment (5.5.27) and download fresh copies of RSF and it's examples. This done on my laptop environment.

Downloaded RSF 0.7.4 from CARET. Downloaded RSFHelloWorld example from CARET.
Built it using maven and the rsf:deploy plugin.

Successfully ran the application using the following URL

http://localhost:8080/RSFWorld/faces/

Created a development sandbox for the Native File system implementation of the Filing OSID (etc) in cc-ocw.
Initial version tested workg against the following URL:

http://localhost:8080/cc-ocw/faces/

The cloned application needs to be cleaned up a bit in terms of package naming, etc, but I'm more interested in installing the v3 OSID JAR and seeing if I can access it in Maven and deploy it into the shared directory of Tomcat.

Added the OSID jar to the .m2 local repository in .m2/repository/org/okiproject/osid/3.0.03d/osid-3.0.03d.jar
I had to rename the jar from OSID-3.0.03d.jar to osid-3.0.03d.jar because I believe Maven is case sensative and must agree with the repository packaging scheme.

I extracted the API sources out of the jar and note that the path is "org.osid", so the maven repository path should be? org/osid/osid/3.0.0d3/osid-3.0.03d.jav.
This is an easy fix. The maven dependency (for maven 2.0) looks like this:

<dependency>
    <groupId>org.osid</groupId>
    <artifactId>osid</artifactId>
    <version>3.0.0d3</version>
    </dependency>

Maven is having problems finding the artifact installed in my local repository. I tried the install:install-file command suggested, but that command doesn't seem to be supported by maven (ironically).

After much playing around, I managed to get the install:install-file plugin to work. The command is:

mvn install:install-file -DgroupId:org.osid -DartifactId=osid -Dversion=3.0.0d3 -Dpackaging=jar -Dfile=C:\osid-3.0.0d3.jar -DgeneratePom=true

Two problems were present. The command didn't like a file name that included spaces ("Documents and Settings") and the generatePom option only works with install-file (not install). A Pom file is needed, I guess.

Got it to build, but is not displaying the right hello message. Shows the old one (Hello Reasonable World) rather than the new one (CC OCW Installer).

Jan. 9, 2009

Got the new app to compile and run. There is a problem with either maven or tomcat when "mvn clean install rsf:deploy" is used. The expanded version of the war app isn't removed and is used in spite of a new war being written to TOMCAT/webapps.

Spoke to Antranig who recommended using Tomcat 6.x. That's a simple thing to try.

Downloaded Tomcat 6.0.18 and set it up for use.

Jan. 12, 2009

Got files organized to create a new osid implementation. Created a sub-project for the impl in the cc-ocw project. This can be broken out later (and should be).

Jan. 15, 2009

Created an Eclipse project for OSID development. Implementation code is in "c:/newdev/oki/osid-v3-impl".

Having trouble with the OSID definitions. Got a project started in Eclipse to work on Filing OSID implementation using the JAR provided in the javakit. However, the interfaces do not match the sources I have. I need the sources.

Successfully downloaded the specification PDFs, which helps some.
Created an OsidFilingTest class that can be used in Eclipse to test Osids being created.
Successfully instantiated a stubbed FilingManager.

I am creating the filing system implementation with the prefix "fs". This follows from the "chs" prefix previously used.

There are several "sessions" supported by the Filing Manager:

• DirectoryEntryLookup
• DirectoryAdmin
• DirectorySearch

Although I think that most of the same things can be done in Search as in Entry Lookup, the entry lookup seems to be the way to go for browsing directories. I have created fsDirectoryEntryLookup based on java.io.File.
Started work on File. Partially implemented File, FileList, Directory, and DirectoryEntry. Partially implemented File, FileList, Directory, and DirectoryEntry.

Jan. 17, 2009

A complete implementation of DirectoryEntry calls for Creation and Modification times. Modification times are available from java.io.File, but create time and users are not available. Implemented org.osid.DateTime against java.util.Date.

Most of the basic capabilities of DirectoryLookupSession are implemented at this point.
Added test to OsidFilingTest to show entries in a directory. this works with the following result:

Jan. 17, 2009

A complete implementation of DirectoryEntry calls for Creation and Modification times. Modification times are available from java.io.File, but create time and users are not available. Implemented org.osid.DateTime against java.util.Date.

Most of the basic capabilities of DirectoryLookupSession are implemented at this point.
Added test to OsidFilingTest to show entries in a directory. this works with the following result:

Filing Manager created.
Directory Entry Lookup Session created for: c:/
The path c:/has the following entries:
	240.txt is a file
	7fecc82ef4bdcc86c4c1913b472f0774 is a directory
	asmruntime.log is a file
	AUTOEXEC.BAT is a file
	autoruns is a directory
	axis is a directory
	axis-1-4.zip is a file
	axis-1_4 is a directory
	axis.txt is a file
	boot.ini is a file
	CONFIG.SYS is a file
	cvs-maven.txt is a file
	cygwin is a directory
	dell is a directory
	dell.sdr is a file
	depit.bat is a file
	Documents and Settings is a directory
	drivers is a directory
	eclipse is a directory
	ex1.txt is a file
	Firefox Setup 1.5.0.9.exe is a file
	FlyingColors is a directory
	ga_root.txt is a file
	gtcaroot.pkcs10 is a file
	hello.txt is a file
	hiberfil.sys is a file
	i386 is a directory
	(etc)

On examination of the osid.Filing interfaces, there is no overt way to read and write a file. As near as I can tell, one must get a FileForm based on a FileType that somehow specifies an object class to use and interface methods. Several documents refer to "getting" an interface specification based on a Type, but I don't know how to do this. While I can appreciate trying to be consistent about design patterns, it's seems a bit silly to be unable to read or write a file.

The V3-Interface-Pattern diagram indicates that OsidObject can be extended to have an interface that examines an object.

I suppose file data could be a property of a File.

This is from osid.Record:

A record is an auxiliary interface that can be retrieved from an OSID object, query, form or search order that contains method definitions outside the core OSID specification. An OSID record interface specification is identified with a Type.

The question is HOW is an auxiliary interface specified? There was some mention of this in the type specification, I think.

Jan. 18, 2009

Lying awake in bed last night, I was thinking about how to get an interface specification (what does one think about in the dead of night but code?). The specification must be in a Metadata object, since that is how the OSIDs define interfaces! osid.OsidCatalogForm has a method to getMetadata().

The osid-javakit contains implementations of the commonly used OSIDs, including metdata, etc. I don't I have ever built the javakit, which requires ANT. Need to download ANT and use it to build the kit.

Jan. 19, 2009

Installed ant (http://ant.apache.org). Build the javakit by cd'ing to osid-javakit/package/build and run the following commands in a Cygwin shell:

ant          // Compiles basic code.
ant jar      // Builds a JAR of the javakit.
ant javadoc  // Creates a javadoc set for the javakit.

There are 134 implementation classes in the javakit. It seems to a pretty functional reference implementation of the core functions. While there is javadoc, it's going to be a real research project understanding it. Still, it's a good way to get into Tom's head w.r.t design patterns and how things are intended to be used.

Added the javakit jar to the test environment I'm building. Successfully loaded the boot, RuntimeManager, and my Filing Manager instance.
I've pretty much come to the conclusion that down-casting cannot be avoided. Given the existence of a type, an application can fetch and object, test it's type, and downcast it to a more specific version of the object. This doesn't eliminate apriori knowledge of an implementation, even if the provider has interfaces defined for extended objects.

Jan. 21, 2009

Here is an interesting thing. In the document "Structure and Definitions of OSIDs", reference is made to an XML schema that defines a way to specify interfaces. In theory, I could use this schema to create extensions of the basic OSID interfaces. In practice, who would bother? Yeah, it keeps things language independent but at a considerable price, IMO. If this were purely a Java binding, we could use introspection, Java beans, or even Java peas to handle most extensions.

I've decided not to worry about how to publish the extended interface an get on with the design. Tom didn't respond to my message asking how to access extended interfaces and I can't wait forever. Some of the new design is being documented in File Types.

Caissons also figure into the definition of extended objects. In "Introduction to OSID v3 for v2 Developers", a createAssess() method is present in AssetAdminSession that takes an Id and an AssetCaisson as arguments. The caisson, in turn, allows values to be collected and updated to an Asset object. I think the Caisson is a wrapper for an object providing access to implementation constructors and protected methods. In Section 5 on types, an example is provided of an Asset with a getType() and getExtentionInterface() methods. In theory, the getExtentionInterface provides access to the information needed to access additional methods on the object (Asset) in this case.

Here is an example provided in this file:

org.osid.type.Type mp3Type = org.osid.util.MakeType("guid", "93489020404234", "mit.edu");

org.osid.repository.AssetCaisson caisson = session.getAssetCaissonForUpdate(assetId);
edu.mit.mp3Caisson = (edu.mit.mp3Caisson) caisson.getCaissonExtension();
org.osid.Metadata metadata = caisson.getTrackMetadata();

if (metadata.isReadOnly()) {
    throw new Exception("cannot update track, you might not be authorized");
    }

// the type spec says the track is a cardinal but it binds to an integer
int[] choices = metadata.getCardinalSet();
boolean good = false;
for (int i: choices) {
    if (track == choices) {
        good = true;
    }
}
if (!good) {
    throw new Exception("track is invalid");
}
try {
    mp3Caisson.setTrack(track);
} catch org.osid.InvalidArgumentException (iae) {
    throw new Exception("track is invalid... the caisson lied", iae);
}

mp3Caisson.update() is then invoked a bit later ...

A couple of things to note about this.

• The implementation surfaces into the consuming application.
• Caissons are used for creation and updating extended objects.
• We might presume that if an object is of an extended type, we can cast it to the implementation for data access purposes.

Looking back at the Filing OSID, File has a method called getFileRecord(Type type). FileRecord states "The methods specified by the record type are available through the underlying object." This suggests, for purposes of accessing data in an object, that an OsidRecord is used instead of a Caisson and that OsidRecords can be down-cast based on type. So, if I have a File that I suspect of being a TextFile, I can ask for the FileRecord of type "TextFile". If successful, then the resulting object can be cased to com.nolaria.osid.filing.TextFileRecord which encapsulates an fsTextFile object. Logical, but who knows if this is right?

Interestingly, the actual OSID definintion for File has a method of:

public File getFileRecord (Type arg0);

It looks like the encapsulation of a File into a FileRecord was lost.
I send Tom a message asking about this.

Meanwhile, it seems that I need to figure out how to create File types.
Created org.osid.filing.type.FileType with nolaria.com as an authority. Added DirectoryType and TextFileType. This last one will really be the test of using Types to created extended interfaces.

As in previous implementations of the OSIDs (including v2), there is is a global need for access to various managers. For example, the FilingManager needs to register these new File types with the TypeManager. Simple enough in the initialize() method of the FilingManager, since the Runtime is passed in. However, the TypeManager won't be generally available later, nor the Runtime. So, rather than pass down access to the Runtime all over the frigging place, I'll create a nice Static object called fsManagerCache that will provide a way to register managers and provide global access to all objects within the scope of the fsFiling implementation. It's a bit of a hack, but there is plenty of code to write without worrying about carefully passing access to managers down through many constructors.

Having created a nice spot to cache OSID managers, I looked through the javakit to see what managers I might use. Well, there aren't many, first off. Type is the obvious one I need right now, but guess what? TypeAdminSession is not implemented. It only registers static package types. Effectively useless for my needs. Also, the IdManager won't allocate new Ids. It relies on static ids passed in.

Started work on TypeManager. That led to an implementation of Id that generates random UUIDs. Frankly, I may as well implement Type, since it is pretty much an extension of Id anyways.

Some thought needs to go into where managers get cached. It could be done in FilingManager, since that is the whole thrust of this implementation package. Or it could be pushed up into the application space, which is not horrible considering that the boot and runtime are started up there. I think that's the more general solution.

Jan. 22, 2009

I wrote to Tom:

Tom replied:

The standard approach is to do:

if (file.implementsRecord(zipFileRecordType)) {
      ZipFileRecord zip = (ZipFileRecord) file.getFileRecord(zipFileRecordType);

      int count = zip.getFileCount();
}

Another approach, considering that the zip file represents a "directory" of files, is to use the 
Directory interface. This can avoid having to rely on a record type for interoperability.

DirectoryEntryLookupSession session = manager.getDirectoryLookupSession(zipFilePath);
FileList files = session.getFiles();  /* gets each file in the zip file */
DirectoryList dirs = session.getSubdirectories(); /* gets each directory in the zip file */

The default getDirectoryLookupSession() with no arguments can return a list of available zip files to choose from.

So this indicates that forms are used for updates and records used for access. File.implementsRecord() is not part of any OSID definition (that I can find, I checked File, DirectoryEntry, and OsidObject). It looks like this method needs to be added to the implementation. It's pretty key to adding and using extension, so how it be used with out being exposed to the consuming applications? Presumably, one can add enough functionality to the extensions to obviate it's use in the application, though it's hard to see how that it is possible.

Ah, here is a bit of insight. Tom points out that a FileSession gives read and write access to Files. Whew! That's a relief. The pattern of putting such things into sessions is counter-intuitive after many, many years of associating operations with the objects themselves. To some extent, this breaks with object oriented programming, which Tom does note in one of the OSID background documents. I guess it comes down to which is more important: OOP or interoperability?

It may be sufficient to define different file types and allow the consuming application to interpret them. Thus, a TextFile could be read using a text reader stream.

Concerning the manager cache, I think I'll set it up such that each manager checks to see if OsidRuntime is initialized in the cache. If not, it is added and it registers itself. If already registered, it does nothing.
Three file types are now registered: FileType, DirectoryType, and TextFileType.

Completed work on fsTypeLookupSession that will allow types to be found.

Wrote some test code to see if Types are getting registered. Well, they are and they aren't. In FilingManager.initialize(), I get the TypeManager and register three types. Later, in the OsidFilingTest application, I call the OsidRuntime again to get the TypeManager. I get a NEW one. In other words, the runtime doesn't maintain singletons.

I could make the managers (Filing and Type) be stateless. That would involve moving the Type registry out of TypeManager and into a static service. I could also modify the implementation of OsidRuntime to create managers and retain them as singletons. If so, then OsidRuntime starts to take on the purpose of fsManagerCache.

Extracted register code out of TypeManager into it's own static class called fsTypeRegistry. This actually simplified code all over the place. The TypeManager is no longer needed in the FilingManager to register types. The TypeManager doesn't have to be passed down into fsTypeLookupSession. Output of OsidFilingTest is now:

With this in place, work can proceed on the Filing sessions. Specifically: FileSession, DirectoryLookupSession, and DirectoryAdminSession.

Jan. 23, 2009

Resumed work on DirectoryLookupSession. Marked many methods as Unimplemented. getFilesByRecordType() will require associating file types with file name extensions, and perhaps MIME types. I added mime and ext to FileType. It's implemented as simple strings, which is actually insufficient since a TextFileType could represent more than one MIME type and handle more than one extension (txt, text, etc.).

In implementing FilingDirectoryLookupSession.getFilesByRecordType(), I need to determine the FileType associated with a File. I have mime types and extensions initialized in the types now, but need to lookup a types on that basis. I was able to implement this using only the fsTypeRegistry.

Created a polymorphic method File.getFileRecord(fsFileType type) to override the File.GetFileRecord(Type type) that returns a File instead of a FileRecord.

All of the File methods in FilingDirectoryLookupSession are now implemented (or throw Unimplemented). All that remain are the DirectoryList methods. DirectoryList needs to be implemented before that can happen.
Finished methods in FilingDirectoryLookupSession. One or two additional OSID typo were noted and submitted as tickets on Assembla.

Created fsFilingSession. Most of these methods have to do with reading and writing files via a Transport. This is the meat of the Filing OSID.
Unfortunately, there is no way (that I can see) to associate a File with a FileSession, making it VERY difficult to implement.

Jan. 24, 2009

Tom pointed out that there is a FilingManager.getFileSessionForPath(String path) method. This allows me to start implementing FileSession. Much of it was quickly done, but I'll need to implement the whole Transport OSID set staring with the Transport manager. All of this was missing last year when I attempted the first implementation of the v3 OSIDs and the CC-OCW installer application.

Jan. 26, 2009

Started work on the Transport OSIDs.
Implemented most of the Transport OSIDs. TransportSession is implemented, which supports both input and output file streams. TransportReciver is not implemented, but not really needed, either.

Shifted initialization of the manager cache into a method, fsManagerCache.initialize(OsidRuntimeManager runtime);
This simplifies initialization of the cache and centralizes instantiation of the known managers.
Completed FileSession methods. Can now read and write to files.

With the completion of FileSession, I am ready to start writing application code that tests the fsOsid implementations. The first test will be to open and display the contents of a text file using file types.

Wrote fsFileRecord. This is the basis of fsTextFileRecord which can provide capabilities above and beyond basic file records.

Jan. 27, 2009

Implemented fsTextFileRecord.

Sent off the following message to Tom Coppeto:

Implemented fsTextFileRecord. Since the intent is to get the contents of the file as a string, I extracted the java.io.File from the osid File and read it directly. The getText() method throws NotFoundException and OperationFailedException (if an I/O error occurs).
Deleted fsTextFile, as it doesn't do anything.

Re-arranged OsidFilingTest to push the various tests into their own methods.
Created OsidFilingTest.textFileTest.
One critical piece is missing to make this work, an implementation of fsFile.getFileRecord(); Added a method fsTypeRegistry.getTypesByDomain() to return a filtered collection.
Implemented getFileRecord(). It only supports fsTextFileRecord. I could have had it return fsFileRecord, but there really isn't any point since FileRecords don't do anything.

After some debugging, was able to get TextFileRecord.getText() to return the contents of c:/test.txt.

To do that, I had to bring several objects from the implementation into my application:

import com.nolaria.osid.fsManagerCache;
import com.nolaria.osid.filing.fsFile;
import com.nolaria.osid.filing.fsFileRecord;
import com.nolaria.osid.filing.fsTextFileRecord;
import com.nolaria.osid.type.fsType;
import com.nolaria.osid.type.fsTypeRegistry;
import com.nolaria.osid.filing.type.fsFileType;
import com.nolaria.osid.filing.type.fsTextFileType;

The fsManagerCache currently needs to be initialized at the app level, but can later be pushed done once I have an implementation of OsidRunTimeManager. Access to the type registery might be avoided by using the TypeManager and a TypeLookupSession. fsType is there because the type registry needs it. fsFileType and fsTextFileType are accessed for constants.

It seems like a useful exercise to see if I can eliminate as many of these dependencies as I can, though there will always be an out of band agreement on expected type definitions and extensions to OsidRecord.

Some of these dependencies could be eliminated, but there are some problems. One of the biggest is that File.getFileRecord(type) returns a File instead of a FileRecord. Create another method in fsFile called getFRecord() that takes an osid.type.Type. Evenually these will all collapse down into a single getFileRecord() that takes an OSID type and returns a FileRecord.

After changing the OsidFilingTest application to use a TypeLookupSession instead of accessing the fsTypeRegistry directly, I am left with the following implementation imports:

import com.nolaria.osid.fsManagerCache;
import com.nolaria.osid.filing.fsFile;
import com.nolaria.osid.filing.fsTextFileRecord;

fsManagerCache must be initialized in the application at this time.
fsFile is needed to invoke the getFRecord() method. It will go away later once the OSID definitions are fixed.
fsTextFileRecord is needed if we are to invoke extension methods beyond those provided by FileRecord.

Not too bad. However, there is a line of code in this re-written test that embodies an out of band agreement with the implementation:

//  Get the fsTextFileType.  Knowing  these strings is a priori knowledge of the implementation!
Type type = tls.getType("nolaria.com", "org.osid.filing.file", "fsTextFileType");

Again, this cannot be helped if the application is to be able to work with text files.

I think I have enough implemented to start something more interesting: support for ZIP files, followed by support for content packages leading to common cartridges. Tom suggested that I treat ZIP files as a kind of directory. I like this for many reasons, most of which is that processing it will fall completely into the implementation. The application will only see a FileList.

To implement this, we will need a new, specialized directory type for archive files. Let's call it fsArchiveType and make it a sub-class of fsDirectory.
The fsArchiveType file is implemented and shows up in the list of registered types.

Update the fileTest() to use a test directory with the following output:

The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file
	no-1.jpg is a file
	subfolder is a directory
	test.txt is a file

The thrust of this next effort is to tag each entry in this directory with it's proper type, including the zip file.

Jan. 28, 2009

Implemented all four of the type methods in DirectoryEntry. I have decided that, in general, genus types can be used to identify file and directory types.

Changed the fileTest() to use getGenusType with the following result:

The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type file
	no-1.jpg is a file of type file
	subfolder is a directory of type directory
	test.txt is a file of type text-file

This is a pretty good interim result. Subdirectories are now of type directory. It correctly typed the single text file. the zip and jpg file are unknown file types, so it defaults to type file. Now we'll add code to handle the special case of an archive.

Changed fsArchiveType to extend fsFileType and added a MIME type (application/zip) and extension string (zip). Zip files now show up correctly as archives in the result:

The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type file
	subfolder is a directory of type directory
	test.txt is a file of type text-file

There are two aspects to handling ZIP files: inspecting them (how many files, getting at files inside, etc), and inflating (decompressing). Inspection is related to a DirectoryLookupSession (IMO), while inflating is a DirectoryAdminSession operation. In both cases, normal operations are extended or overridden.

Jan. 28, 2009

Implemented all four of the type methods in DirectoryEntry. I have decided that, in general, genus types can be used to identify file and directory types.

Changed the fileTest() to use getGenusType with the following result:

The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type file
	no-1.jpg is a file of type file
	subfolder is a directory of type directory
	test.txt is a file of type text-file

This is a pretty good interim result. Subdirectories are now of type directory. It correctly typed the single text file. the zip and jpg file are unknown file types, so it defaults to type file. Now we'll add code to handle the special case of an archive.

Changed fsArchiveType to extend fsFileType and added a MIME type (application/zip) and extension string (zip). Zip files now show up correctly as archives in the result:

The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type file
	subfolder is a directory of type directory
	test.txt is a file of type text-file

There are two (?) aspects to handling ZIP files: inspecting them (how many files, getting at files inside, etc), and inflating (decompressing). Inspection is related to a DirectoryLookupSession (IMO), while inflating is a DirectoryAdminSession operation. In both cases, normal operations are extended or overridden.

Added two methods isArchive() and isArchive(File file) to DirectoryEntryLookupSession, based on file type. These can be used to determine if an archive case is present.

There are a lot of methods that refer to genus types in DirectoryEntryLookupSession. Genus types tell what an entry IS. Record types tell what interfaces exist. Presumably, record types can be used as form types as well. Genus types have a hierarchical relationship, so it may be necessary for me to blend in Hierarchy to the file and directory types. The Hierarchy interface doesn't provide parent/child access. This is all done in HeirarchyTraverseSession, etc. Sigh. Nothing about this is simple. Access to files by genus parents will have to wait.

Added code to test to open up the Cultural_Antropology.zip file as an archive. The DirectoryEntryLookupSession.getFiles() method currently checks for the path as an archive and returns an empty list. Test works (empty list).

Created fsArchiveFile, which extends fsFile currently implemented to be a ZIP file. Now the question arises. How shall I represent files and directories inside of an archive? They need to behave like OSID files, but also need to be relative to the ZIP file that contains them. Perhaps I should rename it to fsArchive to represent the containing structure, and then extend it to have fsArchiveFile and fsArchiveDirectory? This will lead to a weird inheritance tree:

This might not be too bad because the types associated with them would have a different hierarchy:

Having an fsArchiveFile class allows me to encapsulate a ZipEntry and keep access to the ZipFile that it is part of. I think that might be very useful. Similarly with fsArchiveDirectory.

Jan. 29, 2009

On further thought, I don't think that fsArchiveFile should extend fsArchive. Rather, there should be an fsArchiveEntry, fsArchiveFile, and fsArchiveDirectory that implement each of the corresponding OSID interfaces. In this way, ArchiveFiles can be treated like Files and ArchiveDirectories like Directories.

Implemented fsArchiveEntry, fsArchiveFile, and fsArchiveDirectory.
Broke out lookup into an archive as a separate test in OsidFilingTest.
I need to add code to DirectoryEntryLookupSession to detect an attempt to open an archive as a directory.
A bit of code was added to DELS.getFiles() when the directory is an archive. Ran test with following result:

The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Chapter_9__The_Life_Cycle.htm is a unknown of type (no label provided)
	face.jpg is a unknown of type (no label provided)
	Chapter_1__A_Definition.htm is a unknown of type (no label provided)
	About_the_Professor.htm is a unknown of type (no label provided)
	Chapter_6__Religion_and_Culture.htm is a unknown of type (no label provided)
	Chapter_13__Environment__Adaption__and_Subsistence.htm is a unknown of type (no label provided)
	Prof_photo__Richley_Crapo.jpg is a unknown of type (no label provided)










	Chapter_14__Cultural_Evolution.htm is a unknown of type (no label provided)
	Chapter_8__Social_Organization_and_Kinship.htm is a unknown of type (no label provided)
	Chapter_12__The_Economic_System.htm is a unknown of type (no label provided)
	Chapter_2__Culture.htm is a unknown of type (no label provided)
	Chapter_5__Language_and_Culture.htm is a unknown of type (no label provided)
	Chapter_10__Marriage__Family__and_Household.htm is a unknown of type (no label provided)
	Schedule.htm is a unknown of type (no label provided)
	Chapter_11__Politics_and_Culture.htm is a unknown of type (no label provided)
	Syllabus.htm is a unknown of type (no label provided)
	Chapter_7__Aesthetics_and_Culture.htm is a unknown of type (no label provided)
	Chapter_4__Culture_and__Personality.htm is a unknown of type (no label provided)
	Chapter_3__Biology_and_Culture.htm is a unknown of type (no label provided)
	Cultural_Anthropology.html is a unknown of type (no label provided)
	Cultural_Anthropology.zip is a unknown of type (no label provided)
	imsmanifest.xml is a unknown of type (no label provided)

This is actually pretty good. Most of the types in the archive not registered, though I might add them just for kicks. It's not clear why "Cultural_Anthropology.zip" showed up as an entry. I don't think it is an embedded archive.

Types to add: xml, htm, and jpg.

It's also not clear how to get directory structure out of a ZipEntry. There is no method to get relative (etc) paths.

Added new types. They appear in typeTest(), but not when I run archiveTest(). I suspect that the wrong kind of class is getting created. Confirmed. Change fsArchive.getFiles() to create fsArchiveFiles instead of fsFiles. Results are now:

The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Cultural_Anthropology/Cultural_Anthropology/Chapter_9__The_Life_Cycle.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/face.jpg is a unknown of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_1__A_Definition.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/About_the_Professor.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_6__Religion_and_Culture.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_13__Environment__Adaption__and_Subsistence.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Prof_photo__Richley_Crapo.jpg is a unknown of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_14__Cultural_Evolution.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_8__Social_Organization_and_Kinship.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_12__The_Economic_System.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_2__Culture.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_5__Language_and_Culture.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_10__Marriage__Family__and_Household.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Schedule.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_11__Politics_and_Culture.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Syllabus.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_7__Aesthetics_and_Culture.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_4__Culture_and__Personality.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_3__Biology_and_Culture.htm is a unknown of type html-file
	Cultural_Anthropology/Cultural_Anthropology.html is a unknown of type file
	Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip is a unknown of type archive
	Cultural_Anthropology/imsmanifest.xml is a unknown of type xml-file

Types are now coming through, but the basic test for file/directory is failing. This is in part due to the pattern of putting isFile() and isDirectory() in the DirectoryEntryLookupSession, instead of DirectoryEntry, where I think is more logical. The methods in DELS take a path. Should this path be relative to the one defined for the DELS? In fact, it takes a name, not a path at all.

Problem with file/directory test was a copy/paste problem, corrected in DELS.isFile(name). Results are now:

The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Cultural_Anthropology/Cultural_Anthropology/Chapter_9__The_Life_Cycle.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/face.jpg is a file of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_1__A_Definition.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/About_the_Professor.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_6__Religion_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_13__Environment__Adaption__and_Subsistence.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Prof_photo__Richley_Crapo.jpg is a file of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_14__Cultural_Evolution.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_8__Social_Organization_and_Kinship.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_12__The_Economic_System.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_2__Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_5__Language_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_10__Marriage__Family__and_Household.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Schedule.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_11__Politics_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Syllabus.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_7__Aesthetics_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_4__Culture_and__Personality.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_3__Biology_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology.html is a file of type file
	Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip is a file of type archive
	Cultural_Anthropology/imsmanifest.xml is a file of type xml-file

Silly me. The archive directory structure is included in getName(), shown in results above.
It would seem that java.util.zip implements an archive using a federated directory view, rather than an isolated directory view. I have implemented DELS using the isolated directory view, currently. Both views are useful, of course. Implementing federated views on the generic file system is pretty simple, though it does involve directory recursion. To get an isolated view out of the entries returned by ZipFile.entries() (as in the example above), I'd have to parse out the directory structure and probably cache it somehow.

Depending on the directory view mode, I could create a file tree structure when the DELS is created. I think I'll poke at that a bit...

Jan. 30, 2009

Implemented DirectoryEntryLookupSession.buildArchiveStructure() using fsArchiveNodes. It builds a simple tree of names.
Implemented DirectoryEntryLookupSession.printArchiveStructure() to print the structure created by buildArchiveStructure().
Added test to OsidFilingTest.archiveTest().
After some debugging, the following results were obtained:

/
	Cultural_Anthropology
		Cultural_Anthropology
			Chapter_9__The_Life_Cycle.htm

			face.jpg
			Chapter_1__A_Definition.htm
			About_the_Professor.htm
			Chapter_6__Religion_and_Culture.htm
			Chapter_13__Environment__Adaption__and_Subsistence.htm
			Prof_photo__Richley_Crapo.jpg
			Chapter_14__Cultural_Evolution.htm
			Chapter_8__Social_Organization_and_Kinship.htm
			Chapter_12__The_Economic_System.htm
			Chapter_2__Culture.htm
			Chapter_5__Language_and_Culture.htm
			Chapter_10__Marriage__Family__and_Household.htm
			Schedule.htm
			Chapter_11__Politics_and_Culture.htm
			Syllabus.htm
			Chapter_7__Aesthetics_and_Culture.htm
			Chapter_4__Culture_and__Personality.htm
			Chapter_3__Biology_and_Culture.htm
			Cultural_Anthropology.zip
		Cultural_Anthropology.html
		imsmanifest.xml

This gives me the basis for working with an archive as a file tree structure. I can now implement DELS in the various views for both a generic file system and an archive file system.

Implemented DirectoryEntryLookupSession.close() and added checks to other methods to see if it was closed.

Feb. 5, 2009

Two initial views are envisioned for the Filing OSID implementations: ISOLATED and FEDERATED The isolate view show files/directories in the current directory only. The federated view will show files and directories for all nested directories.

There are challenges presented by archive files. A the path for DELS typically points to a directory. If it points to an archive, we can handle that. It is when it points to a directory INSIDE of an archive that we have a problem, because path names inside the directory are relative to the archive itself as a root. Thus the path, "/foo/bar/archive.zip/this/that" is actually two paths: "/foo/bar/archive.zip" that points to the archive, and "/this/that" which points to a directory in the archive.

I believe I have the structures in place to deal with this. It should be a matter of detecting this case when the DELS is created and setting up the appropriate archive structures in it to deal with the sub-path.

DELS.getFilesFederated() calls DELS.buildArchiveStructure() which calls DELS.getFiles(). Infinite loops are bad.
Replaced the call to DELS.getFiles() with the same code that builds a list of files in the archive.

Looking at this, I think that the archive structure should be associated with the fsArchive object, instead of the session. I'm going to shift the archive structure down into fsArchive. Shifted buildArchiveStructure() into fsArchive completely and made it private. Shifted printArchiveStructure() also, but left a simple access method in DELS.

This work led to adding fsArchive.getFilesAt(path) and fsArchive.getDirectoriesAt(path). In the latter, creating an fsArchiveDirectory, the super class, fsDirectoryEntry wants a ZipEntry. Well, that can't be, because only Zip files have ZipEntry's. Directory nodes are implied by elements of the path names in the Zip file names. The basic stuff of directory entries (path, name, etc.) could be handled by pushing ZipEntry down into fsArchiveFile. However, a lot of type recognition is done in fsArchiveEntry. Most of that is for file types, since there is really only one kind of directory type in an archive - a simple one (ignores the case of nested archives).

I have attempted to solve the latter problem by creating fake ZipEntry's for directory nodes. Certain things are likely to fail (like entry.getSize(), but the basis stuff is likely to work. Much testing is needed.

fsArchive.getFilesAt(path) won't work by see if ZipEntry.getName() has a prefix of path. This would return all files below that directory node, which is a federated view. We are after the isolated case.

Feb. 6, 2009

The main thrust of development at this point is to test two different views against two different directory situations:

• File System Directory
  • Isolated
  • Federated
• Archive File Directory
  • File System Directory
  • Archive File Directory

Let's drop back and get the simple case to work: show the files in a file system directory.

Feb. 6, 2009

The main thrust of development at this point is to test two different views against two different directory situations:

• File System Directory
  • Isolated
  • Federated
• Archive File Directory
  • File System Directory
  • Archive File Directory

Let's drop back and get the simple case to work: show the files in a file system directory.

Running OsidFilingTest.fileTest(), we get:

The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type jpeg-file
	subfolder is a directory of type directory
	test.txt is a file of type text-file

First off, I don't think we want to include directories in the file list, so let's eliminate that. Created an inner class on DirectoryEntryLookupSession that implements java.io.FileFilter and uses it to filter out only files. Results are now:

Using isolated directory view.
The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type jpeg-file
	test.txt is a file of type text-file

Note that "Cultural_Anthropology.zip" is included in this output. Archive files are somewhat special. At times, we want to treat them like a file and at other times a directory. I've decided to include them in lists of while files are requested and treat them like a directory when pointed to specifically.

Now we add the federated view:

Using isolated directory view,
The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type jpeg-file
	test.txt is a file of type text-file

Using federated directory view,
The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type jpeg-file
	subfolder is a directory of type directory
	test.txt is a file of type text-file

This is expected at this point. DELS.getFilesFederated() hasn't been updated to get federated files yet.
Re-wrote the generic file system portion of DELS.getFilesFederated(). New test results are:

Using isolated directory view,
The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type jpeg-file
	test.txt is a file of type text-file

Using federated directory view,
The path C:/osid-test has the following entries:
	Cultural_Anthropology.zip is a file of type archive
	no-1.jpg is a file of type jpeg-file
	Sakaiger.AVI is a unknown of type file
	test.txt is a file of type text-file

This is correct, as far as it goes. There is a question to adding the path to the name of more deeply nested files, but I don't think it's required by the specification. There does seem to be a bit of work needed with it's type, but that's less important than getting a federated view.

Testing these two views against an archive requires two cases: a path name to the archive root and one to a directory in the archive. The path to the root in our test case is "C:/osid-test/Cultural_Anthropology.zip". The embedded directory we'll use is "C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology". Let's see what the code gives us now:

Using isolated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Cultural_Anthropology/Cultural_Anthropology/Chapter_9__The_Life_Cycle.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/face.jpg is a file of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_1__A_Definition.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/About_the_Professor.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_6__Religion_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_13__Environment__Adaption__and_Subsistence.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Prof_photo__Richley_Crapo.jpg is a file of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_14__Cultural_Evolution.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_8__Social_Organization_and_Kinship.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_12__The_Economic_System.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_2__Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_5__Language_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_10__Marriage__Family__and_Household.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Schedule.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_11__Politics_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Syllabus.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_7__Aesthetics_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_4__Culture_and__Personality.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_3__Biology_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology.html is a file of type file
	Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip is a file of type archive
	Cultural_Anthropology/imsmanifest.xml is a file of type xml-file

Using federated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Cultural_Anthropology/Cultural_Anthropology/Chapter_9__The_Life_Cycle.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/face.jpg is a file of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_1__A_Definition.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/About_the_Professor.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_6__Religion_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_13__Environment__Adaption__and_Subsistence.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Prof_photo__Richley_Crapo.jpg is a file of type jpeg-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_14__Cultural_Evolution.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_8__Social_Organization_and_Kinship.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_12__The_Economic_System.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_2__Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_5__Language_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_10__Marriage__Family__and_Household.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Schedule.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_11__Politics_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Syllabus.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_7__Aesthetics_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_4__Culture_and__Personality.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology/Chapter_3__Biology_and_Culture.htm is a file of type html-file
	Cultural_Anthropology/Cultural_Anthropology.html is a file of type file
	Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip is a file of type archive
	Cultural_Anthropology/imsmanifest.xml is a file of type xml-file

Now the isolated view is incorrect, while the federated view is correct, though the names include directory paths.
Tried to trim names of path, but that causes exceptions to arise. If I change getName() to return a simple name, then DELS.isDirectory() fails, since it no longer has the full name of the file to test against. Problem appears in fsArchive.getFile(name). If name is a truncated ZipEntry name, it will not be found. Thus, it tests as neither a file or a directory. However, results look better:

Using isolated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Chapter_9__The_Life_Cycle.htm is a unknown of type html-file
	face.jpg is a unknown of type jpeg-file
	Chapter_1__A_Definition.htm is a unknown of type html-file
	About_the_Professor.htm is a unknown of type html-file
	Chapter_6__Religion_and_Culture.htm is a unknown of type html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm is a unknown of type html-file
	Prof_photo__Richley_Crapo.jpg is a unknown of type jpeg-file
	Chapter_14__Cultural_Evolution.htm is a unknown of type html-file
	Chapter_8__Social_Organization_and_Kinship.htm is a unknown of type html-file
	Chapter_12__The_Economic_System.htm is a unknown of type html-file
	Chapter_2__Culture.htm is a unknown of type html-file
	Chapter_5__Language_and_Culture.htm is a unknown of type html-file
	Chapter_10__Marriage__Family__and_Household.htm is a unknown of type html-file
	Schedule.htm is a unknown of type html-file
	Chapter_11__Politics_and_Culture.htm is a unknown of type html-file
	Syllabus.htm is a unknown of type html-file
	Chapter_7__Aesthetics_and_Culture.htm is a unknown of type html-file
	Chapter_4__Culture_and__Personality.htm is a unknown of type html-file
	Chapter_3__Biology_and_Culture.htm is a unknown of type html-file
	Cultural_Anthropology.html is a unknown of type file
	Cultural_Anthropology.zip is a unknown of type archive
	imsmanifest.xml is a unknown of type xml-file

Using federated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Chapter_9__The_Life_Cycle.htm is a unknown of type html-file
	face.jpg is a unknown of type jpeg-file
	Chapter_1__A_Definition.htm is a unknown of type html-file
	About_the_Professor.htm is a unknown of type html-file
	Chapter_6__Religion_and_Culture.htm is a unknown of type html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm is a unknown of type html-file
	Prof_photo__Richley_Crapo.jpg is a unknown of type jpeg-file
	Chapter_14__Cultural_Evolution.htm is a unknown of type html-file
	Chapter_8__Social_Organization_and_Kinship.htm is a unknown of type html-file
	Chapter_12__The_Economic_System.htm is a unknown of type html-file
	Chapter_2__Culture.htm is a unknown of type html-file
	Chapter_5__Language_and_Culture.htm is a unknown of type html-file
	Chapter_10__Marriage__Family__and_Household.htm is a unknown of type html-file
	Schedule.htm is a unknown of type html-file
	Chapter_11__Politics_and_Culture.htm is a unknown of type html-file
	Syllabus.htm is a unknown of type html-file
	Chapter_7__Aesthetics_and_Culture.htm is a unknown of type html-file
	Chapter_4__Culture_and__Personality.htm is a unknown of type html-file
	Chapter_3__Biology_and_Culture.htm is a unknown of type html-file
	Cultural_Anthropology.html is a unknown of type file
	Cultural_Anthropology.zip is a unknown of type archive
	imsmanifest.xml is a unknown of type xml-file

It might be possible to re-code isDirectory() and isFile() based on types, rather than relying on peeking into java.io.File or java.util.zip.ZipEntry. Since these are supposed to be all files, I'm going to take that out of the status print:

Using isolated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.html and has a type of file
	Cultural_Anthropology.zip and has a type of archive
	imsmanifest.xml and has a type of xml-file

Using federated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.html and has a type of file
	Cultural_Anthropology.zip and has a type of archive
	imsmanifest.xml and has a type of xml-file

So the federated view is correct now, but the isolated view is not.

Finish work on fsArchive.getFileAtNode(String path). This should enable the isolated view in DELS.getFilesIsolated() to get files at a specific node of the archive file structure.

Feb. 9, 2009 - Happy Birthday!

Finished coding of fsArchive.getFilesAtNode() and code of DELS.getFilesIsolated(). Test results are now:

Using isolated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	[No files found.]

Using federated directory view,
The path C:/osid-test/Cultural_Anthropology.zip has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.html and has a type of file
	Cultural_Anthropology.zip and has a type of archive
	imsmanifest.xml and has a type of xml-file

This is a successful test run against the root of an archive file. The more interesting case comes when we try to look at a directory embedded in an archive file. We'll try the case of "C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/". We would expect to see just the manifest file in the isolated view, but all files in the federated view.

Created two new methods: DELS.isEmbeddedDirectory() and DELS.splitEmbeddedPath(). Used these in the DELS constructor to check for an embedded directory and strip off the embedded path. So we'll add the embedded case to testArchive() and see what happens:

Isolated view hangs. No files produced.
Fixed bugs in fsArchiveNode.getPath() that caused an infinite loop.
Fixed bugs in fsArchiveFile constructor that threw an exception incorrectly.

Results for the isolated embedded case are:

Using isolated directory view for an embedded directory,
fsArchive.getFilesAtNode() - About to get target node for path /Cultural_Anthropology
fsArchive.getFilesAtNode() - Target node found: Cultural_Anthropology
fsArchive.getFilesAtNode() - Got some children: 3
fsArchive.getFilesAtNode() - Path to try: /Cultural_Anthropology/Cultural_Anthropology.html
fsArchive.getFilesAtNode() - Not a file: /Cultural_Anthropology/Cultural_Anthropology.html
fsArchive.getFilesAtNode() - Path to try: /Cultural_Anthropology/imsmanifest.xml
fsArchive.getFilesAtNode() - Not a file: /Cultural_Anthropology/imsmanifest.xml
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/ has the following entries:
	[No files found.]

The files are being found in the embedded directory, but failing the file check test, which in this case tries to get a ZipEntry for a given path. I suspect the problem is the leading slash. Tried chipping off the leading slash:

Using isolated directory view for an embedded directory,
fsArchive.getFilesAtNode() - About to get target node for path /Cultural_Anthropology
fsArchive.getFilesAtNode() - Target node found: Cultural_Anthropology
fsArchive.getFilesAtNode() - Got some children: 3
fsArchive.getFilesAtNode() - Path to try: Cultural_Anthropology/Cultural_Anthropology.html
fsArchive.getFilesAtNode() - Not a file: Cultural_Anthropology/Cultural_Anthropology.html
fsArchive.getFilesAtNode() - Path to try: Cultural_Anthropology/imsmanifest.xml
fsArchive.getFilesAtNode() - Not a file: Cultural_Anthropology/imsmanifest.xml
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/ has the following entries:
	[No files found.]

No help. Still fails. Was passing the wrong path to fsArchiveFile constructor. Results are now:

Using isolated directory view for an embedded directory,
fsArchive.getFilesAtNode() - About to get target node for path /Cultural_Anthropology
fsArchive.getFilesAtNode() - Target node found: Cultural_Anthropology
fsArchive.getFilesAtNode() - Got some children: 3
fsArchive.getFilesAtNode() - Path to try: Cultural_Anthropology/Cultural_Anthropology.html
fsArchive.getEntry() - Attempting to get Cultural_Anthropology/Cultural_Anthropology.html in the zip file C:\osid-test\Cultural_Anthropology.zip
fsArchive.getFilesAtNode() - Added a file: Cultural_Anthropology/Cultural_Anthropology.html
fsArchive.getFilesAtNode() - Path to try: Cultural_Anthropology/imsmanifest.xml
fsArchive.getEntry() - Attempting to get Cultural_Anthropology/imsmanifest.xml in the zip file C:\osid-test\Cultural_Anthropology.zip
fsArchive.getFilesAtNode() - Added a file: Cultural_Anthropology/imsmanifest.xml
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/ has the following entries:
	Cultural_Anthropology.html and has a type of file
	imsmanifest.xml and has a type of xml-file

I'll back out the printf's and run the full test:

Using isolated directory view for an embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/ has the following entries:
	Cultural_Anthropology.html and has a type of file
	imsmanifest.xml and has a type of xml-file

Using federated directory view for an embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/ has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.html and has a type of file
	Cultural_Anthropology.zip and has a type of archive
	imsmanifest.xml and has a type of xml-file

Now let's add one more test case: "C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology"
Results are:

Using isolated directory view for a deeply embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.zip and has a type of archive

Using federated directory view for a deeply embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.html and has a type of file           <-  Wrong!
	Cultural_Anthropology.zip and has a type of archive         <-  Not clear why this is even here!?!
	imsmanifest.xml and has a type of xml-file                  <-  Wrong!

The isolated directory view is correct: files in that directory are shown. However, the federated view is now wrong (as I suspected it might be). The files "imsmanifest.xml" and "Cultural_Anthropology.html" exist at a higher directory than the one associated with the session. This can be fixed with an isDecendent() test, I think.

Feb. 10, 2009

Added a bit of code to display what we are looking for in decendants:Added a bit of code to display what we are looking for in decendants:

Using federated directory view for a deeply embedded directory,
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_9__The_Life_Cycle.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/face.jpg in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_1__A_Definition.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/About_the_Professor.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_6__Religion_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_13__Environment__Adaption__and_Subsistence.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Prof_photo__Richley_Crapo.jpg in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_14__Cultural_Evolution.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_8__Social_Organization_and_Kinship.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_12__The_Economic_System.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_2__Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_5__Language_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_10__Marriage__Family__and_Household.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Schedule.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_11__Politics_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Syllabus.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_7__Aesthetics_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_4__Culture_and__Personality.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_3__Biology_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology.html in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/imsmanifest.xml in node /Cultural_Anthropology/Cultural_Anthropology
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	[No files found.]

The problem is that full ZipEntry name (Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip) is being checked against node in the archive file system (/Cultural_Anthropology/Cultural_Anthropology) The trick, I think, is to strip of enough of the entry name.

This was done with the following result:

Using isolated directory view for a deeply embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.zip and has a type of archive

Using federated directory view for a deeply embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.zip and has a type of archive

These both look good. The lists are the same because there are no sub-directories to gather federated files from.
All file test cases are working at this point. These cases should also be applied against getFilesByRecordType(). It might be done by calling getFiles() and filtering the result. Implemented getFilesByName() and getFilesByRecordType();

Implemented getSubdirectories() for isolated and federated views of a generic files system. Need to add special case of an archive file.
Added directoryTest() to OsidFilingTest. Initial results are:

Using isolated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory

Using federated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory

The isolated case is right, but the federated view should have another directory in it's list. DELS.buildFederedateDirectoryList() needed some tweaking. New results are:

Using isolated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory

Using federated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory
	subsubfolder and has a type of directory

This correctly includes the subsubfolder in the federated view.

This leaves the archive cases. Implemented archive support for getting directories in both isolated and federated views. Needs testing.

Using federated directory view for a deeply embedded directory,
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_9__The_Life_Cycle.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/face.jpg in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_1__A_Definition.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/About_the_Professor.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_6__Religion_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_13__Environment__Adaption__and_Subsistence.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Prof_photo__Richley_Crapo.jpg in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_14__Cultural_Evolution.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_8__Social_Organization_and_Kinship.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_12__The_Economic_System.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_2__Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_5__Language_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_10__Marriage__Family__and_Household.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Schedule.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_11__Politics_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Syllabus.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_7__Aesthetics_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_4__Culture_and__Personality.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Chapter_3__Biology_and_Culture.htm in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology.html in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip in node /Cultural_Anthropology/Cultural_Anthropology
fsArchiveNode.hasDecendants() - Looking for Cultural_Anthropology/imsmanifest.xml in node /Cultural_Anthropology/Cultural_Anthropology
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	[No files found.]

The problem is that full ZipEntry name (Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip) is being checked against node in the archive file system (/Cultural_Anthropology/Cultural_Anthropology) The trick, I think, is to strip of enough of the entry name.

This was done with the following result:

Using isolated directory view for a deeply embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.zip and has a type of archive

Using federated directory view for a deeply embedded directory,
The path C:/osid-test/Cultural_Anthropology.zip/Cultural_Anthropology/Cultural_Anthropology has the following entries:
	Chapter_9__The_Life_Cycle.htm and has a type of html-file
	face.jpg and has a type of jpeg-file
	Chapter_1__A_Definition.htm and has a type of html-file
	About_the_Professor.htm and has a type of html-file
	Chapter_6__Religion_and_Culture.htm and has a type of html-file
	Chapter_13__Environment__Adaption__and_Subsistence.htm and has a type of html-file
	Prof_photo__Richley_Crapo.jpg and has a type of jpeg-file
	Chapter_14__Cultural_Evolution.htm and has a type of html-file
	Chapter_8__Social_Organization_and_Kinship.htm and has a type of html-file
	Chapter_12__The_Economic_System.htm and has a type of html-file
	Chapter_2__Culture.htm and has a type of html-file
	Chapter_5__Language_and_Culture.htm and has a type of html-file
	Chapter_10__Marriage__Family__and_Household.htm and has a type of html-file
	Schedule.htm and has a type of html-file
	Chapter_11__Politics_and_Culture.htm and has a type of html-file
	Syllabus.htm and has a type of html-file
	Chapter_7__Aesthetics_and_Culture.htm and has a type of html-file
	Chapter_4__Culture_and__Personality.htm and has a type of html-file
	Chapter_3__Biology_and_Culture.htm and has a type of html-file
	Cultural_Anthropology.zip and has a type of archive

These both look good. The lists are the same because there are no sub-directories to gather federated files from.
All file test cases are working at this point. These cases should also be applied against getFilesByRecordType(). It might be done by calling getFiles() and filtering the result. Implemented getFilesByName() and getFilesByRecordType();

Implemented getSubdirectories() for isolated and federated views of a generic files system. Need to add special case of an archive file.
Added directoryTest() to OsidFilingTest. Initial results are:

Using isolated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory

Using federated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory

The isolated case is right, but the federated view should have another directory in it's list. DELS.buildFederedateDirectoryList() needed some tweaking. New results are:

Using isolated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory

Using federated directory view,
The path C:/osid-test has the following entries:
	subfolder and has a type of directory
	subsubfolder and has a type of directory

This correctly includes the subsubfolder in the federated view.

This leaves the archive cases. Implemented archive support for getting directories in both isolated and federated views. Needs testing.

Feb. 11, 2009

Time to move on. The directory stuff needs testing, but I can come back to it later. I need to get archive expansion to work. The idea is to add an expand() method to fsArchive that takes a target path. This is an extension to the official Filing OSID, so it needs to be associated with a record, fsArchiveType in this case. This needs to be done in the context of a DirectoryAdminSession, since files are being created.

This is not as simple as it looks. The DirectoryAdminSession (DAS) has provisions to create a file given a FileForm object. Data elements can be sent in the form, depending on the interface extensions defined by it's record type. Directories can be created in a similar manner. Given all that, how do I create a bunch of files some of which may be in new sub-directories? I can create an ArchiveFileForm with a flag that says to expand it on creation, except the file already exists. Ah, here's an idea. Normally, a DirectoryAdminSession is associated with a directory. Suppose I check for the case where it points to an archive file? when a createDirectory() method is called with a DirectoryForm, a flag can be set to indicate that an expansion should happen. Or maybe that should be flipped: associate the target directory path with the DAS and put the archive file into the DirectoryRecord. I think I like that better. So be it!

Started implementing DirectoryAdminSession. No work had been done previously on it.

The method DirectoryAdminSession.overwrite() is a bit clumsy. It provides a way to indicate that overwrites should be permitted, but there is no way to either
change that status or query it once set. I have implemented this as a toggle, which means that if overwrite is set, calling this will reset it and vice versa.

FileForms allow attributes associated with a file to be set. I will need to shadow the various file types with both forms and records.
Implemented fsDirectoryForm, fsDirectoryRecord, and fsDiretoryFormRecord.

Feb. 12, 2009

Now that we have the basic Record, Form, and FormRecord in place for a directory, we can start thinking about how to handle an archive directory(file). The fsArchiveDirectoryRecord object is coded to recognize a type of ARCHIVE_TYPE_ID.
Coded DirectoryAdminSession.getDirectoryFormForCreation(); Created fsArchiveDirectoryForm which extends fsDirectoryForm.

Created a test case called OsidFilingTest.archiveDirTest() that create a directory admin session for an archive file, then tries to get the directory form for it. It failed on the first run because I was trying to retrieve a type of ("nolaria.com", DIR_TYPE_NS, ARCHIVE_TYPE_ID). However, the archive type is registered under the tuple ("nolaria.com", FILE_TYPE_ID, ARCHIVE_TYPE_ID). Changed over to correct name space. Other problems with name space corrected. Re-wrote getDirectoryFormForCreate() by comparing types instead of checking authority, name space, and id's separately. Better to test for type equality directly.

Okay, now it's time for black magic. I have successfully created a directory admin session for an archive file and gotten a directory form for it (this is actually an fsArchiveDirectoryForm). I propose to add a metadata element which is a boolean flag called "expand" and another String called "target". Set to true, the expand flag will cause the archive to be expanded to the specified target directory. This operation is triggered when the form is passed to DAS.createDirectory().

Methods setExpand() and setTarget() added to fsArchiveDirectoryForm(). Also added a create() method which will be present in all directory and file forms to be called when the file system object is to be created. Results of test are:

Created an admin session for path - C:/osid-test/Cultural_Anthropology.zip
Directory form acquired for archive directory type.
fsArchiveDirectoryForm - Expand the archive into target directory - c:/osid-test/archive
Expanded archive into c:\osid-test\archive

Yippee! Pretty much what I expected. The correct instance of directory form is being detected and the create() method invoked. It, in turn, knows that the expand flag is set and that the target directory should be used to expand the archive. Now I just need to add the code to inflate the archive and we will have a working means to install OCW Common Cartridges.

Hmm, I just noticed something in the java doc. If a session is closed, I should throw IllegalStateException. I should go back and fix these, since they currently throw OperationFailedException.

Feb. 13, 2009

Made a pass over DirectoryEntryLookupSession, DirectoryAdminSession, and FileSession and fixed closed session cases to throw IllegalStateException. Some additional clean-up and organization of methods as well.

Time to have a look at the previous work phase to see how zip files are expanded. The older code was called cc-install, which I renamed to cc-ocw. It was designed to work in a Sakai 2.4.x environment, which is pretty dated by now. When I get around to making the portable application, I'll have to move it up to 2.6.x, though 2.6.1 hasn't been officially released yet.

The following code is excerpted from cc-ocw/src/java/org/sakaiproject/cc/tool/ResultsProducer.java. User interface and debugging information has been stripped out of it to give the gist of the algorithm.

void fillComponents() {
	//	Get the filing manager.
	this.fm = new chsFilingManager();

	//	Get a directory admin session.
	DirectoryAdminSession das = null;

	//	Create the target directory.  Fail with error if it already exists.
	String targetDir = collId + ccName+"/";
	das.createDirectory(targetDir);

	//	Unpack the Cartridge into the target directory.  (InputStream was passed in as a parameter)
	unpack (targetDir, content, in);
}

void unpack(String targetDir, StringBuffer content, InputStream in) {
	ZipInputStream zipin = new ZipInputStream (in);
	ZipEntry entry = null;
	int ct = 1;

	while ((entry=zipin.getNextEntry()) != null) {
		String entryName = entry.getName();
		int entrySize = (int)entry.getSize();

		//	Assemble the entry directory name.
		StringBuffer entryTemp = new StringBuffer(targetDir);
		String[] nodes = entryName.split("/");
		for (int i=0; i<nodes.length-1; i++) {
			entryTemp.append (nodes[i]+"/");
		}
		String entryDir = new String (entryTemp);
		DirectoryAdminSession das = null;
		das = this.fm.getDirectoryAdminSessionForDirectory(entryDir);

		String simpleName = null;
		if (entry.isDirectory()) {
			simpleName = nodes[nodes.length-1]+"/";
			das.createDirectory(entryDir+simpleName);
			}
		else {
			simpleName = nodes[nodes.length-1];
			File file = das.createFile(simpleName);

			FileSession fs = fm.getFileSessionForPath(entryDir+simpleName);

			org.osid.transport.DataOutputStream outStream = fs.getOutputStream();

			/*	Full read approach and direct coupling.  */
			byte[] entryBytes = new  byte[entrySize];
			int rd = zipin.read(entryBytes);

			ByteArrayInputStream bufIn = new ByteArrayInputStream(entryBytes);
			outStream.setDirect(bufIn);

			((org.osid.filing.chsFileSession)fs).commit();	//	Commit the edits.
		}
		ct++;
	}

	zipin.close();
}

One of the key comments here is the "Full read approach and direct coupling." I had a lot of trouble reading from the zip stream and writing out to an entry in the Content Hosting Service, mostly because there was an extra layer of read/write in the osid.Transport. This extra layer introduced I/O blocking problems once the file got large. Some kind of buffered I/O is needed since data files in an archive can be large (consider a video clip, for example).

There are also a number of hacks present here largely due to the early design of the V3 OSIDs at the time. Filing hadn't been considered and largely reflected the older V2 design patterns. Tom has since updated the Filing OSID to the newer patterns and has included org.osid.transport objects.

Feb. 16, 2009

Started writing fsArchiveDirectoryForm.expand(). Added code to open the zip input stream, read entries and print names/sizes:

Created an admin session for path - C:/osid-test/Cultural_Anthropology.zip
Directory form acquired for archive directory type.
fsArchiveDirectoryForm - Expand the archive into target directory - c:/osid-test/archive
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_9__The_Life_Cycle.htm of size: 15350
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/face.jpg of size: 12116
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_1__A_Definition.htm of size: 17262
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/About_the_Professor.htm of size: 1251
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_6__Religion_and_Culture.htm of size: 16402
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_13__Environment__Adaption__and_Subsistence.htm of size: 16965
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Prof_photo__Richley_Crapo.jpg of size: 23127
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_14__Cultural_Evolution.htm of size: 18158
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_8__Social_Organization_and_Kinship.htm of size: 16393
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_12__The_Economic_System.htm of size: 16783
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_2__Culture.htm of size: 16604
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_5__Language_and_Culture.htm of size: 14425
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_10__Marriage__Family__and_Household.htm of size: 16268
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Schedule.htm of size: 3001
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_11__Politics_and_Culture.htm of size: 16059
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Syllabus.htm of size: 525
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_7__Aesthetics_and_Culture.htm of size: 14683
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_4__Culture_and__Personality.htm of size: 16130
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Chapter_3__Biology_and_Culture.htm of size: 16462
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology.html of size: 1189
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip of size: 0
fsArchiveDirectory.expand() - Processing zip entry named: Cultural_Anthropology/imsmanifest.xml of size: 77136
Expanded archive into c:\osid-test\archive

One thing of interest is that the archive entry "Cultural_Anthropology/Cultural_Anthropology/Cultural_Anthropology.zip" has a size of zero. I think I will filter out all files of size zero and thus avoid the problem of creating empty files.

Coded the expansion of archive entries using a BufferedInputStream approach using an 8k buffer size. Some of the images included in this buffer are greater than 8k, so this is a pretty good test of a buffered approach.

Getting closer. Target directory is being created. Archive files are often buried deep in a directory structure that must be explicitly create, if they do not exist.

Amazingly, that was all it took. The zip file is expanded fully. HTML files, jpegs, and manifest are all present. Furthermore, the buffered i/o approach worked, meaning that larger files will be expanded correctly. The phase-1 version attempted to load the whole file into memory all at once and write it out. This worked, but potentially consumed a LOT of memory (think 2GB video files).

Tomcat 6.0.18 was never installed or the older cc-ocw application tried. I need to go back and get the basic shell working again before I can work on adding a UI front end to the new implementation code.

Created javadoc in Eclipse for the impl. Location is c:/newdev/oki/soid-v3-impl/javadoc.

Feb. 17, 2009

Tried to run maven on new version of the OSID implementation. Failed. Can't delete the target, which is open in Eclipse, I would guess. That's a bit of a pain, really, since I want to be able to tweak the impl and then run maven to deploy it. Looks like I can run it without clean, but that's still going to be a problem.

Got apache-tomcat_6.0.18 up and running on my laptop environment.

Made some changes to POM files. The impl dependencies wasn't quite right (referring to "osid" instead of "osid-impl". Gotta get the JAR names to agree or it don't work! The cc-ocw.war is being deployed to tomcat. Interesting. "shared/lib" has become just "lib" in tomcat-6x. That will impact how Sakai is deployed in 6.x. The community isn't ready to switch yet, unfortunately.

Started up tomcat, and used "http://localhost:8080/cc-ocw/faces" to access to the stubbed OCW application. It's nothing more than a hello world application at this point, but it should be simple to bring over the older code into this new environment (ie, outside of Sakai).

Built an eclipse project for cc-ocw. After figuring out the RSF JARS to import (from my local repository), it all compiled in eclipse ok. Just to test things a bit, I changed the main messge to "CC OCW Installer - put code here". Ran maven (with clean - no errors, hmm). Started tomcat, but new message doesn't show. This is the same problem I experienced with tomcat 5.5.27 and the reason I switched to 6.0.18 based on in put from Antranig. Some maven experimentation is called for, I think. It will save much time in development if I can get maven to work peacefully with eclipse. This is one reason why I don't usually use eclipse with deployed projects, BTW.

Looking at tomcat/webapps, there is a folder called cc-ocw and a war called cc-ocw.war.
Running "mvn clean" deletes the target directory, but not deployed versions of it. I think the sakai version does clean things up in tomcat.
Running "mvn install" creates the target directory and new version of the code. No change to tomcat (none expected).
Before running deploy, I looked at the create times. Both the folder and war have a time of 10:43am.
Running "mvn rsf:deploy" updates the create time on the war file, but doesn't affect the expanded folder.

This is pretty much the behavior I saw previously. I'm not sure who deletes the expanded file in tomcat/webapps, but tomcat isn't sensing that a new WAR is available. I could force this using the tomcat manager, but I want it to be automatic, not something I have to poke every time.

Feb. 18, 2009

More experimentation with getting things to build in an automated or semi-automated way. Tried to create Windows BAT file that would delete the expanded form of cc-ocw in Tomcat/webapps. Got the script to work, but it wouldn't actually delete the file. Even CD'ing to the directory (in a CMD shell) and executing "del -f -q cc-ocw" doesn't actually delete anything - with no error messages! The only way to delete this directory (that I have found) is using Windows, which is EXACTLY what I DON'T want, since it means graphical manipulation every time I build the application. Grr. Something is not right somewhere. It feels like a permission problem, but I would expect an error message.

Poked around a bit looking for other CC-OCW instances. I didn't find any on my machines, though I was pretty sure that OU-UK had some. I had a look at the MIT OCW site. They are bundling their OCW courses as IMS-CP. One example, "The Ancient World: Greece" can be downloaded from http://ocw.mit.edu/ans15436/ZipForEndUsers/21H/21H-301Fall-2004/21H-301Fall-2004.zip. The main purpose of this is to get a feel for the manifests. I have a feeling that most, if not all, OCW producing organizations are packaging their content as ZIPs and maybe as IMS-CP. Very, very few will be doing IMS-CC yet. Frankly, there isn't a great deal of incentive for them to use IMS-CC, except that it is an emerging standard and is likely to be more interoperable that plain ole CP (CC is a profile of CP, after all).

A look inside the manifest confirms that MIT is using IMS-CP v1.1, which is the widely accepted version of CP at this point. I think CC is based on 1.2, but most people will not even know the difference. However, the basic CP manifest is also extended using the following namespaces:

• xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" – XML scheme instance
• xmlns:adlcp="http://www.adlnet.org/xsd/adlcp_rootv1p2" – ADL content package conventions
• xmlns:cwsp="http://www.dspace.org/xmlns/cwspace_imscp" – DSpace packaging conventions
• xmlns:ocw="http://ocw.mit.edu/xmlns/ocw_imscp" – MIT's OCW/CP extensions
• xmlns:lom="http://ocw.mit.edu/xmlns/LOM" - MIT's copy of the LOM namespace

Sadly, that's going to REALLY complicate parsing an MIT OCW manifest. Oh, installing the course will be simple - just unzip it. The metadata has value, though, and that's all embedded in the manifest.

Feb. 19, 2009

Spent more time trying to get cc-ocw to deploy to tomcat in something approaching an automated fashion. It does look like cygwin can delete the diretory as "rm -rf cc-ocw", but tomcat cannot be running or it fails. I also had trouble capturing this as a $%#! bash shell. I always did prefer csh. Sigh.

Started work migrating code over to the new OSID application. I think I tried to bite off too much in one chunk. The older app needs to be refactored and rewritten to use a more pure OSID approach. I'm going to drop back to the hello world example again and incrementally migrate over code.

Dammit. I should have backed things up first. The helloworld app is broken even though I cleared out all the imported stuff and reverted to the old web.xml file. I get a startListener error in tomcat start up. Kinda cryptic. It's got to be a misconfiguration of XML files. The java code is simple and compiles.

Started work migrating code over to the new OSID application. I think I tried to bite off too much in one chunk. The older app needs to be refactored and rewritten to use a more pure OSID approach. I'm going to drop back to the hello world example again and incrementally migrate over code.

Dammit. I should have backed things up first. The helloworld app is broken even though I cleared out all the imported stuff and reverted to the old web.xml file. I get a startListener error in tomcat start up. Kinda cryptic. It's got to be a misconfiguration of XML files. The java code is simple and compiles.

Feb. 20, 2009

Ok, I think I have found the problem. When I created the project for cc-ocw in Eclipse, it added content and WEB-INF under src/java for some reason. I've been editing the context files and web.xml files IN THE WRONG PLACE! The eclipse project was badly screwed up. In the end, I deleted the whole thing and re-imported it. In doing so, it identified a missing resource: j-RSFutil. I don't get any compile errors, so this is likely a reference in one of the RSF components. I'll send Antranig a msg and see what he has to say.

Fixed the applicationContext and requestContext files to declare only the CCToolProducer. It still fails. The error message in Tomcat is unusually cryptic for RSF: "SEVERE: m Error listenerStart".
Tried a whole bunch of things none of which solved the problem. So I copied the RSFHelloWorld example into my OKI project. Ran maven, which built and deployed without error. However, when I run "http://localhost:8080/RSFHello/faces", I get an exception that suggests that RSF can't find log4j.

Tried hacking around in maven with dependencies. Different versions, etc. There seem to be several instances and packages of log4j in my repository. Apache suggests that commons-logging.1.1.1 is the latest version, but that didn't do the trick. The root error is identified as "java.log.NoClassDefFoundError: org/apache/commons/logging/LogFactory". This happens in the init (constructor) of ContextLoader. Now I suspect that it isn't my code at all that's the problem. Something about RSF seems to be misconfigured.

Feb. 21, 2009

Feb. 23, 2009

Brigid and I were away for the weekend and the long drive gave me some time to think about the problems I've been having with developing the cc-ocw application. I've spent more than a few hours trying to get Tomcat 6.x to work for me. Well, to hell with it. I'm going back to 5.5.27. Besides working with Sakai, I won't have to worry about getting silly things like commons-logging to work.

Built and deployed RSFHelloWorld to apache-tomcat-5.5.27 and it worked. Whew! At least I'm back to a place I can move forward from. Next up is to get the cc-ocw stub working, too. Somehow, the web.xml file was completely wrong. It was still referring to the Sakai tool listener. I replaced it with the web.xml from RSFHelloWorld, which is completely boilerplate. It doesn't even define the name of the servlet (in the the POM, instead)! Now I even have a sandbox to work in. Huge progress.

Brought over the cctool.html template. This template has support for a hierarchical view of a file system designed to allow the user to select a target directory.

Having trouble with the Messsages bundle. RSF is not seeing it at display time. Had a look at the target and I don't see a Messages.properties file. This is likely a maven problem not building the WAR correctly. Shifted the Messages.properties file to classpath:com/nolaria/ccocw/bundle/Message and it worked. Looks like maven needs to have some directory structure in the WAR to work with. I don't care as long as it works.

I'd like to have a common look and feel for these applications. Since Sakai defines a set of styles, I'd like to use them. I figure I have two approaches to referencing the CSS and JS files:

1. Include them in the WAR
2. Reference them on the Sakai source site.

The latter is a bit of a hack. The former kinda hard codes things, but also gives me more control. I think I will include them.

The old header code looked like this:

    <link href="/library/skin/tool_base.css" rsf:id="scr=portal-matter" type="text/css" rel="stylesheet" media="all"/>
    <link href="/library/skin/default/tool.css" rsf:id="scr=portal-matter" type="text/css" rel="stylesheet" media="all"/>
    <link href="images.css" rel="stylesheet" type="text/css" />
    <script type="text/javascript" rsf:id="scr=portal-matter" language="JavaScript" src="/library/js/headscripts.js"></script>

which is now being replaced by this:

    <link href="tool_base.css" rsf:id="scr=portal-matter" type="text/css" rel="stylesheet" media="all"/>
    <link href="default/tool.css" rsf:id="scr=portal-matter" type="text/css" rel="stylesheet" media="all"/>
    <link href="images.css" rel="stylesheet" type="text/css" />
    <script src="headscripts.js" type="text/javascript" rsf:id="scr=portal-matter" language="JavaScript" ></script>

style sheets and javascript functions have been downloaded into src/webapp/content/templates. Didn't work. I had to add /content/templates to the href's and I'm still not sure if they are working. Not getting any error messages now, however. Styling can wait. I tried using URL references to the Sakai source tree. No visible difference.

Downloaded three icons from the Sakai source system: dir_openroot.gif, dir_open.gif, and dir_closed.gif. These can be referred to directly with no path names. Root directory icon shows.

In the previous version of cc-ocw, the document root was determined by the Sakai course site. This is less obvious in a more generic situation. In fact, several roots are possible and can be accessed via the static File.listRoots() method. I'm planning on using the first root what ever it is. Later we can think about multiple roots. This should all be shifted into the Filing OSID somehow.

Added display of the root directory. Currently comes up at as "c:\".
Created a CC-OCW Things To Do list.

Now that I am transferring code over from Phase-1, I recall what got punted in order to have something that worked for Open-i-World in Lyon. The whole front page view is hard coded to the Sakai Content Hosting Service. Not too much of a problem, I can add in the calls to the Filing OSID at this point. However, state is going to be an issue. RSF normally pushes things into request scope. If I did that with the OsidManager and the FilingManager, a new one would be created for every request. Most of the state is in the TypeRegistry, as I recall. That could be re-built on every request - it's not that hard. There might be other things that require state, however. I need to think about it a bit.

Going to have to force the java-kit into a maven repo. I'm getting a build failure on BootLoader.

Feb. 24, 2009

Application was failing at start up time: OperationFailedException class was not found. This strongly implies a maven problem. Tinkered around a bit with the POM and eliminated <scope>provided</scope> tags in the OSID definitions and impls. That fixed the problem.

Brought over CCToolProducer.fillResources() from the phase 1 version of the application and started converting it to use information from a DirectoryEntryLookupSession.
Scaled back a bit on the code. Stubbed out display of file names showing only directories. I might want to handle archive files here later, though it may require some re-thinking of the user interface flow.

List of directories below the root is displayed. Clicking on one only shows the directory in that directory. Tis a bug in walking the directories.

Feb. 25,2009

Started working on correctly display of intermediate directories when drilling down into the file system.
After much tinkering, I got the directory file system to display on the default view page. I had to add an other parameters to the fillFileTree() method to carry the partial path name. Recursion stops when the partial path name is not longer a prefix of the final path name. I cleaned up the display a bit. Styling is not working so I got rid of the <h3> tags on directory entries and used nbsp for indentation. Much cleaner now. The pop-ids to close directories is broken. This is likely a bug in my DELS implementation on getting the parent of session directory. It should work, but seemingly doesn't.

Here is a view of the application in it's current state of development:

I added a check to OsidFilingTest.testDirectory() for parent directory functionality. That seems to work just fine in case test case I used.

Mar. 2, 2009

I removed the "save" parameter from CCToolParameters. It didn't seem to be needed and I can bring it back again if it really is needed.
Started work on adding the file upload screen. Change the name of the producer to UploadArchive, which is more descriptive than UploadProducer, IMO. Completed coding, added upload.html template, registered producer in requestContext.xml. Added the link in CCToolParameters. Home page now looks like this:

Futzed around with formatting. I still have to address the styling issues, but things are looking better (if very bland). Here is a view of the upload view:

Added the UploadItemBean class to the eclipse project, since it's the next thing in the application. Two problems immediately come up. I need to add the Spring upload JAR and the app relies on the Sakai Session Manager to save some things as state. The bean hands off to the view producer passing things in the session state. I might be able to do something similar in Tomcat, but it seems pretty kludgey.

The (org.springframework / spring / 2.0.6) dependency seems to satisfy the upload code.
At this point, I need to bring in the rest of the application, at least in shell form. ResultsProducer and ErrorProducer were added as classes along with their templates. Changed error template from editerr.html to error.html. ResultsProducer is stubbed to show a message. The rest of it's code needed re-writing anyways to bring it up to V3.

Mar. 3,2009

The request processing bean, UploadItemBean is not being called inspite of being declared in the a UICommand: "#{uploaditem.submit}".
One idea to preserve the information gathered by the upload request is to create a bean in appicationContext that persists over the whole process. Actually a bean should last through a requestContext duration, too.

Mar. 4, 2009

I think I found the problem with the UploadItemBean not being called. It wasn't declared in the applicationContext as an addressable bean. Well, that was likely to be a problem but it still doesn't work. Hmm. There is something wrong with that bean. RSF is not finding and not throwing an error, either - which is rather unusual. RSF is quite good about letting you know what went wrong.

Ok, something is very weird. I added code to inject the UploadItemBean into UploadArchive as a property. A simple println at the start of fillComponents should say if upbean got initialized or not. When I ran it I got NO message (or errors for that matter). I deleted the WAR as well as the expanded copy it in tomcat. Redeployed. Message is now showing that bean is initialized. In a way, that's good news. It means the bean is there, is addressable and can be accessed. What's broken is invoking it on a POST request.

Antranig suggests that I shouldn't have an empty file input control. That removing it might help. Had to adjust the template to remove the rsf:id. Doesn't seem to help.

Mar. 5, 2009

Resolved the POM dependencies. Compiled and now works. As Antranig explains in the chat above, RSF has a default POST resolver in Sakai, but assumes the developer will provide one outside of Sakai. If not is specified, the POST is revolved an empty data set, a valid request it turns out. This empty request then returns to the view making the request - without invoking the action handler! This is further documented now in RSFwiki:

http://www2.caret.cam.ac.uk/rsfwiki/Wiki.jsp?page=ActionHandlerFails

Meanwhile, my application is now correctly advancing to the results page. It's not doing anything with the file uploaded because I stubbed all that code out. Still, the basic structure is there and working and I can get one with it now.

So one of the problems I'm faced with is state across the POST operation. The file name, stream, and target path where saved in a Sakai session properties. Since I don't have Sakai anymore, I need some place else to stash these. It occurred to me that I can create a bean that holds this info and keep it in application context where it will hang around. Technically, this should be in session context, but RSF handles that in a weird sort of way and I'd rather not get into it.

Created ApplicationStateBean that currently has two properties: name and target. Name is the name of the archive to be installed. Target is the target directory. I'm just adding it to UploadArchive initially, to check the injection and spring declarations.

Mar. 6, 2009

Add an input stream to state bean. Wired everything up. Sakai session state is now eliminated. Run and it advanced to the error view. Had to fix up the view to see what error it is and it looks like the action handler is not seeing the multipartMap. Sigh. All of this used to work. It might be that I got rid of the input control on Antranig's suggestion. Now there is no control with an id of "edit-input-file". I will try with "content", that being the same of the input control. Ok, that worked. Actually, I'm a bit surprised.

State variables were messed up a bit. Copy errors in setters. Fixed.

Ok, we now have everything we need at the ResultsProducer view. I need to re-code how the archive is installed using the V3 OSIDs rather than the prototype used in Phase-1.

Mar. 9, 2009

Created a Filing V3 Design page to keep notes on implementation design and possible ideas for metadata and search.
Filing Manager is created.
I'm going to need a DirectoryAdminSession, which I don't think got implemented during the recent round of OSID implementation activity. Looking at fsDirectoryAdminSession, some initial work was done, but it seems to rely on a DirectoryForm.

Creating a directory seems way more complicated that it should be. Given a DirectoryAdminSession, you would think that I could create a directory in it. Nope. I have to request a DirectoryForm based on a Directory Form Type. It's a bit ambiguous on whether this form represents the containing parent directory, of it will represent the directory being created. After all, there is a set method for the name, which would imply that it is different than it's parent. This seems to be the most likely explanation. Unfortunately, I seem to have implemented it using the latter approach (as the directory to be created) for archive expansion. That needs to be fixed. On digging deeper (searching files in Eclipse is cool), I think that I did the right thing. DirectoryAdminSession.getDirectoryFormForCreation() creates DirectoryForms for both plain directories and archives using the containing directory. Looks like better documentation is in order.
The fsArchiveDirectory class contains a method called create() that in turn calls expand(). This is what causes an archive to be expanded. It requires a target, so some fix-up may, in fact, be needed.

Created a new test in OsidFilingTest called dirCreateTest(). It uses a DirectoryAdminSession to get a creation form for a plain directory type. The name of the sub-directory is set in this form, and DAS.createDirectory() is called with the modified form. Works. This needs to be tested against archive expansion, but since I'm headed in that directory anyways, I'll test it at that time.

Mar. 11, 2009

Added code to get the type manager, the basic directory type, and create the install directory in the target directory based on the name provided by the user.

Mar. 12, 2009

Getting tired of too many OSID exceptions. I'm going to write a cover method that catches all OSID errors and displays generic messages. That should GREATLY simplify the code.
Fixed code.
Forgot to compile and deploy changes to OSID impl.
Install directory is now being created correctly in the right target directory.

Taking a second look at things, I don't really need to create a directory for the archive. Expanding the archive handles that for me. However, writing the code to create basic directory has caused me to think about how it happens. I mention above that I'd have to re-visit archive expansion and I think the time is now. Added code to application to write the uploaded archive file to the target directory selected by the user.

The current implementation of fsArchiveDirectoryForm has a setExpand() flag and a setTarget() method to specify the target directory of the expansion. While it's useful to be able to specify a target directory other than the one of the current DirectoryAdminSession, it should default to the DAS directory. That's the first change to make. Secondly, I don't really think I need a flag to indicate expansion. After all, this is a very special form and if an archive directory is being created, it is safe to assume that it means expanding the archive into the target. The nice part of this is that I can eliminate an out of band agreement (OBA). I'm all for that, since they are basically evil (with regard to portability and interoperability).

Removed the expand flag. Hmm. The expand() implementation uses the super.directory as the reference to the archive file (which is a directory, so that's understandable). This means that a target directory is required. There is needs to be a way to specify both of these things, after all and one of will be an extension to the official OSID. I still need to make some changes because I have different conventions for regular directories vs. archive directories. Creating a regular directory I assume the super.directory is the target directory and the name is the name of the new directory to be created. In the archive form, the super.directory refers to the archive file and the name is the name of the expansion. A target string is used to specify the target directory. I think these need to be brought into alignment, which means swapping the use of super.directory and target. I'll rename target to archive.

But wait! This starts to deviate from the idea of treating an archive file like it was a directory. This is WHY super.directory is used to refer to the archive. We are treating it like a directory. Using create to cause expansion is the weirdness being introduced. The true semantics should be that create causes a directory to be created in the archive file. That's a bit skewed, too, since you can't create an empty directory in an archive. It must contain at least one file. In fact, archives don't really support directories - they just have the notion of an internal path to a file.

So let's leave it as is. An OBA can't be avoided here because there really is at least one new piece of information (the target directory path).

Mar. 13, 2009 - Archive Upload and Installation

Polished off remaining code to upload and expand an archive file. It seems that the target name must include a .zip extension. Added code to test for this and if not present then append it. The file extension is how the system determines file type, so it must be present before expanding the file.

Archive expansion is failing. I'm getting a permission violation of some kind: "c:\a-osid-target\a-subfolder (access is denied). I check the permissions on the folder in Windows and (naturally), the Read-only checkbox on the folder is checked with a grey checkmark. This is supposed to indicate that one or more folders or files in it are read-only. However, that is not the case here. It also shouldn't prevent me from creating files. Looking more closely, it is failing on opening the archive, specifically when creating a FileInputStream from a File. I suspect the problem is the changes I made when playing with create directory. The super.directory should be the archive in this case, but it may be defaulting to the session directory. Confirmed. I think this can be fixed by changing the DirectoryAdminSession initialization. One problem was fixed but another has appeared. Now I'm getting a "directory exists" error. The code code assumes that the target directory doesn't exist and wants to create it. I need fix the target directory using the install name. Fixed. Uploaded archive is now expanded correctly.

There are some questions around the UI design. Given that OCW packages are usually course materials, it may not be desirable to allow the user to specify a target name. Rather, it may be better to upload the archive and expand it in place. Multiple versions of an installed packages doesn't seem like a common use case and it can be worked around by renaming directories.

Things To Do

1. Metadata extraction from manifest
2. Other manifest considerations (authorization?)
3. Back port to Sakai

March 16, 2009

9pm to 10pm

Implemented TypeAdminSession and TypeForm. Needed to add methods to fsType and fsTypeRegistry.

March 17, 20009

8:30a to 12:00n

Started to work on XML file support, which will bring in a new dependency on XML (Xerces, etc.). I don't remember how to open and parse and XML file, so I'll have to hunt up an example. This is needed to open and parse the manifest file in a common cartridge.

Need to implement FileForm. I note that in FileForm and DirectoryForm, there is a method called getFileFormRecord() or getDirectoryFormRecord() given a type. If the record was initialized to the current values of the form, this would be a way to get at them. It might explain the existance of this kind of record, which wasn't really obvious before.

The javadoc for FileFormRecord states: "A record for a FileForm. The methods specified by the record type are available through the underlying object." Furthermore, it doesn't have get methods for the data supported by FileForm (comment, name, genus type, and owner). Is that a bug? It sounds like it.

Fixed up getDirectoryFormRecord() in DirectoryForm to match how it is handled in FileForm.
Implemented fsXmlFileForm.getDocument(). Need to add XML dependencies to POM and test it.

March 18, 2009

8:30p to 10:00p

Worked on filling out the FilingProfile. Several of these methods could be implemented, but were not. Many things in the Filing OSID are still not supported at this time, but the profile should be kept up to date with what is and isn't.

Added a fsTypeRegistry.getTypesByNamespace() method to implement the collection methods in fsFilingProfile.

Started work on File Searching. I'd like to implement regular expression searching, as implemented by the java.util.regex classes. Other kinds of searching are possible including startsWith (XXX*), endsWith(XXX), and contains (*XXX). Regular expressions cover all of this, of course.
Started implementation of fsFileSearchSession.
Created fsOsidEquery, fsDirectoryEntryQuery, and fsFileQuery classes. These hold query information.
Created fsSearchOrder (implemented) and fsFileSearchOrder (implemented).

Implemented basic match criteria records in fsOsidQuery, fsDirectoryEntryQuery, and fsFileQuery.

March 19, 2009

10:00a - 12:00n OKI Meeting Part 1

Mark Report

• CC-OCW, a V3 Filing Implementation
• Interest in Educational Service Bus
• Evolving and refining the OSID architecture

1:00p - 4:00p

Presented lessons learned.
Mark to start OSID Bus project.

7:00p - 10:00p

Worked on file search again.
Implemented fsOsidSearch and fsFileSearch.

It came out in today's meeting that one way to avoid some methods in sessions that largely go unimplemented is to put them into a base class and then elminate them from class extensions. Implemented fsOsidSession. This allows me to remove authentication and transaction methods in other sessions.

Created an OsidFilingTest.testFileSearch focused on match all intially.
Basic search against matchAll() works in both isolated and federated directory views. The matchAll doesn't take any criterion data, it just always matches which garauntees that all files will be in the result set. A more interesting example would be matchName(), which does take a key and a search type. More than one kind of search can be implemented using different kinds of search types. Some examples would include:

• STRING_ALL - match the key against the full name.
• REGEX - apply the regular expression against the file name.
• PREFIX - full name starts with key.
• SUFFIX - full name ends with key (good for searching for file extensions)
• CONTAINS - full name contains the key string.

Created the following search types:

1. fsDirectoryEntrySearchType
2. fsFileSearchtype
3. fsFileNameAllSearchType
4. fsFileNameRegexSearchType
5. fsFileNamePrefixSearchType
6. fsFileNameSuffixSearchType
7. fsFileNameContainsSearchType

Implemented fsDirectoryEntryQuery.getStringMatchTypes()

Test results are:

Had suffix, prefix, and contains working. Broke it while trying to get multiple criteria to work.
Query search is basically working at this point, though there are a lot more criteria that could be implemented.

March 23, 2009 - In flight to Los Angeles

Debugged matching algorithm and got it to work again for NAME matches. Mutliple criteria also works.
Coded up searching by file size. Four types are supported:

• Inclusive: min <= size <= max
• Exclusive: min < size < max
• Greater than: size > min
• Less than: size < max

This overlap a bit, but I think they cover the common use cases for searching by size. Other flavors are possible, equal for example.
Well, after a fair bit of work, I find that matchSize() in FileQuery doesn't take a type! I guess I am to assume inclusive or exclusive, so I'll go with inclusive.
Test cases don't seem to be working:

Fixed. Results are now:

Spent some time working up OSID Bus notes.

Take some time to create an authentication implementation. Lightwieght, focused on login/authentication plus access to agents. Tom's javakit may be of help here. This can serve as a test bed for the OSID Service Bus. A working set of agent calls would be useful anyways (persist to XML?).

March 24, 2009 - Long Beach

5:00a - 6:30a

Started work on an Authentication implementation. Eventually, I expect more than one implementation to authorize against systems like LDAP, CAS, etc. In this first approach, I'm going to use files to hold passwords and user attributes.

To do this, I'm going to need a system configuration parameter that indicates where these special files are kept. The OsidRuntimeManager has provisions for such a thing. I need to play around with it a bit first.

March 25, 2009 - Long Beach

9:00p - 11:00p - LA Airport

The configuration system is totally over the top. I can't do even the simplest thing without an Id, Type, Session, etc. All I want to do is designate a particular directory as a system directory to hold agent data. Screw it. I'm going to hardwire it into the AuthenticationManager.

Well, here's a good one. There is no AuthenticationManager OSID included in the d3 jar. That kinda makes it hard to implement authentication, eh?

It looks like AgentQuery is included, so it might be done from there. What I really need is an AgentLookupSession, but that's not included in the OSIDs either. The whole point of this was to set up some simple authentication to test the OSID service bus concepts, but that pretty much cannot happen without the OSIDs. It's a bit strange, since the OSID shows up in the JavaDoc. It's a bug of the first order.

Ok, given the JavaDoc as an assumed definition of the missing OSIDs, I have created the following interfaces in org.osid.authentication:

• AuthenticationManager - only supported sessions.
• AgentAdminSession
• AgentLookupSession
• AgentForm - stubbed

Started an implementation of fsAuthenticationManager. The agent sessions are going to need to know the location of the agent files, currently saved in the AuthnManager. If the sessions are created from the manager (the proper manner), then I can initialize this info in the sessions. However, if the sessions are constructed (as is done elsewhere in some cases), I won't know the directory.

March 26, 2009

10:00a - 11:30a - Newark Liberty Airport

Implemented fsAgent against a file in /agents. Windows-XP didn't like naming a directory with a leading dot.
Wrote an agentTest with the following results:

So the basic agent properties work and can be extended to include any user metadata.
Added code to AuthenticationManager to register itself with the fsManagerCache and to create a new AgentLookupSession.
Implemented AgentLookupSession.getAgent().
Added a second user (brigid) and looked up the agent using an Id:

• Got an agent via an agent lookup session for A. Brigid Cassidy

Need to add AuthenticationValidationSession to the stubbed AuthenticationManager OSID and implement it.

March 29, 2008

Revisited the Configuration OSID. Value extends Parameter. If I implement a ValueLookupSession and pass an Id who's identifier is the key to ValueLookupSession.getValues(), it will return a ValueList with (hopefully) a single entry. Configuration id's could be used to refer to parameter files.

I think the way to initialize the Configuration Manager is to re-write the OsidBootstrap and OsidRuntimeManager.

Created a com.nolaria.runtime.fsOsidRuntimeManager implementation based on Tom's RuntimeManager code that folds in the fsManagerCache methods. This should allow static access to the runtime manager from anywhere in the Filing OSID code (though that will increase brittleness). More importantly, it lets me leverage the context provided by a BootLoader (to be written)and initialize the ConfigurationManager appropriately. Tom's BootLoader can be co-opted to create a new, simpler one. I don't need much of the stuff that in it now. The getRuntimeManager() method is the main piece that needs to be moved over.