2006 Stanford Summer Pilot

Stanford Summer Pilot 2006 - Four English for Foreign Students Language Courses

Ken Romeo, Stanford Languange Instructor/Language Lab staff

-Assessment-taking:
The inclusion of a "Submit" button in addition to the "Save" button seemed to create more confusion than convenience. The problem was that "Submit" is irrevocable.
For students, the fact that quizzes do not import most recent responses on successive attempts was a major inconvenience. Perhaps the concept was that quizzes are more formal, but this was the only arena for a short answer type assignment. Also, quizzes were not automatically submitted from the saved state on the deadline - I think it is a fair assumption to think that they would be but it was not so.

-Assessment authoring:
There is no way to show a solution on assignments. Perhaps the idea was to accomplish this with a quiz, but quizzes do not handle media. My assignment was a transcription of a short audio clip and I wanted to keep the audio as secure as possible (read: I did not want to just post it on a web-server). In the end, I posted the solution as an announcement.

Resubmission before the deadline has to be granted on an individual basis. In general, if there was confusion, there were many students who had problems, and giving individual permission for resubmission is very time-consuming. Also, the definition of "Resubmission" was not really clear: with paper assignments, the due date is somewhat flexible and can be granted individually, but with the checkbox in CourseWork, the submission date could not be beyond the retraction date of the assignment. The obvious solution was to make the retraction date for all assignments be the end of the course, but I did not know this when I set them up before the course.

The tests and quizzes interface has far too many options. While some of them are very interesting, there is actually a much smaller subset that is actually useful. However, putting the problem in terms of complexity of the interface avoids what I think is the major issue:

From a software point of view, it is quite an accomplishment to have all this functionality in a fairly reliable package that can run on such a large scale. And from a student point of view, it is mostly very workable and useful (the conditions surrounding Save and Submit are, unfortunately a step in the wrong direction from previous versions). However, setting up tests and quizzes, and assignments for that matter, is a nightmare for teachers. My own personal experience has been that this has been the weakest point of CourseWork from the beginning: it seems that it would benefit from more input from teachers. The idea of using dropdown selections and separating each item is good for e-commerce, but that is simply not the way that teachers have ever made tests, quizzes or assignments. Most teachers approach them as just text, with a question, and a blank for the answer, or three selections for a multiple choice. Also, quizzes are often administered on a very regular basis, often with a similar pattern but always with different content, so streamlining the input process is essential. Here, streamlining means less mouse use: everything on one screen, tabbing between text boxes or directly inputting code (XML tags?). Often, weekly quizzes are designed once and then copied as new documents with the questions changed. I realize that you do have that to a certain extent with the template capability, which can save the settings. However, the templates only address the settings and lack the format and structure of the quiz, which is the key feature for teachers. Also, the basic interface for quiz creation is spread out over several screens - my screen will hold about 3 items, which was about 3 too few this summer. It could probably be argued that I am describing an input method that is very similar to MSWord or MSExcel, but again, I think that is how many people approach composing on a computer. Just as an idea, imagine an interface like this:

All cells are text boxes except for QuestionType, which is a dropdown selection. If the QuestionType is not Multiple Choice (MC), Selections 1, 2 and 3 are blanked or ignored. Clicking "Save" would give a WYSIWYG display of the student interface. Ideally, this would be editable, but if not, a wiki-like editing function would be fine, as long as the placement of the media could be controlled as well.
As I mentioned, the capabilities of the tool are quite impressive, but I think that there is a set, finite number of ways that teachers will use it, especially in the beginning. The number of settings for each assessment is just far too large to even be functional. CourseWork 3 had a good set of choices, even if it had the same problems with number of screens and too much mouse use (I must admit, however, that the reload time in CW3 was so frustrating that I am tempted to forgive any other difficulties that CW5 might have). It may be that most CW use is for paper submissions and this tool is not that important. However, speaking for the Language Center instructors, I think it would be well worth your time, as far as post-release support goes, to sit down with some teachers who do use CW3 for quizzes, to explore possibilities for the teacher interface. One particularly important group of instructors is the coordinators for each language program, who compose the Oral Diagnostic Assessments (ODAs), which most courses in the Language Center require. It is difficult to understate how important these are to most language instruction at Stanford, but they require attached media and recording capabilities, so the whole issue may be a little bit farther in the future.