This page outlines a distributed, multi-institutional user interviewing process for the Library & Sakai 3 Integration Project.

Goals

The overarching goals for this user interview process are to collect data to answer our main research question and build models of our users and their behavior. Our models take the form of personas, mental models, and workflow models. Personas and mental models help us to formalize the goals, behaviors, attitudes and aptitudes of our users. Workflow models help us understand precisely how they do their work, including where there are breakdowns and frustrations.

Process

Define a research question

The research question for user interviews defines the overarching question that we want to answer by speaking with users. For example:

• Why and how do instructors use scholarly resources while preparing their courses in Sakai?

The research question defines a particular activity to investigate as well as the users that perform this activity. The research question is not directly asked of users. It acts more like a guide for the many other questions developed to ask users that will uncover parts of this larger research question.

The overarching research question for this project will be discussed at the 17 Sept Library & Sakai 3 Integration Web Meeting. More info on potential research questions at Library & Sakai 3 Problem & Vision.

Develop a data analysis group

The data analysis group is responsible for defining and analyzing the data collected by user interviewers and developing personas, mental models and workflow models based on this data.

Joining the data analysis group is a serious time and effort commitment. Only those who have experience in user interviewing and developing personas, mental models and workflow models should join the data analysis group. Being a part of the data analysis group will require:

• defining interviewee audience and recruitment plan
• scheduling and conducting interviews
• having one-hour debrief sessions with interviewers at other campuses to properly understand user interview data
• building personas, mental models and workflow models
• working closely with the rest of the data analysis group to distribute work for debriefing sessions and building models

Define the target interviewee audience

Number of interviewees

Generally, around six interviewees in each major user role are needed for strong persona development. For example, if studying instructor use of scholarly resources for course preparation in Sakai, we may want to interview the following numbers and types of instructors:

• 6 Professors (including Assistant Professors and Adjunct Professors)
• 6 Graduate Student Instructors (a.k.a. Teaching Assistants)
• 6 Instructor Assistants (staff that create Sakai sites for professors)

Behavioral & Demographic Variables

We want to have a set of interviewees that represent as best as possible the larger user group. To determine what qualifies as "representative," it is important to use behavioral and demographic variables to distinguish between different types of users.

For example, if we are studying instructor use of scholarly resources for course preparation in Sakai, we would want instructors who have used scholarly resources within Sakai to prepare for a course before. Some good behavioral and demographic variables to get a representative set of interviewees would be:

• Age (from young to old)
• Status (graduate student instructor, adjunct instructor, instructor assistants, assistant professor, professor, professor's aide, etc.)
• Department (humanities, social sciences, arts, engineering, etc.)
• Department size (few colleagues to many colleagues)
• Commitment to teaching and research (from all teaching to all research)
• Teaching primarily large or small courses (from all large to all small)
• Teaching primarily undergraduate or graduate courses (from all undergrad to all grad)
• Comfort level with online scholarly research (from very uncomfortable to very comfortable)
• Amount of time spent conducting online scholarly research (from none to a lot)
• Comfort level with Sakai (from very uncomfortable to very comfortable)
• Amount of time spent using Sakai (from none to a lot)

Develop an interviewee recruitment plan

Using the demographic and behavioral variables defined, a diverse set of interview archetypes can be developed that cover the entire spectrum of users. Each variable will have a natural distribution that we will want to match. For example, age of instructor. In the real world, if we are considering just professors, there will likely be a few professors around 30 years old (20%), a majority of professors who are around 50 years old (60%) and a few professors who are around 70 years old (20%). If we want to interview 5 professors, we would want to interview one around 30 years old, three around 50 years old and one around 70 years old.

We can expand our variables and combine them into a list of desired interviewees, or interviewee archetypes. With a set of "interviewees" to choose from, different institutions participating in our user interview research project can select those archetypes that they think match real users they have easy access to.

Using the variables from above, below is a sample list of interviewee archetypes (for illustration purposes only):

Develop an interview protocol

An interview protocol is used to structure and run an interview. It contains:

• how long the interviews should take
• an introduction to the interviewee
• privacy and confidentiality rules and a statement to be signed by interviewee
• questions to ask the interviewee

For this project:

• interviews should take one hour
• privacy and confidentiality rules should allow us to not require IRB approval
• introduction and questions shall be defined after research question and preceding steps are completed

In general, because we want to develop specific types of models from user interview data, we will need to collect user data in the following areas:

• (Personas) Goals and motivations for the activity and workflow.
• (Personas/Workflow) When, why, and how an activity is performed. A good practice is to ask the interview to recount, in great detail, the last time they completed an activity. Follow up questions to understand average behavior are asked later.
• (Personas/Workflow) Problems and frustrations with the activity.
• (Mental) How users think about their jobs and activities, as well as what expectations users have about systems they encounter.

Schedule and conduct interviews

Scheduling interviews may take up to an hour of interviewer time. The interviews themselves should take one one hour at the place where the user does the work under study.

The following general best practices should be discussed with the data analysis group before the interviews and followed during the interviews (from About Face 3.0, p. 65):

• Interview where the interaction happens
• Avoid a fixed set of questions (we will need to adjust for this given our distributed nature)
• Focus on goals first, tasks second
• Avoid making the user a designer
• Avoid discussions of technology
• Encourage storytelling
• Ask for a show and tell
• Avoid leading questions

The Interview Team

Ideally, the interview team will consist of two people: one note-taker and one interviewer. The interviewer engages the interviewee by asking questions and may take light notes. The note-taker takes detailed notes of the entire interview. The note-taker may take notes by hand or via laptop. In the end, to be able to share notes with others, an electronic document will be necessary. The interview team can optionally create an audio recording of the interview for later reference and to help clean up or analyze notes. Transcription of audio recordings is too time consuming to be widely recommended. Audio recordings should not be shared outside of the interviewing team because they will likely contain personally identifiable information about the interviewee.

Post-interview Debrief

After the interviewing team has finished the interview, the interviewer and note-taker should have a debrief session where they review the interview. This should happen as soon as possible after the interview. It will be important for the team to discuss particularly interesting items and may optionally write a brief summary of their learnings at the end of the interview notes.

During the this debrief, the interview team should go through the notes and be sure to obscure any information that may be used to identify the interviewee. The notes should also be cleaned up so they can be understood by others outside of the interview (view sample interview notes).

At this point, the interview notes should be shared with the data analysis group.

Debrief with Data Analysis Group

Once the data analysis group has an opportunity to do an initial analysis of the notes from one interview, the person or team doing the initial analysis should have a maximum one-hour debrief with the interview team. Though the interview notes may be exemplary, it will be difficult for the data analysis group to get a strong idea of the user just from notes. By doing an initial analysis and getting the chance to ask some follow-up questions of the interviewers, the data analysis group can get a much better feel for the interviewee.

After debriefing with the interviewing team, the data analysis group members will need to coordinate building personas, mental models and workflow models, likely keeping in touch with interview teams via email or phone.