The user interview is our chance to learn about our users directly by asking them questions and observing their behavior and environment. This Interview Protocol outlines the procedure and methods for conducting user interviews.

Interviewee Consent Form

Confidentiality and consent forms are standard practice for research studies that involve direct interaction with people. They serve to make sure that interviewees understand what they are getting into, can choose to discontinue their participation at any time they see fit and that the data they provide will be kept in the strictest confidentiality. The form also helps set the interviewee at ease to provide honest, critical answers to questions that they may normally gloss over if they knew their answers were going to be made more publicly available.

All our interview teams need to carefully abide by the directives set forth in our confidentiality and consent forms. In summary, these directives are:

• Raw data from the interview will not be shared with anyone outside the interview team.
• The interview team may share the data from the interview with other research colleagues, but all personally identifiable information about the interviewee or anyone else the interviewee personally identifies will be removed. There should be no way the data can be linked back to the interviewee.
• The interviewee is free to deny responding to any questions or prompts presented by the interview team.
• The interviewee is free to end the interview at any time.

IRB Approval

An Institutional Review Board (IRB) is a committee that has been formally designated to approve, monitor, and review biomedical and behavioral research involving humans with the aim to protect the rights and welfare of the research subjects. Your institution likely has an IRB and needs to be consulted when conducting different types of research on human subjects.

Because our work for this project is related to the design of a new system, we are not asking interviewees to expose themselves to any risk or harm outside of that encountered in daily life and we are not intending to formally publish (i.e. in a journal or archived conference proceedings) our findings, we can conduct our work without IRB approval.

You may be interested, however, in seeking IRB approval at your institution for your user interviews. This would allow you to do your own analysis of the data you have collected and publish the results. This project likely qualifies as an exempt IRB case, so IRB processing may be a far faster process than normal. More information on IRB processing coming soon ...

Interview Script

The Interview Script is a document that the interview team can follow to conduct the interview. It contains an introduction, the questions to be asked of the interviewee and topics to focus on in observation.

Interview Introduction

A good introduction before beginning the interview is necessary to make sure the interviewee understands the goals of the interview, the relationship between the interviewee and the interview team and how best to respond to questions and other prompts.

The Interview Introduction is included below and is part of the Interview Script.

Interview Questions

Interview Phases

The interview questions can be broken down into the following sections:

• Warm-up: getting to know the instructor a little bit better and giving them some simple questions to get them feeling confident.
• General Questions: learning a little bit more about the goals, attitudes and activities of the instructor with more open-ended questions.
• Observation: focusing in on a specific activity and having the instructor think aloud, sharing details of why and how they perform tasks that comprise an activity.
• Follow-up: asking questions to clarify missing, ambiguous or confusing information from observation.
• Wrap-up: asking questions about the interviewing process, anything they would like to add and if it would be all right to contact them with follow-up questions or the opportunity to give feedback on prototype designs.

Format of a question

The format of the questions in our Interview Script are as follows:

[type] Question to be posed to instructor. Topics to cover, potential follow-up questions, focusing questions.

Example:
[Motivation] What do you enjoy most about your job as an instructor? Why? What activities do you always tackle first?

Types of questions:

• [Variables] - Verifying demographic and behavioral variables
• [Mental Model] - How does the instructor think about work, resources, services & systems?
• [Activities] - What kinds of activities does the instructor perform? Do they occur regularly or occasionally?
• [Motivation] - What are the instructor's goals? What does the instructor enjoy doing?
• [Opportunities] - What are the instructor's pain points? What do they avoid?
• [Interactions] - Who or what does the instructor consult or work with?
• [Process] - How does the instructor make decisions and perform tasks?

Observation

In this part of the interview, the interview team acts as apprentices observing as the instructor walks through an activity, interrupting with questions only occasionally. The goal is for the interview team to be able to re-create the work of the instructor. The activity of interest should be one that the instructor has said they perform regularly in preparing or conducting their course and scholarly resources are used in the activity.

For example, the instructor may regularly use journal articles as reading assignments when building a syllabus in preparation for a course. Given this example, the interview team may request the instructor give a running description of how they set up a syllabus:

Be sure to have the instructor focus on just one activity, think aloud and actually complete the activity. If time does not allow for the completion of the activity, be sure to ask how the instructor would finish the activity.

Areas to focus on during observation:

• Sequences - in what order is the instructor performing tasks? Are there interruptions? Are they doing more than one thing at a time? We want to be able to create rich workflow models (exactly how the user completes certain tasks) from this kind of data
• Tools - what is the instructor using to perform their work?
• Problems - what is frustrating or unexpected for the instructor?
• Mental models - how does the instructor think about their work? What expectations do they have?
• Interactions - do they consult different people, reference materials, processes? If so, what for?

Interrupting the instructor during observation:

The interview team should interrupt the instructor only in the following circumstances:

• The instructor is doing a lot of work without speaking about it – Ask the instructor questions about what they are doing and remind them to give a running description of what they are doing so that you, the interview team, may be able to re-create their work.
• The instructor is giving very general or broad details about an important task or action – Ask the instructor questions about what they are doing and remind them that you are looking for details of their work so that you can re-create it. Remind them that they can recount the details of the last time they went through this activity to make it more concrete.
• It is unclear what the instructor is talking about – the instructor may use jargon that the interview team is unfamiliar with. As an apprentice, it is important to know what you are dealing with and it is all right to interrupt the instructor to ask a clarifying question when you do not know what they are talking about.

Follow-up Questions

In this part of the interview, the interview team takes time to reflect on the observation:

• Are there gaps in the process the instructor has described to you? Now is the time to investigate and make sure all the steps involved in completing the activity are recorded. The interview team may end up delving back into observation mode as the instructor remembers something they forgot to mention and begins another description of tasks and actions.
• Are there other parts of the process that are unclear? Now is the time to clarify.
• Were there instances where the interviewer or note-taker had a question, but did not ask because it would have adversely interrupted the instructor? Now is the time to ask these questions.

Wrap-up Questions

Note-taking

Guidelines:

• Try and capture every word the user says!
• Take notes directly on a laptop if the note-taker is a fast typist
  • Taking notes by hand (can be useful in creating diagrams) or making an audio recording are also suitable options if the note-taker is more comfortable with these methods. We do require that interview notes shared with the rest of the group be typed up. Taking notes by hand or audio recording and then transcribing them may take a significant amount of time.
• Abbreviations are helpful, paraphrasing is OK
  • Try and record especially poignant quotes directly and mark them as quotes (i.e. "The library is too overwhelming to approach cold.")
• Raw notes do not need to be perfect
  • Clean up notes during debrief soon after the interview
• It's OK for the note-taker to ask to repeat something or ask for clarification (without interrupting the instructor too much)
  • in between questions are a good time
• If the interviewer comes up with new questions, record them in the notes

Post-interview Debrief

• Arrange a debriefing session as soon as possible after the interview
• Discuss and record your initial general impressions from the interview in the Debriefing Notes section at the end of the Interview Script
• Save a second file (Save as...) as the "clean" version of the notes
• Go through the notes removing any personally identifiable information (names, non-work related places, etc.)
  • Names/places should be substituted with anonymous titles:
    • Gaurav Bhatnagar helped me ... --> [a faculty colleague] helped me ...
• Go through the notes adding text to make the notes understandable by those outside of the interview
  • Complete fragmented sentences or thoughts
  • Expand abbreviations