empirical-methods

Homepage for 17-803 "Empirical Methods" at Carnegie Mellon University

Project maintained by bvasiles Hosted on GitHub Pages — Theme by mattgraham

L5: Conducting Interviews (pdf, video)

Lecture5-Interviews

Interviews are probably the most common method of data gathering in qualitative research. There are a variety of forms of qualitative research interviews, among which semi-structured interviews are the most typically used in computer science. Interviews are a great way to validate data and wonderful tools for exploratory investigation. The can often drive the formation of theories and hypotheses, and they are very common in mixed-methods designs. In this lecture we go over all the steps to rigorously conduct interviews. We discuss in detail how to create the interview guide (or “protocol”), how to recruit participants, and how to carry out the actual interviews (including practical guidance for motivating participants, asking, listening, understanding, probing, maintaining control, reinforcing, and recording).

Digital versions of all readings are available with CMU login.

Lecture Readings

King, N. (2004). Using interviews in qualitative research. In C. Cassell & G. Symon (Eds.), Essential Guide to Qualitative Methods in Organizational Research (pp. 11-22). London: Sage.

This chapter describes the main practical issues involved in conducting qualitative research interviews. It covers four steps in the process of constructing and using qualitative research interviews: defining the research question; creating the interview guide; recruiting participants; and carrying out the interviews. General advantages and disadvantages of the method are also discussed.

Weiss, R. S. (1995). Learning from strangers: The art and method of qualitative interview studies. Simon and Schuster. (Ch 4).

This chapter is another great discussion of the practical issues involved in conducting qualitative interviews. There are also many examples of interview transcripts illustrating the different best practices and anti-patterns.

Seidman, I. (2012). Interviewing as qualitative research: A guide for researchers in education and the social sciences: Teachers college press. (Ch 4, Ch 6).

Ch4: Establishing access to, making contact, and selecting participants. The perils of easy access (e.g., interviewing people whom you supervise, your students, acquaintances, friends). Access through formal (unavoidable) and informal (useful if buy in) gatekeepers. Purposeful sampling to select participants (instead of random sampling as typical in experimental and quasi-experimental studies, since it is not feasible). Criteria for deciding how many participants are enough (sufficiency, saturation of information).

Ch6: Techniques and skills of interviewing. Listen more, talk less. Follow up on what the participant says (ask questions when you do not understand; ask to hear more about a subject; explore rather than “probe”). Avoid leading questions. Ask open-ended questions. Follow up, but don’t interrupt. Ask participants to reconstruct, not to remember. Keep participants focused and ask for concrete details. Follow your hunches. Use the interview guide (‘protocol’) cautiously (prefer questions that follow from what the participant has said, even if out of order).

Example Papers

Barwulor, C., McDonald, A., Hargittai, E., & Redmiles, E. M. (2021). “Disadvantaged in the American-dominated internet”: Sex, Work, and Technology. In Proceedings of the 2021 ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) (pp. 931-936).

Wash, R. (2010). Folk models of home computer security. In Proceedings of the Sixth Symposium on Usable Privacy and Security (SOUPS) (pp. 1-16).

Singer, L., Figueira Filho, F., & Storey, M. A. (2014). Software engineering at the speed of light: how developers stay current using Twitter. In Proceedings of the 36th International Conference on Software Engineering (pp. 211-221).

Cherubini, M., Venolia, G., DeLine, R., & Ko, A. J. (2007). Let’s go to the whiteboard: how and why software developers use drawings. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 557-566).

Aranda, J., & Venolia, G. (2009). The secret life of bugs: Going past the errors and omissions in software repositories. In Proceedings of the 31st International Conference on Software Engineering (pp. 298-308).

Example Interview Protocols

Bogart, C., Kästner, C., Herbsleb, J., & Thung, F. (2021). When and how to make breaking changes: Policies and practices in 18 open source software ecosystems. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(4), 1-56.

Nahar, N., Zhou, S., Lewis, G., & Kästner, C. (2022). Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process. International Conference on Software Engineering (ICSE). ACM.

Egelman, C. D., Murphy-Hill, E., Kammer, E., Hodges, M. M., Green, C., Jaspan, C., & Lin, J. (2020). Predicting developers’ negative feelings about code review. International Conference on Software Engineering (ICSE) (pp. 174-185). IEEE.

Seering, J., Wang, T., Yoon, J., & Kaufman, G. (2019). Moderator engagement and community development in the age of algorithms. New Media & Society, 21(7), 1417-1443.