empirical-methods
Homepage for 17-803 "Empirical Methods" at Carnegie Mellon University
Project maintained by bvasiles Hosted on GitHub Pages — Theme by mattgraham
L8 & L9: Survey Design (pdf, video - part I, video - part II)
In this lecture series we talk about how to conduct effective probability sample surveys. We discuss the fundamental requirements that must be met if one wants to generalize results with statistical confidence from the few who are surveyed to the many they are selected to represent.
In particular, we cover four main types of error that surveyors need to try to minimize in order to improve the survey estimates:
- Coverage Error occurs when the list from which sample members are drawn does not accurately represent the population on the characteristic(s) one wants to estimate with the survey data
- Sampling Error exists anytime we decide to survey only some, rather than all, members of the sample frame.
- Nonresponse Error occurs when those who do not respond are different from those who do respond in a way that influences the estimate.
- Measurement Error occurs when respondents are unable or unwilling to provide accurate answers, which can be due to poor question design, survey mode effects, interviewer and respondent behavior, or data collection mistakes.
Readings
Dillman, D., Smyth, J. D., & Christian, L. M. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method (4th ed.). Hoboken, NJ: Wiley.
- Chapter 1: Sample Surveys in our Electronic World
- Chapter 2: Reducing People’s Reluctance to Respond to Surveys
- Chapter 4: The Fundamentals of Writing Questions
- Chapter 5: How to Write Open and Closed Ended Questions
Main source for the lecture.
Hof, M. (2012). Questionnaire Evaluation with Factor Analysis and Cronbach’s Alpha. Student project. Seminar in Methodology and Statistics. University of Groningen
Yong, A. G., & Pearce, S. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutorials in quantitative methods for psychology, 9(2), 79-94.
Good examples of how to assess the reliability and validity of a survey scale (factor analysis, Cronbach’s Alpha).
Cairns, P. (2019). Doing better statistics in human-computer interaction. Cambridge University Press.
- Chapter 15: What Makes a Good Likert Item?
- Chapter 16: The Meaning of Factors
- Chapter 17: Unreliable Reliability: The Problem of Cronbach’s Alpha
- Ch18: Tests for Questionnaires
Lots of detail on Likert items and scales. Examples of what can go wrong when questions are worded or answers options are given even slightly differently.