Homepage for 17-803 "Empirical Methods" at Carnegie Mellon University

Project maintained by bvasiles Hosted on GitHub Pages — Theme by mattgraham

Empirical Methods (thanks for the promo, @JoshQuicksall!)

This is the Spring 2021 offering of this course. For older versions, see here: Fall 2018.


Empirical methods play a key role in the design and evaluation of tools and technologies, and in testing the social and technical theories they embody. No matter what your research area is, chances are you will be conducing some empirical studies as part of your work. Are you looking to evaluate a new algorithm? New tool? Analyze (big) data? Understand what challenges practitioners face in some domain?

This course is a survey of empirical methods designed for computer science PhD students, that teaches you how to go about each of these activities in a principled and rigorous way. You will learn about and get hands-on experience with a core of qualitative and quantitative empirical research methods, including interviews, qualitative coding, survey design, and many of the most useful statistical analyses of (large-scale) data, such as various forms of regression, time series analysis, and causal inference. And you will learn how to design valid studies applying and combining these methods.

There will be extensive reading with occasional student presentations about the reading in class, homework assignments, and a semester-long research project for which students must prepare in-class kickoff and final presentations as well as a final report.

After completing this course, you will:

As a side effect, this course helps you develop a healthy dose of skepticism towards scientific results in general. Does the study design really allow the authors to make certain claims? Does the analysis technique? Is the evidence provided as strong as it could be? Are there fundamental flaws and threats to validity?


Course Syllabus and Policies

The syllabus covers course overview and objectives, evaluation, time management, late work policy, and collaboration policy.

Learning Goals

The learning goals describe what I want students to know or be able to do by the end of the semester. I evaluate whether learning goals have been achieved through assignments, written project reports, and in-class presentations.


Below is a preliminary schedule for Spring 2021. Each link points to a dedicated page with materials and more details. Note: The schedule is subject to change and will be updated as the semester progresses.

Date Topic Notes
Tue, Feb 2 Introduction slidesvideo
Thu, Feb 4 Formulating research questions slidesvideo
Tue, Feb 9 The role of theory slidesvideo
Thu, Feb 11 Literature review slidesvideo
Tue, Feb 16 Conducting interviews slidesvideo
Thu, Feb 18 Exemplar interview papers slidesvideo
Tue, Feb 23 Break Day; No Classes  
Thu, Feb 25 Qualitative data analysis slidesvideo
Tue, Mar 2 Survey design (part I) slidesvideo
Thu, Mar 4 Survey design (part II) slidesvideo
Tue, Mar 9 Project proposal presentations  
Thu, Mar 11 Numbers and nonsense slidesvideo
Tue, Mar 16 Causal relationships slidesvideo
Thu, Mar 18 Experimental design slidesvideo
Tue, Mar 23 Intro to regression modeling slidesvideo
Thu, Mar 25 Linear regression diagnostics slidesvideo
Tue, Mar 30 Standardized coefficients + Mixed-effects slides1slides2video
Thu, Apr 1 Exemplar regression papers video
Tue, Apr 6 Simpson’s paradox + Mixed-effects slidesvideo
Thu, Apr 8 Interrupted time series design slidesvideo
Tue, Apr 13 Diff-in-diff + CausalImpact slidesvideo
Thu, Apr 15 Spring Carnival; No Classes  
Tue, Apr 20 Mixed-methods designs slidesvideo
Thu, Apr 22 Stepping up your graphics, slide design, and writing slidesvideo
Tue, Apr 27 Agree to disagree  
Thu, Apr 29 Agree to disagree / Retrospective  
Tue, May 4 Final presentations (part I)  
Thu, May 6 Final presentations (part II)