empirical-methods

Homepage for 17-803 "Empirical Methods" at Carnegie Mellon University


Project maintained by bvasiles Hosted on GitHub Pages — Theme by mattgraham

Empirical Methods (thanks for the promo, @JoshQuicksall!)

This is the Spring 2024 offering of this course. For older versions, see here: Fall 2022Spring 2021Fall 2018.

Overview

Empirical methods play a key role in the design and evaluation of tools and technologies, and in testing the social and technical theories they embody. No matter what your research area is, chances are you will be conducting some empirical studies as part of your work. Are you looking to evaluate a new algorithm? New tool? Analyze (big) data? Understand what challenges practitioners face in some domain?

This course is a survey of empirical methods designed primarily for computer science PhD students, that teaches you how to go about each of these activities in a principled and rigorous way. You will learn about and get hands-on experience with a core of qualitative and quantitative empirical research methods, including interviews, qualitative coding, survey design, and many of the most useful statistical analyses of (large-scale) data, such as various forms of regression, time series analysis, and causal inference. And you will learn how to design valid studies applying and combining these methods.

There will be extensive reading with occasional student presentations about the reading in class, homework assignments, and a semester-long research project for which students must prepare in-class kickoff and final presentations as well as a final report.

After completing this course, you will:

As a side effect, this course helps you develop a healthy dose of skepticism towards scientific results in general. Does the study design really allow the authors to make certain claims? Does the analysis technique? Is the evidence provided as strong as it could be? Are there fundamental flaws and threats to validity?

Coordinates

Course Syllabus and Policies

The syllabus covers course overview and objectives, evaluation, time management, late work policy, and collaboration policy.

Learning Goals

The learning goals describe what I want students to know or be able to do by the end of the semester. I evaluate whether learning goals have been achieved through assignments, written project reports, and in-class presentations.

Schedule

Below is a preliminary schedule for Spring 2024. Each link points to a dedicated page with materials and more details. All videos are published on this YouTube channel.

Note: The schedule is subject to change and will be updated as the semester progresses.

Date Topic Notes
Tue, Jan 16 Introduction slides
Thu, Jan 18 Formulating research questions slidesvideo
Tue, Jan 23 The role of theory slidesvideo
Thu, Jan 25 Literature review slidesvideo
Tue, Jan 30 Exemplar interview papers slides
Thu, Feb 1 No class (Bogdan traveling)  
Tue, Feb 6 Conducting interviews slidesvideo
Thu, Feb 8 Qualitative data analysis slidesvideo
Tue, Feb 13 In-class activity: qualitative coding & thematic analysis slides
Thu, Feb 15 Qualitative analysis in the age of LLMs slides
Tue, Feb 20 Project proposal presentations slides
Thu, Feb 22 Types of errors in probability survey research slides
Tue, Feb 27 Questionnaire design and multi-item scales slides
Thu, Feb 29 Experimental design slides
Tue, Mar 5 Spring break, no class  
Thu, Mar 7 Spring break, no class  
Tue, Mar 12 Example experimental papers slides
Thu, Mar 14 Interaction effects and power analysis slides
Tue, Mar 19 Regression modeling diagnostics slides
Thu, Mar 21 Simpson’s paradox, exemplar papers, in-class activity  
Tue, Mar 26 Interrupted time series design  
Thu, Mar 28 In-class activity: interrupted time series analysis  
Tue, Apr 2 Social network analysis (part I)  
Thu, Apr 4 Social network analysis (part II)  
Tue, Apr 9 Diff-in-diff + CausalImpact  
Thu, Apr 11 Carnival, no class  
Tue, Apr 16 Research vs researcher  
Thu, Apr 18 Agree to disagree  
Tue, Apr 23 Final presentations (part I)  
Thu, Apr 25 Final presentations (part II)