empirical-methods

Homepage for 17-803 "Empirical Methods" at Carnegie Mellon University

---

Project maintained by bvasiles Hosted on GitHub Pages — Theme by mattgraham

L12: Experimental Design Part II - Counterbalancing (pdf, video)

This lecture is the second part of a series on designing experiments. We discussed Latin Squares and counterbalancing.

We also discussed and critiqued examples of studies using experiments, including a true (randomized) experiment in the Tomkins et al study of double blind reviewing at the Conference on Web Search and Data Mining.

Lecture Readings

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Publishing.

The discussion of cause as an inus condition – “insufficient but nonredundant part of an unnecessary but sufficient condition” – follows Chapter 1 from the book (Experiments and generalized causal inference).

---

Tomkins, A., Zhang, M., & Heavlin, W. D. (2017). Single versus double blind reviewing at WSDM 2017. arXiv preprint arXiv:1702.00502

A nice example of a randomized experiment carried out to assess the impact of single vs double-blind reviewing of conference papers. The paper reports that:

• “Reviewers in the single-blind condition […] preferentially bid for papers from top universities and companies.”
• “Single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors [odds multiplier 1.64], top universities [1.58], and top companies [2.10].”