How do you infer causality from natural experiments and quasi-experiments?

1 What are natural experiments?

Natural experiments are situations where an external factor, such as a policy change, a natural disaster, or a historical event, creates a random or quasi-random assignment of treatment and control groups. For example, the 2016 US presidential election was a natural experiment that affected the mental health of many Americans. By comparing the changes in mental health outcomes before and after the election across different groups of voters, you can estimate the causal effect of the election on mental health.

2 What are quasi-experiments?

Quasi-experiments are situations where you can create or exploit a non-random assignment of treatment and control groups based on some criteria, such as age, location, or time. For example, you can use a regression discontinuity design to compare the outcomes of students who barely passed or failed a standardized test, assuming that the test score is the only factor that determines their treatment status. By controlling for other confounding variables, you can estimate the causal effect of passing or failing the test on the outcomes.

3 How do you analyze natural and quasi-experiments?

In order to analyze natural and quasi-experiments, you need to employ statistical methods that can account for potential discrepancies between the treatment and control groups, and isolate the effect of the treatment from other factors. Difference-in-differences is a common method that evaluates changes in outcomes between the treatment and control groups before and after the intervention, assuming that the trends in both groups are parallel in the absence of the intervention. Matching pairs each treated unit with a similar untreated unit based on observable characteristics, then calculates the average difference in outcomes between them. Instrumental variables utilize a variable that affects the treatment status but not the outcome as an instrument to estimate the effect of the treatment on the outcome.

4 What are the advantages and disadvantages of natural and quasi-experiments?

Natural and quasi-experiments have some advantages and disadvantages compared to controlled experiments. For instance, they can overcome ethical and practical limitations, such as consent, cost, or feasibility. Moreover, they can capture real-world settings and behaviors, increasing the external validity and generalizability of the results. Additionally, they can provide new insights and opportunities for causal inference that would otherwise be unavailable. On the other hand, they can suffer from selection bias, confounding, or reverse causality, which can invalidate or weaken the causal claims. Furthermore, they can be sensitive to the choice of methods, assumptions, and data quality, and difficult to replicate or verify. Finally, they are subject to alternative interpretations or criticisms.

5 How do you improve your causal inference skills?

Causal inference is a complex and evolving field that requires both theoretical and practical knowledge. To hone your skills, you should become familiar with the fundamentals of causal inference, such as potential outcomes, counterfactuals, and identification strategies. Additionally, you should read and critique papers that use natural and quasi-experiments, evaluating their validity, reliability, and relevance. Furthermore, it is important to practice applying different methods and tools to analyze natural and quasi-experiments while communicating your findings and limitations.

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?