Doing Economics

2.5.2 Robustness Checks
After presenting the core results in a paper, it is time to turn to robustness
checks. There was a time when it was sufficient to present one or two tables
of empirical results to convince the reader that there was a “there” there;
times have changed, and as a consequence of the Credibility Revolution
(Angrist and Pischke 2010), which has led to greater emphasis being placed
on causal identification and inference, standards of evidence are
considerably higher than they were in the early to mid-2000s. Authors now
have to work hard to convince readers that their results were not cherry-
picked, which means that establishing the robustness of a finding involves
its own set of complications.
In many cases, the outcome we are interested in has more than one
measurement. “Welfare,” for instance, can be measured in a number of
ways: household income, household income per capita, household income
per adult equivalent, household consumption expenditures, household

---

consumption expenditures per capita, household consumption expenditures
per adult equivalent, subjective well-being of the respondent, and so forth.
If you have access to all seven of those measures of “welfare,” one first step
toward establishing that your result is robust might be simply to re-estimate
your core equation for each of those measures, showing that the result holds
across all of them.
Similarly, you may have different measures of the treatment variable. In
most randomized controlled trials (RCTs), there is one (and only one)
treatment variable (unless there are several treatment arms, and unless those
treatment arms are interacted). But with observational data, it might be
possible to look at different measures of the treatment variable. In the
contract farming literature, for example, one can look at whether a
household participates in contract farming (i.e., contract farming at the
extensive margin), but one could also look at the proportion of one’s crop
acreage that is under contract (i.e., contract farming at the intensive
margin).
Now imagine that you have those two measures for the treatment
variable, and the aforementioned seven measures for the outcome variable.
This allows estimating 14 different specifications of the core equation of
interest. If the finding holds for each one of those specifications, that goes a
long way toward establishing that the finding is robust.
One can also check for robustness by conducting placebo and
falsification tests. In the former case, a “fake” treatment (i.e., a variable that
is correlated with the treatment, but which presumably does not cause the
outcome) is used in lieu of the actual treatment. In the latter case, a “fake”
outcome (i.e., a variable that is correlated with the outcome, but which
presumably is not caused by the treatment) is used in lieu of the actual
outcome. In both cases, robustness comes from the lack of a statistically
significant finding, since a statistically significant finding hints at the fact
that the core results might be spurious. In difference-in-differences studies
—a methodology where the frontier has been evolving rapidly over the past
few years—one should analyze trends.
Yet another kind of robustness check comes in the form of looking at
different estimators. Most empirical economics articles, for instance, rely on
some linear, fully parametric regression. If the treatment is continuous, it
might be useful to estimate specifications that allow for a more flexible

---

functional form (e.g., a restricted cubic spline), which would allow one to
determine whether the relationship between y and D is generally monotonic.
Very often, robustness checks of this kind are where modest methodological
contributions—a paper’s third contribution, as listed in the introduction—
come from.