Despite the availability of granular measures of advertising exposures and outcomes at the individual level, measuring the causal effect of digital advertising campaigns is very challenging. A recent literature (Lewis and Rao 2015) suggests that observational methods such as matching and regression—given what we know apriori about digital advertising—are likely to suffer from selection bias. In principal, this concern can be addressed using randomized controlled trials (RCTs). In practice, however, few online ad campaigns rely on RCTs. In this paper, we empirically test the prediction that observational methods are likely to yield biased estimates of the causal effects of online advertising. This analysis is of particular interest because there have been enormous recent improvements in observational methods for causal inference (Imbens and Rubin 2015). Using data from 15 US advertising lift studies at Facebook comprising 500 million user-study observations and 1.6 billion total impressions, we contrast the experimental results to those obtained from a variety of observational methods. We show that observational methods often fail to produce the same results as the randomized experiments, even after conditioning on information from thousands of behavioral variables and using non-linear models. Our findings suggest that commonly-used approaches in industry fail to measure the true effect of advertising accurately