Randomized trials and their observational emulations: a framework for benchmarking and joint analysis

March, 2022

Abstract

A randomized trial and an analysis of observational data designed to emulate the trial sample observations separately, but have the same eligibility criteria, collect information on some shared baseline covariates, and compare the effects of the same treatments on the same outcomes. Treatment effect estimates from the trial and its emulation can be compared to benchmark observational analysis methods. In a simplified setting with complete adherence to the assigned treatment strategy and no loss-to-follow-up, we show that benchmarking relies on an exchangeability condition between the populations underlying the trial and its emulation, to account for differences in the distribution of covariates between them. When this exchangeability condition holds, and the usual conditions needed for the estimates from the trial and its emulation to have a causal interpretation also hold, we derive restrictions on the law of the observed data. When the data are compatible with the restrictions, joint analysis of the trial and its emulation is possible. When the data are incompatible with the restrictions, a discrepancy between (1) estimates based on extending inferences from the trial to the population underlying the emulation and (2) the emulation itself may reflect either inability to benchmark (e.g., due to selective participation into the trial) or a failure of the emulation (e.g., due to unmeasured confounding), but we cannot use the data to determine which is the case. Our analysis reveals how benchmarking attempts combine causal assumptions, data analysis methods, and substantive knowledge to examine the validity of observational analysis methods.