Predictive Performance of International COVID-19 Mortality Forecasting Models
November, 2020
See Code - Global COVID-19 Forecast Comparison
https://github.com/pyliu47/covidcompare
Here we provide the codebase and analytical/graphing framework to collate and compare all historical model-verions of the below global COVID-19 forecasting models. We encourage use of data to assess the historical and ongoing performance of models, develop new analyses for assessing performance, and hope that they will spur the requisite conversation for improving the community of forecasting models:
- DELPHI-MIT (Delphi)
- Imperial College London (Imperial)
- The Institute for Health Metrics and Evaluation (IHME)
- The Los Alamos National Laboratory (LANL)
- SIKJalpha
- Youyang Gu (YYG)
Abstract
Forecasts and alternative scenarios of COVID-19 mortality have been critical inputs into a range of policies and decision-makers need information about predictive performance. We identified n=386 public COVID-19 forecasting models and included n=8 that were global in scope and provided public, date-versioned forecasts. For each, we examined the median absolute percent error (MAPE) compared to subsequently observed mortality trends, stratified by weeks of extrapolation, world region, and month of model estimation. Models were also assessed for ability to predict the timing of peak daily mortality. The MAPE among models released in July rose from 1.8% at one week of extrapolation to 24.6% at twelve weeks. The MAPE at six weeks were the highest in Sub-Saharan Africa (34.8%), and the lowest in high-income countries (6.3%). At the global level, several models had about 10% MAPE at six weeks, showing surprisingly good performance despite the complexities of modelling human behavioural responses and government interventions. The framework and publicly available codebase presented here (https://github.com/pyliu47/covidcompare) can be routinely used to compare predictions and evaluate predictive performance in an ongoing fashion.
