Quantile Forecasters with Conformal Calibration for Epidemic Uncertainty

Beyond Point Predictions: Quantile Forecasters with Conformal Calibration for Epidemic Uncertainty

Your Statistical Safety Net for Reliable Disease Forecasts

🎯 Introduction: The Critical Need for Uncertainty Quantification

Imagine your weather app predicts exactly 2.3 inches of rain tomorrow—but doesn’t tell you whether it could be anywhere from 0.5 to 4.0 inches. You’d probably carry an umbrella just in case, right? Yet for decades, many epidemic forecasting models have done exactly this: providing precise point predictions without reliable measures of uncertainty.

Enter Quantile Forecasters with Conformal Calibration—a sophisticated machine learning approach that doesn’t just predict what will happen, but how confident we should be in that prediction. Instead of saying “we expect 150 cases next week,” these models say “we’re 90% confident that cases will be between 80 and 250 next week.”

This shift from point predictions to prediction intervals is crucial for public health decision-making. During the COVID-19 pandemic, models that provided reliable uncertainty estimates helped officials distinguish between genuine emerging threats and normal statistical fluctuations [1-2]. Developed from foundational work in quantile regression [3] and modern conformal prediction theory [4], these models provide the statistical rigor needed for high-stakes epidemic forecasting.

🧮 Model Description: The Mathematics of Uncertainty

Quantile forecasters treat epidemic prediction as a distributional learning problem, where the goal is to estimate specific quantiles of the future case distribution rather than just the mean.

Core Quantile Regression Framework

For a given quantile level τ (tau) ∈ (0, 1), the model learns a function qₜ(τ) that satisfies:

P(Yₜ ≤ qₜ(τ) | Xₜ) = τ

Where:

  • Yₜ: True future incidence at time t
  • Xₜ: Input features (lagged cases, covariates, etc.)
  • qₜ(τ): Predicted τ-quantile of Yₜ

Pinball Loss Function

Quantile models are trained using the pinball loss (also called quantile loss):

Lₜ(u) = max(τ · u, (τ − 1) · u)
where u = Yₜ − qₜ(τ)

This asymmetric loss function penalizes under-predictions and over-predictions differently:

  • For τ = 0.5 (median): Lₜ(u) = |u| (absolute error)
  • For τ = 0.9: Under-predictions (u > 0) are penalized 9× more than over-predictions
  • For τ = 0.1: Over-predictions (u < 0) are penalized 9× more than under-predictions

Typical Implementation

Following your reference, a common implementation uses lagged features:

qₜ(τ) = fₜ(Xₜ; θₜ)
where Xₜ = [Yₜ₋₁, Yₜ₋₂, …, Yₜ₋L, covariatesₜ]

The model is trained by minimizing total pinball loss:

Loss = (1/T) · ∑₍t=1₎ᵀ Lₜ(Yₜ − qₜ(τ))

Using gradient-based optimization:

θ ← θ − η · ∇θ(Loss)

Where η (eta) is the learning rate.

Conformal Recalibration

Raw quantile predictions often don’t achieve exact coverage due to model misspecification or distributional shifts. Conformal prediction provides a distribution-free method to calibrate prediction intervals:

  1. Split data into training set and calibration set
  2. Train quantile model on training set
  3. Compute conformity scores on calibration set: Sᵢ = |Yᵢ − qᵢ(0.5)|
  4. Find (1−α) quantile of conformity scores: Ŝ = quantile(S, 1−α)
  5. Final prediction interval: [qₜ(0.5) − Ŝ, qₜ(0.5) + Ŝ]

For quantile-based conformal prediction, we directly adjust the quantile levels to achieve exact coverage [5].

📊 Key Parameter Definitions and Typical Values

Understanding these parameters is essential for implementing reliable quantile forecasters.

τQuantile level0.1, 0.5, 0.9 (common)τ = 0.9 means 90% of observations ≤ qₜ(0.9)
LLag window size7 – 52 time pointsLonger L = more historical context
TTraining period100 – 1000 daysLonger T = better quantile estimation
HForecast horizon1 – 4 weeksShorter H = more reliable intervals
ηLearning rate0.001 – 0.1Smaller η = more stable training
KTraining epochs50 – 500More epochs = better convergence
αSignificance level0.1 (for 90% PI)Smaller α = wider prediction intervals

Common Quantile Combinations

  • 90% prediction interval: τ = 0.1 and τ = 0.9
  • 80% prediction interval: τ = 0.2 and τ = 0.8
  • Median prediction: τ = 0.5

The prediction interval width is qₜ(0.9) − qₜ(0.1), which naturally adapts to changing uncertainty levels—wider during volatile periods, narrower during stable periods.

Coverage Calibration Metrics

  • Marginal coverage: Proportion of true values within prediction intervals
  • Conditional coverage: Coverage within specific subpopulations or time periods
  • Interval width: Average width of prediction intervals (trade-off with coverage)

Target: Marginal coverage ≥ 1−α with minimal interval width.

⚠️ Assumptions and Applicability: When Quantile Forecasters Work Best

Quantile forecasters with conformal calibration are powerful but rely on specific conditions for optimal performance.

✅ Ideal Applications

  • Operational forecasting: When decision-makers need reliable uncertainty estimates
  • High-stakes decisions: Resource allocation, intervention timing, public communication
  • Non-stationary time series: Where uncertainty varies over time (epidemic waves vs. troughs)
  • Model ensembles: Combining multiple models with quantile aggregation
  • Regulatory requirements: When statistical guarantees are required for model deployment

❌ Limitations and Challenges

  • Small sample sizes: Conformal calibration requires sufficient calibration data (typically >30 points)
  • Distributional shifts: Sudden changes in disease dynamics can break exchangeability assumptions
  • Computational overhead: Training multiple quantile models takes more time than point prediction
  • Interpretability trade-offs: Quantile models can be less interpretable than simple statistical models
  • Extreme quantiles: Very high/low quantiles (τ < 0.05 or τ > 0.95) require large datasets

💡 Pro Tip: Always validate both coverage and interval sharpness—a model that achieves 90% coverage with extremely wide intervals isn’t very useful for decision-making [6].

🚀 Model Extensions and Variants: Advanced Uncertainty Quantification

The basic quantile forecaster framework has inspired numerous sophisticated extensions for real-world epidemiological challenges.

1. Quantile Regression Forests

Replace neural networks with ensemble methods:

qₜ(τ) = quantile({tree_predictions}, τ)

Where each tree in a random forest provides a point prediction, and quantiles are computed across the ensemble [7].

2. Deep Quantile Regression

Use deep neural networks for complex feature relationships:

qₜ(τ) = f(Xₜ; θₜ)
f(·) = multi-layer neural network with quantile-specific outputs

Enables learning of complex, nonlinear quantile functions [8].

3. Distributional Forecasting

Instead of discrete quantiles, model the full distribution:

Yₜ ~ Normal(μₜ, σₜ) or Yₜ ~ NegativeBinomial(μₜ, φₜ)
μₜ, σₜ = f(Xₜ; θ)

More efficient than multiple quantile models when distributional assumptions hold [9].

4. Adaptive Conformal Prediction

Handle time-varying uncertainty with online calibration:

Ŝₜ = quantile(S₁, …, Sₜ₋₁, 1−αₜ)
αₜ = α · (1 + γ · (coverage_errorₜ₋₁))

Where the significance level adapts based on recent coverage performance [10].

5. Multivariate Quantile Forecasting

Generate joint prediction regions for multiple locations or outcomes:

P((Y₁,ₜ, Y₂,ₜ, …, Yₙ,ₜ) ∈ Cₜ) ≥ 1−α

Using copula methods or multivariate conformal prediction [11].

6. Causal Quantile Forecasting

Estimate quantile treatment effects for interventions:

qₜ(τ | do(intervention)) − qₜ(τ | no intervention)

Combines quantile regression with causal inference methods [12].

🎯 Conclusion: Embracing Uncertainty in Epidemic Intelligence

Quantile forecasters with conformal calibration represent a mature approach to epidemic forecasting that acknowledges a fundamental truth: all predictions are uncertain, and good science quantifies that uncertainty. By providing statistically rigorous prediction intervals rather than overconfident point estimates, these models enable more robust and reliable public health decision-making.

What makes this approach particularly valuable is its distribution-free nature—conformal prediction provides finite-sample coverage guarantees without requiring strong assumptions about the underlying data distribution. This is crucial in epidemiology, where disease dynamics can be highly non-stationary and model assumptions are often violated.

In an era where epidemic models directly influence policy decisions affecting millions of lives, the ability to say “we’re 90% confident that cases will be between X and Y” is not just statistically sound—it’s ethically essential. Whether you’re forecasting seasonal influenza, monitoring emerging variants, or evaluating intervention impacts, quantile forecasters provide your ML Epidemics Toolbox with the statistical foundation for honest, reliable epidemic intelligence.

The next time you see an epidemic forecast, don’t just ask “what do they predict?”—ask “how certain are they?” Because in epidemic preparedness, knowing what you don’t know is often more important than knowing what you do know.

📚 References

[1] Reich, N. G., McGowan, C. J., Yamana, T. K., Tushar, A., & Sheldon, D. (2019). Scenario-based forecasting for influenza and SARS-CoV-2. Harvard Data Science Review, 1(2).

[2] Ray, E. L., Wattanachit, N., Niemi, J., Kanji, A. H., House, K., Cramer, E. Y., … & Reich, N. G. (2022). Ensemble forecasts of coronavirus disease 2019 (COVID-19) in the US. Harvard Data Science Review, 4(1).

[3] Koenker, R., & Bassett, G. (1978). Regression quantiles. Econometrica, 46(1), 33–50. https://doi.org/10.2307/1913643

[4] Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. https://doi.org/10.1007/b106715

[5] Romano, Y., Patterson, E., & Candes, E. (2019). Conformalized quantile regression. Advances in Neural Information Processing Systems, 32, 3538–3548.

[6] Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437

[7] Meinshausen, N. (2006). Quantile regression forests. Journal of Machine Learning Research, 7, 983–999. https://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf

[8] Takeuchi, I., Le, Q. V., Sears, T. D., & Smola, A. J. (2006). Nonparametric quantile estimation. Journal of Machine Learning Research, 7, 1231–1264. https://www.jmlr.org/papers/volume7/takeuchi06a/takeuchi06a.pdf

[9] Salinas, D., Flunkert, V., Gasthaus, J., & Januschowski, T. (2020). DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3), 1181–1191. https://doi.org/10.1016/j.ijforecast.2019.07.001

[10] Gibbs, I., Candes, E., & Sesia, M. (2021). Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34, 1660–1672.

[11] Gupta, C., Kuchibhotla, A. K., & Ramdas, A. (2022). Nested conformal prediction for multivariate time series. Journal of Machine Learning Research, 23(1), 1–43.

[12] Chernozhukov, V., Fernández-Val, I., & Melly, B. (2013). Inference on counterfactual distributions. Econometrica, 81(6), 2205–2268.