Comprehensive Evaluation of Machine Learning for Type 2 Diabetes Risk Prediction: Large-Scale External Validation and Fairness Analysis
Internally validated diabetes models lose discrimination on 1.28M external records, and adults over 60 lose the most.
The discovery in one figure
Swipe to see the whole figure
The paper in five minutes
A disease-risk model that shines on its home dataset can still mislead in the real world. We trained a standard diabetes risk model the careful way, with strict cross-validation and tuned hyperparameters, and then did what most papers skip: tested it on 1.28 million people from a completely different national survey. Overall accuracy dropped about ten percent, and the drop was not shared equally. For people over 60 the model degrades far more than for the young, a fairness failure that the single headline number hides.
The research question
Clinical risk models are usually reported with internal validation only. Do the numbers survive a different population, survey instrument, and label definition?
How it works
XGBoost and baselines developed on NHANES (n = 15,685) under strict nested cross-validation, then externally validated on BRFSS (n = 1,285,783) with DeLong CIs, calibration analysis, SHAP interpretability, and a subgroup fairness audit. TRIPOD-AI reporting.
- 01Develop carefullyNHANES cohort (15,685) · nested cross-validation · Bayesian tuning
- 02ExplainSHAP attribution over eight non-laboratory predictors
- 03Externally validateBRFSS, 1,285,783 records: different survey, different population
- 04Audit subgroupsage, sex, BMI, with DeLong confidence intervals
- 05Report to standardTRIPOD-AI checklist
Experimental results
External AUC drops from 0.794 (95% CI 0.788 to 0.800) to 0.717, a 9.7% relative decline. The elderly subgroup degrades far more than the young (AUC 0.607 vs 0.742, gap 0.135): aggregate accuracy hides an age-fairness failure. The Brier score is 0.123, with risk overestimated at high thresholds.
0.7940.7170.7420.607External validation costs about 10% overall; adults over 60 lose far more (gap 0.135, p < 0.001).
- 0.794 → 0.717internal → external AUC (−9.7%)
- 0.135age-group AUC gap (18-39 vs 60+)
- 1,285,783external validation records
Stated honestlyThe model uses eight non-laboratory predictors. Diabetes is defined from laboratory measures, self-report and medication in NHANES, but from self-reported diagnosis in BRFSS.
What this changes
The foundation result of the research program: benchmark performance is not deployment performance, and the failure is structured, not random.
Citation
Pall, R. S., Yadav, S., Bhalerao, S., Sahu, S., Ahluwalia, R., & Awadhiya, B. (2026). Comprehensive evaluation of machine learning for type 2 diabetes risk prediction: Large-scale external validation and fairness analysis. CIPHER-2026. IEEE. https://doi.org/10.1109/CIPHER70417.2026.11523789
Published work. Use the citation above. Reach me at rajveerpall04@gmail.com.