T2D Risk Screener

A calibrated diabetes pre-screening tool built on Korean national health data. It flags patients likely to have undiagnosed Type 2 diabetes so a fasting plasma glucose (FPG) test gets ordered before symptoms show up.

At a Glance

Domain: Clinical/healthcare ML, trained on Korean NHIS public health examination data (~1M patients)
Role: Team lead of a 4-person course project, Spring 2026; I built the data pipeline, modeling, and deployment
Stack: Python, CatBoost, scikit-learn, SHAP, FastAPI, Pydantic v2, vanilla JS + Plotly
Headline metric: ROC-AUC 0.816 | sensitivity ≥ 85% at operating threshold
Status: Complete (live demo at tysonjohnson.dev/t2d)

Problem / Context

An estimated 30 to 40% of people with Type 2 diabetes are undiagnosed at any given time. South Korea's National Health Insurance Service (NHIS) runs annual health exams for roughly 1 million adult subscribers, collecting routine vitals, body composition, and optional lab panels. Most of that data never triggers a diabetes referral unless a clinician happens to flag a single abnormal value. I wanted to see whether the full signal already sitting in a routine exam (waist-to-height ratio, blood pressure, liver enzymes, lipid ratios, kidney markers) could surface the patients most likely to have undiagnosed diabetes and route them to a confirmatory FPG or HbA1c test.

Approach

I ingested the publicly released 2024 NHIS General Health Examination CSV (~91 MB, ~1M rows) and engineered 40 features from the raw exam fields: derived body-composition indices (BMI, waist-to-height ratio, weight-adjusted waist index), hemodynamic composites (pulse pressure, mean arterial pressure), liver-function ratios (AST/ALT), log-transforms of right-skewed lab values (GGT, triglycerides, liver enzymes), and full lipid-panel ratios. Binary missingness flags were added for each optional lab so the model could learn from absence as well as value.

The dataset is class-imbalanced: roughly 7.9% positive (~12:1 ratio). I used stratified 60/20/20 splits and applied inverse class-frequency weighting inside CatBoost rather than resampling, which keeps the calibration target honest.

I evaluated six candidate models on the same held-out test split: CatBoost, a PyTorch tabular neural net, an sklearn MLP, logistic regression, an RBF-kernel SVM, and a decision tree. CatBoost and the PyTorch tabular net finished effectively tied on ROC-AUC (0.817 vs. 0.816), but I picked CatBoost because it computes exact, fast per-prediction SHAP values natively and runs on CPU at negligible cost. Getting an equivalent explanation out of the neural net would have needed a slower, approximate surrogate (KernelSHAP), and a clinical screening tool needs a transparent reason behind every positive flag.

Raw CatBoost sigmoid outputs ranked well but were overconfident at low probabilities relative to the 7.9% base rate. I applied Platt scaling (a one-parameter logistic regression fit on validation-set logits) to bring the probabilities into alignment with the empirical positive rate. The operating threshold (0.066) was then tuned on the validation set to the lowest value achieving at least 85% sensitivity, reflecting the asymmetric clinical cost: a missed screen means a patient never gets an FPG ordered.

The serving layer is a FastAPI app backed by a committed model bundle (~1.6 MB: catboost_model.cbm, calibrator.pkl, metadata.json, global_shap.json). A vanilla-JS single-page frontend collects exam inputs, calls the API, and renders the calibrated probability, risk tier, per-prediction SHAP waterfall (via Plotly), and a matched clinical recommendation.

Results

Evaluated on the held-out 20% test slice (n ≈ 200k):

Metric	Value
ROC-AUC	0.816
PR-AUC (Average Precision)	0.273
Brier score	0.064
Operating threshold	0.066 (≥ 85% sensitivity)
NPV at operating threshold	0.98

The number I'm proudest of is NPV: 0.98 at the 0.066 threshold (114,435 true negatives vs. 2,318 false negatives on the test slice). When the tool calls someone low risk, it's right 98% of the time, and safely ruling people out is a screening tool's main job.

ROC comparison of six candidate models on the same test split. CatBoost and the PyTorch Tabular Net are effectively tied.

Global mean-|SHAP| feature importance across a 5,000-row test sample — Global mean-|SHAP| feature importance. Waist-to-height ratio, age, and lipid ratios dominate; liver enzyme log-transforms contribute meaningfully.

Confusion matrix at the operating threshold — Confusion matrix at the 0.066 operating threshold. High sensitivity is the explicit design target for a screening context.

Raw vs. Platt-calibrated reliability diagram — Reliability diagram before and after Platt scaling. Raw outputs cluster above the diagonal at low probabilities; calibrated outputs track the empirical rate.

Screener frontend with a sample prediction — The deployed frontend: calibrated probability, risk tier, SHAP waterfall, and clinical recommendation rendered in the browser.

Key Decisions / What I Learned

Keeping FPG out of the feature set was non-negotiable, since FPG is what defines the label (has_diabetes = FPG ≥ 126 mg/dL); including it would make the model a trivial re-thresholding of itself. The real engineering work was building a strong signal from indirect metabolic indicators alone.

Choosing CatBoost over the statistically tied PyTorch tabular net came down to what the product needed. The AUC gap was 0.001, but only CatBoost gave fast, exact, per-prediction explanations without a GPU, and a clinical score without an explanation attached is close to unusable.

Calibration mattered for the same reason. A model can rank patients correctly and still output a 0.4 when the true base rate is 7.9%, which erodes a clinician's trust the moment they look at the number. Platt scaling, one logistic regression fit on held-out logits, is a small addition that makes the displayed probability mean what it says.

Limitations

I want to be direct about what this project is and is not.

Single Korean cohort, single year. All training data comes from the 2024 NHIS random sample. Performance on other populations, years, or health systems is untested.
Label noise. FPG ≥ 126 mg/dL at one time point will misclassify some patients with known but poorly controlled T2D as "undiagnosed," because the NHIS public release does not include treatment history.
Selection bias in optional labs. Lipid and liver-enzyme panels were ordered at clinician discretion. Patients with missing labs are on average healthier than those who were tested, so the model treats lab absence as a mild healthy signal. Submissions with all lab fields blank will likely underestimate risk; the frontend surfaces a warning in that case.
Coarse age encoding. NHIS releases age only as 5-year bands; the model uses the band midpoint.
Not clinically validated. This is a research portfolio project. It is not a diagnostic tool and has not undergone prospective validation.

Tech Stack

Modeling: Python, CatBoost, scikit-learn (Platt calibration, stratified splits, evaluation)
Explainability: SHAP (native CatBoost exact SHAP)
API: FastAPI, Pydantic v2, Uvicorn
Frontend: Vanilla JS, Plotly (SHAP waterfall charts)
Data: Korean NHIS 2024 General Health Examination (data.go.kr)