PaperlayerView exact revision on arXiv

Source-linked AI summary

FairGlucose: A CGM Fairness Benchmark Reveals Subgroup Disparities Hidden in Population-Level Validation

Junjie Luo, Xuzhe Zhi, Rui Han, Abhimanyu Kumbara, Anand K. Iyer, Mansur E. Shomali, Ritu Agarwal, Guodong Gordon Gao

arXiv:2608.18296v1cs.CYcs.AI

TL;DR

Whether CGM-based AI forecasting is equitable across demographic subgroups remains insufficiently evaluated. FairGlucose addresses this gap with a balanced cohort and fairness-aware benchmark of 33 models, finding stable population-level metrics alongside substantial subgroup disparities across models.

  • Problem

    Whether CGM-based AI forecasting accuracy is equitable across demographic subgroups remains unresolved, while existing benchmarks provide limited balanced subgroup evidence.

  • Method

    FairGlucose constructs a balanced 300-patient cohort across 12 demographic strata and benchmarks 33 models using subgroup-disaggregated, event-aware forecasting metrics.

  • Results

    Aggregate external-validation metrics near 1.0 coexist with subgroup ratios ranging from 0.8 to 1.4 across all 33 evaluated models.

  • Takeaways & Limitations

    Population-level validation alone is insufficient for assessing CGM forecasting equity, supporting subgroup-disaggregated reporting before clinical deployment.

  • Takeaways & Limitations

    The US-based mobile health dataset and 12-stratum design do not establish generalization across other populations, regions, devices, or fairness-relevant factors.

Abstract

from arXiv · show

As CGM-based AI tools approach clinical deployment, whether their accuracy is equitable across patient demographics remains insufficiently tested. To enable this evaluation, we constructed FairGlucose, a 300-patient CGM cohort balanced across 12 demographic strata (age x gender x type 1/type 2 diabetes), with 132,480 forecasting samples and 3,945 unique behavioral events (meals, exercise, medication) logged by 81 patients. Benchmarking 33 models across four families on 2-hour glucose forecasting, we find that population-level external validation can conceal substantial subgroup disparities. Aggregate out-of-distribution metrics appear stable (approximately 1.0), yet subgroup-level ratios range from 0.8 to 1.4, with T1D patients showing 6 mg/dL higher prediction error than T2D (p < 0.001). This disparity persists across all 33 models, suggesting a property of the prediction task rather than any single architecture. Further analysis shows that subgroup performance gaps align with the proportion of clinically hard cases, and that input-length sensitivity varies across demographics, motivating personalized configurations. Frontier LLMs underperform specialized neural models by 1-6 mg/dL; behavioral events contribute negligibly (approximately 0.1 mg/dL) even under oracle event access. These findings establish that population-level validation alone is insufficient for equity assessment of digital health AI, motivating subgroup-disaggregated reporting as a default standard.

Introduction

FairGlucose addresses the largely absent fairness evaluation of CGM forecasting by introducing a balanced benchmark and clinically meaningful accuracy and safety tasks. Its results show that aggregate validation can conceal persistent subgroup disparities across models, while demographic sensitivity and limited event-feature gains motivate disaggregated evaluation.

  • Motivation: CGM fairness evaluation remains virtually absent, despite evidence that high average clinical-AI performance can conceal significant subgroup underperformance.Existing CGM benchmarks are demographically imbalanced and generally lack fairness metrics.
  • Benchmark: 300 patients were balanced across 12 demographic strata, yielding 132,480 forecasting samples and 3,945 timestamped behavioral events logged by 81 patients.FairGlucose includes uniform data quality across subgroups and held-in/held-out evaluation splits.
  • Evaluation tasks: rMSE measures 2-hour-ahead forecasting accuracy, while Clarke A reports predictions in the most clinically accurate zone for safety assessment.Clarke A represents predictions with no clinical errors under the stated criteria.
  • Key findings: ≈1.0 population-level OD/ID ratios contrasted with 0.8–1.4 subgroup-level ratios, while T1D patients consistently showed higher prediction error than T2D patients.Performance also varied significantly across age-gender intersections.
  • Key findings: 33 evaluated models spanning four paradigms showed persistent subgroup disparities, indicating that the gap is a task-level property rather than an architecture-level artifact.Subgroup performance gaps aligned with the proportion of clinically hard cases, and input-length sensitivity varied by demographic.
  • Model benchmarking: 1–6 mg/dL lower performance characterized frontier LLMs relative to specialized neural models, while behavioral events improved top-model performance by only ∼0.1 mg/dL under oracle access.The event-feature result suggests modern forecasters implicitly capture event-driven dynamics from CGM signal alone.

Results

Results show strong overall forecasting generalization and competitive specialized models, yet population-level validation conceals substantial demographic disparities. Subgroup performance varies with diabetes type, intersectional demographics, input history, and prediction difficulty, while behavioral events contribute limited information.

  • Cohort characteristics: T1D patients had higher glycemic variability and lower time-in-range than T2D patients despite balanced sampling.T1D CV was ∼0.29 with TIR ∼65%, versus T2D CV ∼0.23 and TIR ∼69%.
  • Behavioral events: 22,239 of 132,480 samples (16.8%) contained at least one event within their window, with event density varying by age and diabetes type.Patients aged 65+ logged 22% fewer events, while T2D patients logged 12% more events than T1D patients.
  • Model performance: NS-Transformer ranked first among forecasting models with rMSE=25.6 mg/dL and Clarke A=76.4%.Transformer, TimeXer, and PatchTST followed closely, while LightGBM achieved rMSE=26.0 mg/dL and Clarke A=76.8%.
  • Model performance: Frontier LLMs trailed top neural models by 1–6 mg/dL in rMSE and achieved Clarke A rates of 70–75%.TimeGPT was competitive at rMSE=26.6 mg/dL, whereas Chronos2 reached rMSE=39.7.
  • Subgroup disparities: 6.3 mg/dL higher rMSE occurred for T1D than T2D patients, with 95% bootstrap CI [4.3, 8.2] and permutation p < 0.001.Gender and age marginal differences were not significant: ∆M−F = −0.19 mg/dL, p = 0.86; ∆18–39 vs 65+ = 1.85 mg/dL, p = 0.16.
  • Subgroup disparities: OD/ID ratios near 1.0 across models masked subgroup ratios ranging from 0.8 to 1.4.Gender effects were conditional on diabetes type and age, including ∆F−M = +3.3 mg/dL among younger T1D patients and −3.4 mg/dL among middle-aged T2D patients.
  • Personalized configurations: Input-length performance varied across subgroups: shorter windows nearly matched the 288-step baseline for younger patients, whereas older T2D patients degraded more with truncation.PatchTST, TFT, and N-HiTS showed the largest gains as input length increased, supporting subgroup-adaptive input histories.
  • Prediction difficulty: Young male T1D and middle-aged female T1D patients had the highest proportions of hard cases, while older T2D males had the highest proportion of easy cases.Top neural models improved more on easy cases, with easy ratios 0.77–0.89 versus hard ratios 0.89–0.92.

Discussion

Population-level validation can falsely reassure about CGM forecasting equity: aggregate out-of-distribution performance appears stable while subgroup disparities persist across all 33 evaluated models. FairGlucose therefore supports subgroup-disaggregated evaluation, personalized configurations, and broader validation while highlighting important data and coverage limitations.

  • Equity implications: 0.8 to 1.4 subgroup-level ratios coexist with approximately 1.0 aggregate out-of-distribution performance across 12 demographic strata.This disparity persists across all 33 evaluated models spanning four paradigms, suggesting it is not limited to one architecture.
  • Equity implications: 31.9 mg/dL versus 19.9 mg/dL in-distribution error distinguishes young T1D females from the best-served subgroup, while T2D females aged 40–64 have the worst generalization at OD/ID = 1.42.Accuracy and generalization identify different vulnerable subgroups, so population-level metrics alone may systematically underserve some patients.
  • Benchmark contribution: 300 patients across 12 balanced strata and a 33-model reference leaderboard enable reproducible fairness research in glucose forecasting.FairGlucose also includes behavioral event annotations and is intended to support more equitable diabetes care.
  • Personalization: Longer CGM histories stabilize prediction for older T2D patients, whereas younger T1D patients tolerate short input windows with minimal accuracy loss.These subgroup differences motivate personalized or subgroup-adaptive input-length strategies rather than fixed-length configurations.
  • Limitations: Three major limitations are app-logged event annotations, a US-based mobile health platform, and stratification limited to age, gender, and diabetes type.The cohort does not capture passive sensing modalities or fairness-relevant factors such as race, socioeconomic status, and comorbidities, and broader validation across populations, regions, and devices is needed.
  • Future directions: Four future directions are multi-site cohorts, personalized event-aware forecasting, fairness-aware architectures, and hybrid or mechanistic models.The standardized protocols may inform regulatory approval and clinical deployment decisions.

Methods

FairGlucose uses a quality-controlled, demographically balanced CGM cohort with structured forecasting samples and behavioral-event annotations. Evaluation combines subgroup fairness, input-length sensitivity, instance difficulty, event-aware comparisons, and patient-level uncertainty procedures across 33 models.

  • Sample construction: Each qualified patient-day required a complete 24-hour CGM trace with adjacent continuity, yielding hourly pairs with 288-value inputs and 96-value outputs.The primary analysis evaluates the first 2 hours, or 24 output steps, while the full window supports horizons up to 8 hours.
  • Cohort and data partitioning: 300 patients across 12 balanced demographic subgroups contributed 5,520 patient-days and 132,480 input-output pairs in non-overlapping train, validation, test-id, and test-od partitions.Each instance includes static covariates for age group, gender, and diabetes type, plus time-of-day indicators.
  • Event annotation: 3,945 unique behavioral events were logged by 81 of 300 patients, including 2,809 medication, 648 meal, and 488 exercise events.Annotations include nutritional details for meals, duration and intensity for exercise, and dosage and timing for medication.
  • Statistical inference: B = 1,000 patient-level bootstrap resamples provide confidence intervals, while two-sided patient-level permutation tests use N = 10,000 permutations for specified rMSE differences.Patient-level aggregation addresses dependence from overlapping sliding windows.
  • Fairness evaluation: The fairness framework evaluates subgroup rMSE disparities using rMSE Spread and the Gini Index, alongside input-length sensitivity, instance-level difficulty, and event-aware performance.Input lengths are L ∈ {7, 37, 73, 145, 288}; hard and easy samples are defined by model-vote quartiles, and event comparisons account for post-prediction event availability.
  • Model benchmarking: The benchmark compares 33 models spanning four families, including statistical baselines such as Naive and ARIMA.The Naive method predicts the most recent value, while ARIMA captures temporal structure.

Dataset Statistics and Details

FairGlucose characterizes subgroup CGM dynamics using sequence-level clinical and signal statistics, revealing multidimensional structure, correlated features, and distributional differences across strata. The benchmark supports reproducible evaluation through controlled data access and an API-based model-to-data server.

  • Sequence-level statistics: 288 CGM measurements spanning a full day formed each sequence-level calculation, which was then averaged across subgroup sequences.Metrics included mean, median, standard deviation, extrema, CV, IQR, selected percentiles, TIR%, MAGE, rates of change, peaks, valleys, trend length, and entropy.
  • Subgroup structure: Composite age × gender × diabetes-type coloring produced the most pronounced t-SNE cluster separation, whereas univariate colorings substantially overlapped.The result supports multidimensional subgroup structure in CGM dynamics that univariate stratification would miss.
  • Feature relationships: Mean glucose strongly correlated positively with TAR and inversely with TIR, while peaks and slope captured weaker but distinct temporal signatures.Pairplots examined feature distributions, pairwise correlations, and cluster separability across strata and patient attributes.
  • Distributional differences: Younger adults aged 18–39 showed wider dispersion in mean glucose and TIR, while some older T2D strata had narrower distributions.These cross-stratum differences indicate heterogeneous feature landscapes relevant to prediction consistency and fairness-aware modeling.
  • Access and reproducibility: The API evaluation server returns standardized rMSE, MAE, Clarke Error Grid zone counts, and subgroup-disaggregated metrics without exposing raw data.Researchers can alternatively request de-identified cohort access under a DUA, while metadata, strata definitions, splits, and the 33-model leaderboard are planned for open release.

Model Evaluation Details

Across 30-minute, 1-hour, and 2-hour forecasts, attention-based neural models lead accuracy and robustness evaluations, while foundation models underperform and errors rise with longer horizons. Subgroup analyses reveal persistent intersectional disparities, input-length sensitivity differences, and a link between performance gaps and hard-case prevalence.

  • Forecasting performance: 30-minute evaluations favor attention-based models, with NS-Transformer, PatchTST, and N-HiTS among the strongest performers across error and clinical metrics.TFT is also competitive, while LightGBM and CatBoost show greater variability in TIR prediction across subgroups.
  • Forecasting performance: Foundation models consistently underperform across 30-minute evaluations, often approaching or falling below the naive baseline in rMSE and TIR error.The evaluated models include GPT-5.1, Gemini 3 Flash, and TimeGPT.
  • Forecasting performance: 25.6 mg/dL is NS-Transformer’s best overall 2-hour rMSE, followed by Transformer at 25.7 and TimeXer and PatchTST at 25.8.Performance deteriorates across models as the forecasting horizon extends, although NS-Transformer remains strongest overall.
  • Subgroup fairness: OD/ID error ratios are typically close to 1.0 overall, but subgroup-level ratios span 0.8–1.4 and expose vulnerabilities hidden by population-level stability.Most models remain near 1.0 for T2D patients, whereas T1D subgroups show greater vulnerability.
  • Subgroup fairness: T1D patients consistently show higher rMSE than T2D patients, while age and gender interact with prediction difficulty across subgroups.Male patients generally perform better than females among ages 18–39, whereas the reverse pattern appears in the 65+ group.
  • Personalization and case difficulty: Younger T1D patients are relatively insensitive to shorter input histories, whereas older T2D patients show larger degradation when histories are reduced.Across horizons and metrics, subgroup disparities also align with the proportion of hard cases, including in young male T1D and middle-aged female T1D groups.

Experiment Implementation Details · Compute Resources · Hyperparameters

Experiments used a single dual-GPU compute node, with hyperparameters selected by validation-based Optuna tuning or fixed for TSLib architectures. Training time ranged from under one hour for shallow models to 24 hours for the most intensive transformers.

  • Compute Resources: 2 NVIDIA RTX 4090 GPUs and 128GB RAM supported all experiments on a single compute node.The GPUs each had 24GB memory.
  • Hyperparameters: Optuna selected the best-performing configurations based on validation performance.This tuning procedure was applied across the experiments.
  • Experiment Implementation Details: StatsForecast and MLForecast implemented statistical and machine learning models through the Nixtla forecasting library.These libraries provided the implementation framework for the corresponding model families.
  • Experiment Implementation Details: TSLib implemented 20 neural time-series architectures with fixed hyperparameters.The passage identifies TSLib as the framework used for these neural architectures.
  • Compute Resources: Within 1 hour, ARIMA and linear regression completed training on the FairGlucose splits.These were the shallow models described as completing training within one hour.
  • Compute Resources: Less than 2 hours was required by N-HiTS in the largest FairGlucose training runs.N-HiTS was among the most efficient deep learning models.
  • Compute Resources: Up to 24 hours was reached by Transformer and Autoformer training, making them the most computationally intensive models.Training time varied with model complexity and dataset size.

Statistical Models

The statistical forecasting framework used StatsForecast models with Optuna-based direct hyperparameter search to minimize validation RMSE. It included ARIMA and Naive, with ARIMA configured for non-seasonal forecasting and Naive serving as the untuned baseline.

  • Model development: Optuna directly optimized statistical model hyperparameters against validation-set RMSE, substantially reducing computational cost.Direct ARIMA optimization took approximately 30 seconds per trial, compared to 2–5 minutes for ARIMA.
  • Model development: The framework considered ARIMA with direct optimization of (p, d, q) and (P, D, Q), alongside Naive as a baseline without tunable hyperparameters.Models were implemented using the StatsForecast library.
  • Hyperparameter configuration: ARIMA used the optimal non-seasonal configuration p = 2, d = 1, q = 1, with seasonal parameters fixed at P = D = Q = 0.The search ranges were p ∈[0, 5], d ∈[0, 2], and q ∈[0, 5].
  • Hyperparameter configuration: Naive had no tunable hyperparameters.It served as the baseline model in the statistical forecasting framework.

Machine Learning Forecasting

Machine-learning forecasting was implemented with Nixtla’s MLForecast framework, which generates lag and date features and supports recursive multi-step prediction. Four scikit-learn-compatible regressors were evaluated and tuned using validation rMSE before final refitting.

  • Forecasting framework: MLForecast automatically converts time-series data into supervised features, supports recursive forecasting with feature updates, and enables horizontal scaling.The framework uses lags, rolling statistics, and date features.
  • Model families: The evaluated regressors were LinearRegression, LightGBM, XGBoost, and CatBoost, each instantiated through the MLForecast pipeline.The models follow the scikit-learn fit/predict API.
  • Hyperparameter optimization: Hyperparameters were optimized with model-specific Optuna search spaces using the full lag set up to INPUT_LENGTH and validation average root-mean-squared error (rMSE).Forecasts were recursive and multi-step, with lags = [1, 2, ..., INPUT_LENGTH] and no additional lag transforms.
  • Training and refitting: All models used recursive forecasts over OUTPUT_LENGTH at 5-minute frequency, with Optuna run for 50 trials before refitting the best configuration.The refitted MLForecast models were prepared for out-of-sample forecasting and evaluation.

Neural Time-Series Models (TSLib)

The study trained 20 neural time-series architectures in a unified TSLib pipeline for 2-hour glucose forecasting, using standardized training settings and architecture-specific configurations. Models span attention-based transformers, decomposition and linear models, hierarchical interpolation, temporal fusion, and neural basis expansion approaches.

  • Neural Time-Series Models (TSLib): 20 neural time-series architectures were trained through TSLib’s unified pipeline with fixed hyperparameters, 20-epoch training, early stopping, 24-hour input, and 2-hour prediction horizons.Early stopping used patience 5, with input length 288 and prediction length 24 for the main evaluation.
  • Neural Time-Series Models (TSLib): PatchTST uses channel-independent sequence patching for scalable long-horizon forecasting, with Optuna tuning patch length, stride, hidden size, attention heads, dropout, learning rate, batch size, and training steps.Its optimal configuration began with patch_len = 8.
  • Neural Time-Series Models (TSLib): DLinear separately models trend and seasonality through linear decomposition, enabling quick convergence for long sequences.The optimal configuration used learning_rate = 1e−3 and batch_size = 64.
  • Neural Time-Series Models (TSLib): The vanilla Transformer provides a standard encoder–decoder multi-head attention baseline for long-range dependencies, while TFT adds gating, recurrent encoding, attention decoding, and variable selection.The optimal TFT configuration used hidden_size = 64, n_rnn_layers = 3, and dropout = 0.1.
  • Neural Time-Series Models (TSLib): NHITS uses frequency-specialized hierarchical interpolation, Autoformer combines trend-seasonal decomposition with auto-correlation, and N-BEATS uses interpretable residual basis-expansion stacks.These architectures target efficient modeling of long sequences, periodic patterns, or trend and seasonality components.

LLM for Glucose Forecasting

The section presents a prompt-based LLM agent for short-term glucose forecasting from historical CGM sequences, using domain knowledge and physiological constraints to guide predictions. A classical Nixtla fallback and shared specifications support comparisons across model families.

  • LLM forecasting approach: The agent converts CGM sequences sampled every 5 minutes and a specified prediction horizon into natural-language prompts for an LLM backend.The prompt encodes domain knowledge and physiological constraints.
  • Physiological constraints: 30–400 mg/dL defines the permitted glucose-value range, while typical rate-of-change bounds are −5 to +5 mg/dL per minute.The prompt encourages physiologically plausible outputs, and nonconforming predictions are flagged or discarded.
  • Physiological constraints: Glucose inertia, dawn phenomenon, and meal-related spikes are included as known dynamics that the forecasting prompt encourages models to consider.Predictions violating structural or physiological constraints are flagged for review or discarded.
  • Comparability and fallback: The agent provides a fallback Nixtla time-series API for non-LLM forecasting, while all models use the same prompt and output specification for comparability.The fallback uses classical forecasting methods.

Prompt Template

The paper includes the prompt issued to the evaluated LLMs.

  • The prompt issued to the LLMs is provided in the paper.

Implementation Details

The study implements a modular end-to-end pipeline with standardized CGM handling, model inference, and evaluation across LLM and time-series backends. It supports parallel execution, deterministic forecasting configurations, structured result storage, and subgroup-level reporting.

  • Pipeline Architecture: The modular pipeline standardizes CGM input handling, model inference, and evaluation across multiple backends.It benchmarks LLM-based glucose forecasting using the FairGlucose dataset.
  • Execution and Access: Command-line parameters configure model type, model name, task index, sample count, input length, and output horizon for batched parallel execution over patient records.Credentials for OpenAI, Anthropic, and Google Vertex AI are configured dynamically through environment variables.
  • Inference: For each record, the GlucosePredictor agent builds structured prompts, handles model-specific API interaction and parsing, and recovers predictions; time-series backends receive timestamped series directly.Thread-pool parallelism accelerates batch inference.
  • Evaluation and Reporting: Results retain patient ID, prediction time, input sequence, ground truth, and model prediction before Parquet storage and sequence-based evaluation.The evaluation toolkit calculates rMSE over multiple horizons and supports subgroup analyses by diabetes type, age interval, and prediction hour.
  • Forecasting Configuration: The forecasting configuration uses an input context window of 288 time steps, a forecast horizon of 24 time steps, and temperature 0.0.These correspond to 24 hours of history and 2 hours ahead at 5-minute intervals; all listed LLMs share the same prompt template and configuration.
Loading 2608.18296v1…