Source-linked AI summary
Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validation
Taiyu Zhu, Kezhi Li, Pau Herrero, Pantelis Georgiou
TL;DR
Type 1 diabetes requires exogenous insulin, while glucose dynamics and technology limitations continue to challenge closed-loop control. The paper develops deep reinforcement-learning controllers for insulin alone or insulin with glucagon and evaluates them in silico. The authors report significantly improved glycemic outcomes over low-glucose insulin suspension in virtual adult and adolescent populations.
Problem
Closed-loop glucose control in Type 1 diabetes remains challenging because of complex glucose dynamics and technology limitations.
Source: §1 · pp. 1–2Method
The paper develops personalized single- and dual-hormone basal controllers using deep reinforcement learning with dilated recurrent networks and evaluates them in silico.
Source: §2.1.1 · p. 3, §2.4 · p. 6, §5 · p. 9Results
The percentage of CVGA points in A+B zones increased from 26% to 94% for the adult cohort and from 27% to 76% for the adolescent cohort with DRL-DH versus LGS.
Source: §3.2 · p. 7Takeaways & Limitations
The proposed approach significantly improves glycemic outcomes over low-glucose insulin suspension in virtual adult and adolescent populations.
Source: §5 · p. 9Clinical validation is still required because real-world uncertainties and perturbations remain, and the simulator overestimates glycemic-intervention efficacy.
Source: §4.2 · p. 9
Abstract
from arXiv · showhide
People with Type 1 diabetes (T1D) require regular exogenous infusion of insulin to maintain their blood glucose concentration in a therapeutically adequate target range. Although the artificial pancreas and continuous glucose monitoring have been proven to be effective in achieving closed-loop control, significant challenges still remain due to the high complexity of glucose dynamics and limitations in the technology. In this work, we propose a novel deep reinforcement learning model for single-hormone (insulin) and dual-hormone (insulin and glucagon) delivery. In particular, the delivery strategies are developed by double Q-learning with dilated recurrent neural networks. For designing and testing purposes, the FDA-accepted UVA/Padova Type 1 simulator was employed. First, we performed long-term generalized training to obtain a population model. Then, this model was personalized with a small data-set of subject-specific data. In silico results show that the single and dual-hormone delivery strategies achieve good glucose control when compared to a standard basal-bolus therapy with low-glucose insulin suspension. Specifically, in the adult cohort (n=10), percentage time in target range [70, 180] mg/dL improved from 77.6% to 80.9% with single-hormone control, and to $85.6\%$ with dual-hormone control. In the adolescent cohort (n=10), percentage time in target range improved from 55.5% to 65.9% with single-hormone control, and to 78.8% with dual-hormone control. In all scenarios, a significant decrease in hypoglycemia was observed. These results show that the use of deep reinforcement learning is a viable approach for closed-loop glucose control in T1D.
1 Introduction
Type 1 diabetes requires exogenous insulin, and current automated delivery systems still face challenges in reaching optimal therapeutic targets. This paper explores deep reinforcement learning for closed-loop blood glucose control in silico.
1 Introduction
- Type 1 diabetes involves insufficient pancreatic β-cell insulin production, so treatment requires exogenous insulin delivered as meal boluses and basal replacement.Basal insulin may be injected as slow-acting insulin or continuously infused through a pump.Source: §1 · p. 1
- Closed-loop artificial-pancreas systems have improved glycemic control, but further work is needed to reach optimal therapeutic targets.These systems combine continuous glucose monitoring, a control algorithm, and an insulin pump; some may also include glucagon delivery and activity monitoring.Source: §1 · pp. 1–2
- Deep reinforcement learning fits sequential action-selection problems, but human exploration can be dangerous and collecting offline training data is expensive and time-consuming.Dilated recurrent neural networks have performed well on long-term dependencies and future glucose prediction.Source: §1 · p. 2
- The FDA-accepted UVA/Padova T1D simulator is available for developing and evaluating insulin and glucagon delivery strategies.It provides an in-silico setting for the paper’s exploration of deep reinforcement learning for closed-loop blood glucose control.Source: §1 · p. 2
2 Methodology
The methodology formulates closed-loop glucose control as deep reinforcement learning and develops single- and dual-hormone controllers through generalized training followed by subject-specific fine-tuning. Dilated recurrent networks process time-series inputs, while double DQN and safety constraints support personalized learning.
2.1 Problem Formulation
- Closed-loop control is formulated as an infinite-state Markov decision process whose states, actions, rewards, and discount factor define the learning problem.Actions are insulin and glucagon controls, and rewards represent glycemic outcomes.Source: §2.1 · p. 3
2.1.1 Agent states
- The controller approximates physiological state from time-aligned CGM glucose, meal, insulin, and glucagon data, with observations accounting for measurement and estimation errors.Insulin input combines meal-bolus and basal doses; observations are available every five minutes.Source: §2.1.1 · p. 3
2.1.2 Actions
- The single-hormone action space adjusts or suspends basal insulin, while the dual-hormone space adds glucagon delivery capped at 1 mg per day.Glucagon doses are fixed at 0.3 µg/kg, and basal-rate options are subject-specific.Source: §2.1.2 · p. 4
2.1.3 Rewards
- The reward favors next-step glucose in the target range and penalizes values outside it; exploration terminates below 30 or above 300 mg/dL.The target range is 70–180 mg/dL, with the highest reward assigned to 90–140 mg/dL.Source: §2.1.3 · p. 4, Figure 2 · p. 4
2.2 Two-step Learning Framework
- The framework uses long-term generalized training to initialize personalized DQNs, then fine-tunes them on short subject-specific data with safety constraints.The generalized model is trained with dilated recurrent neural networks on glucose, hormone-dose, and meal time series.Source: §2.2 · p. 4, §2.4 · p. 5
- Double DQN separates action selection from value evaluation, while prioritized replay emphasizes important transitions to accelerate learning under limited clinical-trial data.The generalized controller uses separate neural networks for action selection and evaluation.Source: §2.2 · p. 4, §2.3 · p. 5
2.3 Generalized DQN Training
- Generalized training uses dilated recurrent layers to process time-aligned sequences, with dilation providing a large receptive field for glucose time-series features.The design uses three DRNN layers with exponentially increasing dilation and separate networks for action selection and value evaluation.Source: §2.3 · pp. 4–5, Figure 3 · p. 5
2.4 Personalized DQN Training
- Personalized training can fine-tune all model layers or only higher-level layers; prioritized replay uses transition importance, and action constraints suspend insulin below 80 mg/dL or glucagon above 160 mg/dL.The authors report that earlier layers contain more generic features useful across subjects; L2 regularization is used to mitigate over-fitting.Source: §2.4 · pp. 5–6
3 Experiments
In silico testing compared personalized single- and dual-hormone deep RL controllers with low-glucose insulin suspension in adult and adolescent cohorts. Both controllers improved glycemic outcomes, with dual-hormone control achieving the best performance.
3 Experiments
- A generalized model trained for 1,500 simulated days was personalized for each cohort subject with 30 simulated days, then tested over 90 days.Generalized training was intended for the simulator; personalized training had potential for a clinical setting.Source: §3 · p. 6
3.1.1 In Silico environment
- Testing used 10 virtual adults and 10 virtual adolescents, with shared daily events and variability time series across evaluated methods for a fair comparison.The simulator added variability in meals, carbohydrate estimation and bioavailability, meal absorption, and insulin sensitivity.Source: §3.1.1 · pp. 6–7
3.1.2 Baseline method
- The baseline was low-glucose insulin suspension, paired with a standard bolus calculator for meal dosing.LGS suspends basal insulin to reduce hypoglycemia.Source: §3.1.2 · p. 7
3.2 Results
- Performance was assessed using time in range [70, 180] mg/dL, hypoglycemia, hyperglycemia, mean blood glucose, and risk index, reported as mean ± SD.The three-month demonstration for selected subjects used glucose profiles and CVGA; its results were consistent with the overall population tables.Source: §3.2 · p. 7
- DRL-DH increased CVGA A+B-zone points from 26% to 94% in adults and from 27% to 76% in adolescents versus LGS.The broader cohort results also show both DRL models reducing hypoglycemia and hyperglycemia and increasing time in range; DRL-DH had the best performance.Source: §3.2 · p. 7
4 Discussion
The discussion reports better in-silico glycemic outcomes than LGS while noting that comparisons with other controllers were not head-to-head. It also identifies simulator and clinical-validation limits and describes a possible mobile-app implementation path.
4.1 Comparison with State-of-the-art
- No head-to-head comparison with existing closed-loop systems was performed because testing scenarios and controller tuning are difficult to replicate.An informal comparison notes that prior work reported adult TIR of 89%, but optimized both basal and bolus insulin and used different simulator variability; the paper positions its results as a future benchmark.Source: §4.1 · p. 7
4.2 Limitations and Future Work
- Clinical validation remains necessary because real-world uncertainties and perturbations are substantial, and the simulator may overestimate glycemic-intervention efficacy.The authors propose adding features related to glucose dynamics, including physical activity and health conditions.Source: §4.2 · p. 9
4.3 Towards Clinical Trials
- The models were developed in TensorFlow and could be implemented on smartphones or embedded devices using TensorFlow Lite.The authors describe potential ongoing refinement using incoming device data and user input, including CGM, pump, activity-monitor, and meal information.Source: §4.3 · p. 9
5 Conclusion
The conclusion presents a transfer-learning deep RL approach using dilated RNNs within double DQNs to personalize basal insulin and glucagon delivery. In virtual adult and adolescent populations, it significantly improved glycemic outcomes over low-glucose insulin suspension.
5 Conclusion
- The proposed deep RL algorithm optimizes basal insulin and glucagon delivery, using dilated RNNs in double DQNs and a two-step transfer-learning framework for personalization.Against low-glucose insulin suspension, the approach significantly improved glycemic outcomes in virtual adult and adolescent populations.Source: §5 · p. 9
7 Appendix
The appendix selects DRNNs for the DQNs, reports reward-scheme outcomes, and lists hyperparameter choices used across virtual subjects.
7.1 Neural Network Selection
- DRNNs were selected as DQNs because they showed higher TIR than LSTMs and less variability and higher mean TIR than NNs.The DRNN curve had a positive trend during personalized training; the figure reports mean TIR and 95% confidence intervals for adult and adolescent cohorts.Source: §7.1 · pp. 9–10, Figure 5 · p. 10
7.2 Reward Function
- Reward Scheme 4 achieved a TIR of 93% after 1.3 million training steps.The reported score applies when TIR converges for adult dual-hormone training with Algorithm 1.Source: §7.2 · p. 10, Table 3 · p. 10
7.3 Hyper-parameters
- Hyperparameters received limited tuning based on a state-of-the-art DQN, and all parameter settings were identical across virtual subjects.The listed DRNN dilation values are [1, 2, 4], with hidden-node counts of [32, 64, 128] across DRNN layers.Source: §7.3 · p. 10