Source-linked AI summary
Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning
Daniel J. Luckett, Eric B. Laber, Anna R. Kahkoska, David M. Maahs, Elizabeth Mayer-Davis, Michael R. Kosorok
TL;DR
Existing dynamic-treatment-regime methods are not designed for the indefinite horizons and fine-grained decisions common in mobile health. The paper proposes V-learning, a policy-based reinforcement-learning method for mHealth data, and establishes its theoretical properties and diabetes application. The method estimates regimes for controlling blood glucose in type 1 diabetes and is intended for settings with small samples and many decision points.
Problem
Existing methods for estimating optimal dynamic treatment regimes are designed for few fixed decision points, whereas mHealth applications involve fine-grained decisions and rich longitudinal data.
Method
V-learning estimates an optimal policy within a prespecified policy class using minimal assumptions and supports randomized decision rules for online estimation.
Results
The estimators are consistent and asymptotically normal under the stated assumptions, and the method estimates regimes to reduce hypo- and hyperglycemic episodes in type 1 diabetes.
Takeaways & Limitations
V-learning provides a framework for estimating and implementing dynamic treatment regimes in mHealth settings, including glucose-management applications for type 1 diabetes.
Takeaways & Limitations
The method’s implementation is evaluated against GGQ in a setting where mHealth applications commonly have small samples and many fine-grained decision points, and future work identifies added flexibility and feasibility constraints as open areas.
Abstract
from arXiv · showhide
The vision for precision medicine is to use individual patient characteristics to inform a personalized treatment plan that leads to the best healthcare possible for each patient. Mobile technologies have an important role to play in this vision as they offer a means to monitor a patient's health status in real-time and subsequently to deliver interventions if, when, and in the dose that they are needed. Dynamic treatment regimes formalize individualized treatment plans as sequences of decision rules, one per stage of clinical intervention, that map current patient information to a recommended treatment. However, existing methods for estimating optimal dynamic treatment regimes are designed for a small number of fixed decision points occurring on a coarse time-scale. We propose a new reinforcement learning method for estimating an optimal treatment regime that is applicable to data collected using mobile technologies in an outpatient setting. The proposed method accommodates an indefinite time horizon and minute-by-minute decision making that are common in mobile health applications. We show the proposed estimators are consistent and asymptotically normal under mild conditions. The proposed methods are applied to estimate an optimal dynamic treatment regime for controlling blood glucose levels in patients with type 1 diabetes.
1 Introduction
The paper introduces V-learning to estimate optimal dynamic treatment regimes from rich longitudinal mHealth data, addressing treatment decisions made at fine time scales. It is designed for indefinite-horizon outpatient applications and supports both offline and online estimation.
- mHealth technologies can collect rich longitudinal data and deliver interventions tailored to each patient’s evolving health status.
- Dynamic treatment regimes represent individualized treatment as a sequence of decision rules mapping current patient information to treatments over time.
- Existing Q-function approaches risk model misspecification with parsimonious models or unintelligible regimes with complex models, while also involving non-smooth estimation.
- V-learning estimates the optimal policy within a prespecified policy class, requires minimal data-generating assumptions, and permits randomized decision rules for online implementation.
- The paper develops offline and online estimation procedures and presents consistency, asymptotic normality, simulations, and a type 1 diabetes case study.
2 Offline estimation from observational data
The offline procedure models patient trajectories as a time-homogeneous Markov process and evaluates candidate policies through discounted utilities. V-learning estimates each policy’s state value and selects the policy with the highest estimated value.
- Each trajectory records patient states, treatments, and follow-up time; in type 1 diabetes, states may include glucose, diet, and activity, while actions may represent insulin injections.
- The framework assumes Markov state transitions, follow-up status contained in the state, and a known utility measuring the goodness of treatment-induced transitions.
- The value function discounts future utilities with γ ∈(0, 1), and the optimal regime maximizes value over a prespecified class of regimes.
- Treatment probabilities are known in micro-randomized trials and estimated in observational studies, enabling importance-weighted value estimation.
- V-learning estimates the state-value function through a penalized estimating equation, then maximizes the estimated value over the policy class.
- The implementation uses randomized policies parameterized through treatment-specific probabilities and basis-function approximations for state values.
3 Online estimation from accumulating data
The online procedure updates treatment policies as data accumulate, using newly observed states and outcomes to estimate and deploy policies sequentially. Randomization supports exploration, while policies may also be individualized across patients.
- At each time point, the method estimates a policy from data collected so far, takes the next action under that policy, and updates it using the resulting observations.
- For randomized policy classes, the next treatment is sampled from the estimated policy distribution conditional on the newly observed state.
- Deterministic policies can be randomized with ε-greedy exploration, which selects the estimated optimal action with probability 1 − ε and otherwise samples other actions.
- Policy updates may occur in batches, and uncertainty-based strategies such as upper confidence bound sampling can encourage exploration.
- When patient processes differ, each patient can receive an individualized policy estimated from that patient’s accumulated data, with decreasing exploration over successive updates.
4 Theoretical results
The theoretical analysis establishes existence and asymptotic behavior of the V-learning estimators under regularity conditions. It also shows convergence of the estimated optimal policy and its value within the policy class.
- Because patient trajectories are stationary Markov chains, the proof uses asymptotic theory for stationary processes rather than independent observations.
- The policy-value map is required to have a unique, well-separated interior maximum, corresponding to a unique optimal decision in each state.
- The analysis assumes regularity conditions including finite moments, dependence decay for patient trajectories, policy-class entropy control, and smoothness.
- Theorem 4.2 gives a unique estimating-equation solution and weak convergence of the estimated value-function parameters to a mean-zero Gaussian process.
- Theorem 4.3 establishes convergence in probability of the estimated optimal policy to the true optimal policy over the policy class.
5 Simulation experiments
The simulations evaluate V-learning against GGQ in offline and online settings, including fine-grained type 1 diabetes scenarios. Results favor V-learning with flexible basis functions, while individualized policies outperform universal policies when optimal treatment varies across patients.
- 5.1 Greedy gradient Q-learning: V-learning directly maximizes estimated policy value over a policy class, whereas GGQ depends on a non-smooth max operator and can be difficult to estimate with limited data.V-learning models the policy and value function rather than the data-generating process.
- 5.2 Offline simulations: The simulations compare linear, polynomial, and Gaussian radial basis-function value models, with regularization used for policy coefficients and value-function estimation.The Gaussian basis uses prespecified centers and scales after state variables are normalized between 0 and 1.
- 5.2 Offline simulations: Offline simulations generate patient trajectories, estimate policies with V-learning and GGQ, and evaluate each policy using mean utility from 100 simulated patients over 100 time points.The treatment assignment is generated from a Bernoulli distribution with mean 1/2, corresponding to a known treatment mechanism in a micro-randomized trial.
- 5.2 Offline simulations: In simulated type 1 diabetes cohorts, decisions occur hourly and utilities penalize hypo- and hyperglycemia, with values ranging from −6 to 0.The state includes recent blood glucose, dietary intake, and physical activity, and the goal is to control future blood glucose near the optimal range.
- 5.2 Offline simulations: Gaussian V-learning generally performs best in the type 1 diabetes simulations, while linear V-learning and GGQ underperform considerably.The reported comparison concerns estimated policy values and standard errors across the simulated methods.
- 5.3 Online simulations: For online simulations, polynomial V-learning generally performs best in the simple model, Gaussian V-learning performs best in simulated type 1 diabetes cohorts, and outcomes improve with longer follow-up.When the optimal policy varies across patients, individualized policies achieve better outcomes than universal policies.
6 Case study: Type 1 diabetes
The case study applies V-learning to outpatient type 1 diabetes data to estimate dynamic treatment strategies based on glucose, diet, and physical activity. Policies using multiple actions improved estimated value over the observational policy, while an example patient received action probabilities tailored to elevated glucose.
- Data and setup: The study uses outpatient data from 14 insulin-pump patients, representing 28 patient-days, to estimate an optimal treatment policy.The broader dataset included 31 patients aged 12–18, monitored with continuous glucose monitoring and accelerometers.
- Data and setup: The patient state summarizes recent glucose, physical activity, and food intake, while the policy determines when to administer insulin.The utility weights glycemic events before and after each decision according to their clinical importance.
- Treatment strategies: V-learning estimates policies in a binary action space and a multiple-action space combining insulin, physical activity, and food-intake recommendations.The multiple-action space represents a hypothetical intervention pairing an insulin pump with mobile-app suggestions.
- Results: 64% increase in value is achieved by the polynomial-basis policy with γ = 0.7 over the observational policy in the multiple-action setting.The Gaussian-basis policy with γ = 0.8 achieves a 68% increase in value over the observational policy.
- Example recommendation: A hyperglycemic example patient receives probability 0.79 for insulin and 0.21 for insulin combined with activity.The estimated policy assigns actions intended to lower glucose because the patient’s recent glucose levels exceeded the optimal range.
7 Conclusion
The paper concludes that V-learning can estimate dynamic treatment regimes for mHealth and potentially other settings. It reports potential benefits for patient outcomes while identifying future extensions for flexibility, tuning, and implementation constraints.
- Contributions: V-learning directly estimates policies that maximize value over a policy class and requires minimal assumptions on the data-generating process.It also supports randomized decision rules for online exploration and multiple treatment options.
- Contributions: Different reference distributions can support estimation of optimal policies for different populations.
- Conclusion: The method can estimate regimes intended to reduce hypo- and hyperglycemic episodes in patients with type 1 diabetes.
- Future directions: Future work includes more flexible state-value models, data-driven tuning, and policy constraints reflecting patient availability and treatment feasibility.These constraints are intended to keep resulting interventions implementable and consistent with domain knowledge.
Appendix
The appendix establishes the regularity and empirical-process results used to justify the paper’s theoretical guarantees. It verifies properties of the policy and function classes and derives a Gaussian-process limit for the estimator.
- Theoretical verification: The appendix proves consistency-related properties for the policy-specific parameters and establishes uniform continuity and boundedness conditions.These arguments use the stated assumptions, compactness, moment bounds, and nonsingularity conditions.
- Empirical-process conditions: The policy and function classes satisfy bracketing-entropy conditions that support Donsker and Glivenko–Cantelli arguments.The appendix derives entropy bounds for products, unions, finite classes, and the parameterized policy class.
- Estimator control: The proof controls the regularized estimator uniformly over policies and shows that the regularization term is asymptotically negligible when λ_n = o_P(n^-1/2).The argument combines uniform convergence, stochastic equicontinuity, and Donsker properties of auxiliary function classes.
- Asymptotic distribution: The appendix derives a weak-convergence result to a mean-zero Gaussian process indexed by policies, with covariance w_0(π_1, π_2).The convergence is stated in ℓ∞(Π) after normalization by w_1(π)^-1.
- Connection to main results: The theorem proofs combine the established empirical-process results with a unique, well-separated maximum condition and continuity of the value function.These ingredients yield the stated theorem conclusions, while the detailed proofs and policy-class entropy verification are placed in the appendix.