Source-linked AI summary
Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity
Peng Liao, Kristjan Greenewald, Predrag Klasnja, Susan Murphy
TL;DR
Mobile health JITAIs must adapt intervention decisions to users’ changing contexts, despite noisy data, limited interaction time, and delayed effects. This paper develops an online RL algorithm for HeartSteps V2, using HeartSteps V1 data to inform its design and simulation evaluation; preliminary synthetic experiments show better performance than Thompson Sampling Bandit.
Problem
Mobile health RL must address delayed treatment effects, noisy in-situ data, limited opportunities for learning, and changing context-to-reward relationships.
Method
The paper develops an online RL algorithm using informative priors, low-dimensional action-centered reward modeling, and a dosage variable for treatment burden.
Results
Preliminary validation demonstrates good performance over the Thompson Sampling Bandit algorithm in synthetic experiments based on HeartSteps V1.
Takeaways & Limitations
The algorithm is being used in HeartSteps V2 to decide whether to deliver context-tailored physical activity suggestions five times per day.
Takeaways & Limitations
The proxy for delayed effects needs improvement because its response to excessive suggestion doses can lag behind declining treatment responsiveness.
Abstract
from arXiv · showhide
With the recent evolution of mobile health technologies, health scientists are increasingly interested in developing just-in-time adaptive interventions (JITAIs), typically delivered via notification on mobile device and designed to help the user prevent negative health outcomes and promote the adoption and maintenance of healthy behaviors. A JITAI involves a sequence of decision rules (i.e., treatment policy) that takes the user's current context as input and specifies whether and what type of an intervention should be provided at the moment. In this paper, we develop a Reinforcement Learning (RL) algorithm that continuously learns and improves the treatment policy embedded in the JITAI as the data is being collected from the user. This work is motivated by our collaboration on designing the RL algorithm in HeartSteps V2 based on data from HeartSteps V1. HeartSteps is a physical activity mobile health application. The RL algorithm developed in this paper is being used in HeartSteps V2 to decide, five times per day, whether to deliver a context-tailored activity suggestion.
1 Introduction
JITAIs use current user context to determine whether and what intervention to deliver, while the paper develops an RL algorithm that learns and optimizes this policy online for HeartSteps V2.
- JITAIs use real-time context, such as time, location, activity, and stress, to adapt intervention type and timing.
- The paper develops an RL algorithm that continuously learns and optimizes the JITAI treatment policy as users experience the intervention.
- HeartSteps V2 uses the algorithm to learn whether to deliver a context-tailored physical activity suggestion as the trial progresses.
- The paper evaluates the proposed algorithm using a generative model built from HeartSteps V1 data.
2 HeartSteps V1 and V2: Physical Activity Mobile Health Study
HeartSteps V2 is an ongoing 90-day trial using a mobile application, Fitbit tracker, and context-tailored suggestions to improve physical activity. HeartSteps V1 data informs both algorithm design and simulation-based evaluation.
- HeartSteps V2 is a 90-day trial for improving physical activity among individuals with stage 1 hypertension.Participants use a Fitbit tracker and mobile phone application.
- The intervention may deliver contextually tailored physical activity suggestions at five user-specified times each day.These times roughly correspond to morning commute, midday, mid-afternoon, evening commute, and post-dinner periods.
- HeartSteps V1 was a 42-day study of 37 healthy sedentary adults with randomized suggestions at five daily decision times.Suggestions were randomized with a constant probability of 0.30.
- HeartSteps V1 data informs HeartSteps V2 variable selection, treatment-efficacy modeling, prior construction, and generative simulation evaluation.
3 Challenges to Applying RL in mHealth
Applying RL to mobile health requires handling delayed treatment effects, rapid learning under noisy observations, non-stationarity, and the feasibility of later secondary analyses.
- RL learns action policies through repeated state, action, and reward interactions, balancing exploitation against exploration.In HeartSteps, states summarize current and past context, actions determine suggestion delivery, and rewards reflect near-term activity.
- Current actions can improve immediate rewards while reducing future rewards through habituation or burden, requiring longer-term optimization.
- Mobile health RL must learn quickly from noisy, uncontrolled data because users may disengage before many environment interactions occur.Step counts and context measurements contain sensor and unobserved-context noise.
- Context-to-reward relationships may be non-stationary because engagement and burden are partly unobserved and treatment effects can decline over study time.
- The treatment policy should preserve feasibility for off-policy learning, causal inference, and other secondary analyses after the study.
4 Existing RL-based Mobile Health Studies
Prior mobile health studies applied RL or contextual bandits to intervention selection, but the paper argues that these approaches do not address the full set of mobile-health challenges.
- Few mobile health studies continuously learn and improve an individual’s treatment policy in real time rather than using a pre-specified policy.
- Existing studies used contextual bandits with Softmax or Upper Confidence Bound methods to select physical-activity or stress-management suggestions.
- The paper argues that prior approaches inadequately incorporate prior participant data, tolerate reward-model misspecification, and optimize beyond immediate rewards.It identifies these gaps as requiring generalization in several directions.
5 Reinforcement Learning Algorithm in HeartSteps V2
The HeartSteps V2 algorithm uses reinforcement learning to select activity suggestions from participant context while accounting for delayed treatment effects, uncertainty, and continued exploration. It updates treatment-effect beliefs nightly and incorporates treatment history through a decaying dosage variable.
- 5.1 Addressing the Challenges: The policy uses a low-dimensional linear treatment-effect model and Thompson Sampling to balance uncertainty-driven exploration with exploitation.Informative priors incorporate prior knowledge and are intended to accelerate early learning while reducing variance.
- 5.1 Addressing the Challenges: Action-centering protects treatment-effect estimation from misspecification of the baseline reward model, while probability clipping preserves exploration and supports later off-policy analyses.The clipping bounds are ϵ0 = 0.2 and ϵ1 = 0.1 in HeartSteps V2.
- 5.2 Reinforcement Learning Framework: The algorithm selects binary activity-suggestion actions from states containing availability, user context, and a treatment-burden dosage variable.HeartSteps decision times occur five times daily; context features include location, recent and historical step counts, temperature, and recent activity.
- 5.2 Reinforcement Learning Framework: A decaying dosage variable captures treatment burden by multiplying prior dosage by λ and adding one when recent suggestions occurred.The dosage starts at X1 = 0 and is updated as Xt+1 = λXt + 1Et+1, with λ selected from HeartSteps V1 analyses.
- 5.3 Action Selection: The algorithm performs nightly updates that map the accumulated daily history to a posterior distribution for treatment effects and an updated delayed-effect proxy.These outputs are used for action selection on the following day.
- 5.3 Action Selection: A proxy for delayed effects adjusts action selection beyond immediate rewards, with ηd(x) representing the long-term negative effect of sending a suggestion at dosage x.When ηd(x) = 0, the method reduces to a bandit formulation that maximizes immediate rewards without considering future effects.
6 Simulation Study
The simulation study uses HeartSteps V1 data in a three-fold cross-validation framework to tune and evaluate the proposed algorithm against Thompson Sampling Bandit. In testing, the proposed method achieved higher total rewards for most participants and improved average total reward over the comparator.
- 6 Simulation Study: The study uses HeartSteps V1 data to evaluate input selection, tuning parameters, proxy delayed effects, and action-centering through a generative simulation model.The evaluation targets validity of the procedure and the algorithmic components addressing delayed effects and model misspecification.
- 6.1 Training Phase: Three-fold cross-validation assigns two folds to training and one fold to testing in each iteration.Training data construct priors, estimate noise variance, and select tuning parameters; testing data evaluate performance.
- 6.2 Testing Phase: The proposed algorithm is compared with a Thompson Sampling Bandit that selects actions probabilistically to maximize immediate reward.Both methods use comparable reward features, training-derived priors, error variance, and probability constraints.
- 6.2 Testing Phase: 29 out of 37 participants had higher total rewards with the proposed algorithm than with TS Bandit.The average improvement in total rewards over TS Bandit was 29.753.
- 6.2 Testing Phase: The simulation results indicate that action-centering and the proxy delayed effect were effective for addressing model misspecification and negative delayed treatment effects.This interpretation is reported for the simulation study using HeartSteps V1-derived generative models.
7 Pilot Data From HeartSteps V2
HeartSteps V2 deploys the RL algorithm to decide whether to deliver context-tailored activity suggestions at five daily decision times. Pilot and synthetic evaluations show increased activity for most participants, while participant-specific responsiveness and delayed burden expose calibration challenges.
- The RL algorithm is used in HeartSteps V2 to decide whether to trigger a context-tailored activity suggestion at each of five decision times per day.
- 7.1 Initial Assessment: 125 more steps were taken on average in the 30-minute window after decision times during intervention weeks than during the initial week.Eight participants had been in the field for over one week; all but participant ID 4 showed positive increases.
- 7.2 Lessons: For participant ID 4, posterior treatment-effect estimates fell below 0 while randomization probabilities remained between 0.2 and 0.4.The paper characterizes this participant as unresponsive because suggestions did not significantly improve step count.
- 7.2 Lessons: For participant ID 7, responsivity began decreasing around 07-10, but RL randomization probabilities did not begin decreasing until 07-16.The authors identify the proxy value as needing improvement because it should respond more quickly to excessive intervention dose.
- 7.2 Lessons: The proxy adjustment reduced the probability of sending walking suggestions when delayed effects were present, but the authors wanted a larger gap between adjusted and unadjusted probabilities.
8 Conclusion and Future Work
The paper develops an RL algorithm for HeartSteps V2 and finds preliminary advantages over Thompson Sampling Bandit in synthetic experiments based on HeartSteps V1. Future work will assess the algorithm further and address personalization, burden modeling, engagement, and intervention reduction.
- Conclusion: The paper develops a Reinforcement Learning algorithm for use in HeartSteps V2.
- Conclusion: Preliminary validation demonstrates good performance over Thompson Sampling Bandit in synthetic experiments constructed from HeartSteps V1 data.
- Future Work: The completed HeartSteps V2 data will be used to further assess the algorithm’s performance and utility.
- Future Work: Future work may pool information across similar participants to speed learning and improve performance, especially for participants entering the study later.
- Future Work: Future work may use more sophisticated burden and engagement measures to approximate delayed effects and respond more quickly to prevent disengagement.