PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal DAG-Guided Reinforcement Learning for Long-Term Physical Activity Maintenance: Using Prior Data to Inform Future Deployment

Xueqing Liu, Young Hyun Cho, Bhanu Teja Gullapalli, Predrag Klasnja, Susan Murphy

arXiv:2610.36019v1stat.APcs.LG

TL;DR

Sustaining physical activity is difficult, and digital interventions face sparse participant-specific data, partially observed states, frequent actions with weekly outcomes, and possible intervention burden. This paper develops a causal-DAG-guided RL system and an MRT-calibrated simulator; the simulator broadly reproduces observed trajectories, while learned reward redistribution with shaping improves on random sending in some tested settings. The authors identify incomplete modeling of data latency and reliance on the working DAG and associated models as limitations.

  • Problem

    Long-term physical activity maintenance remains difficult, while participant-specific RL must learn from sparse data, partial observations, frequent decisions, and delayed weekly outcomes.

    Source: §1 · p. 1

  • Method

    An expert-specified causal DAG guides RL state construction, partially observed-state estimation, benefit-and-burden pathways, and delayed credit assignment, while an MRT-calibrated simulator evaluates candidate designs.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    89.9–97.7% of observed MRT measurements fell within pointwise 95% simulation intervals, and learned reward redistribution stacked on shaping improved on random send in some environments.

    Source: §4.3 · p. 5, RL Variants Comparison · p. 6

  • Takeaways & Limitations

    The simulator broadly reproduces observed participant-level trajectories and supports pre-deployment comparison of candidate RL designs under modeled state evolution, missingness, and participant heterogeneity.

    Source: §1 · p. 2, §4.3 · p. 5

    The simulator does not fully reproduce delayed data availability, and robustness depends on the working DAG and associated models.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Digital health interventions can provide scalable, personalized support for physical activity (PA), yet sustaining PA over the long-term remains difficult. Because affective experiences with PA are important for activity sustainment, we developed an online reinforcement learning (RL) system for the forthcoming ADAPT factorial trial that personalizes twice-daily activity suggestions aimed at improving affective associations with PA. This setting poses several challenges: participant-specific data are scarce relative to the candidate state space, important states and rewards are partially observed, interventions are frequent while the primary outcome is measured weekly, and activity suggestions can both improve PA-related experiences and increase intervention burden. We address these challenges using an expert-specified causal directed acyclic graph as an integrated RL design tool, guiding state construction, estimation of partially observed states, delayed credit assignment, and representation of beneficial and adverse intervention pathways. We further construct a longitudinal simulator calibrated from a prior micro-randomized trial to compare candidate RL designs under realistic state evolution, missingness, and participant heterogeneity before deployment. Together, this work illustrates how causal domain knowledge and prior trial data can make long-horizon, participant-specific RL more practical for digital health interventions targeting sustained behavior change.

1 Introduction

The forthcoming ADAPT trial uses online reinforcement learning to personalize activity suggestions intended to improve affective associations with physical activity. The paper uses an expert-specified causal DAG and an MRT-calibrated simulator to address challenges in learning and evaluating this policy.

1 Introduction

  • ADAPT will personalize activity suggestions twice daily over a 9-month trial, optimizing a weekly measure of how participants feel about being physically active.The algorithm draws on wristband, app, and survey data.Source: §1 · p. 1
  • Participant-specific data are sparse relative to the many candidate state variables, and sensor or survey information may be missing or delayed.These conditions can increase estimation variance and complicate learning early in deployment.Source: §1 · p. 1
  • Suggestions may improve affective associations but also increase burden or reduce perceived utility, while 12 weekly decisions share one delayed CAE reward.The policy must account for possible downstream costs as well as benefits.Source: §1 · pp. 1–2
  • An expert-specified causal DAG guides state construction, estimation of partially observed states, benefit-and-burden pathways, and delayed credit assignment.The DAG identifies intermediate variables along pathways from suggestions to future CAE.Source: §1 · p. 2
  • The authors construct an ADAPT simulator calibrated with prior micro-randomized trial data to evaluate candidate RL designs before deployment.It reproduces the trial’s decision schedule, feedback pathways, and observation processes.Source: §1 · p. 2

2 Related Work

The paper situates its approach among online RL, causal RL, and pre-deployment simulation research. It distinguishes its work by using one expert-specified DAG to guide several components of a long-horizon, participant-specific system and by extending simulation to ADAPT’s multi-timescale processes.

Online RL in digital health

  • Online RL has been deployed across digital health domains, but most systems use contextual bandits targeting relatively proximal outcomes.Some systems address longer-term effects through engineered features, reward adjustments, or nonstationary dynamics.Source: §2 · p. 2

Causal RL

  • Prior causal RL work uses structure for exploration, policy search, state construction, treatment-effect estimation, partial observability, and delayed credit assignment.The cited work includes expert-specified DAGs for Markovian states and belief-state modeling.Source: §2 · p. 2
  • This work uses a single expert-specified DAG to jointly guide state construction, partially observed-state estimation, benefit and burden pathways, and mediator-based credit assignment.ADAPT combines multiple within-week decisions sharing a weekly outcome with partially observed states and effects that may propagate across weeks.Source: §2 · p. 2

Pre-deployment evaluation

  • The ADAPT simulator extends historical-data-based pre-deployment evaluation by modeling multi-timescale states, mediators, outcomes, and observation processes.It enables evaluation under delayed rewards, participant heterogeneity, and incomplete observations.Source: §2 · p. 2

3 The ADAPT RL System

The ADAPT system personalizes activity suggestions using participant data, historical trial data, and causal domain knowledge. Its design uses the DAG to construct a compact state, estimate partially observed weekly states, represent delayed benefits and burdens, and guide online value learning.

3 The ADAPT RL System

  • The planned system personalizes app-delivered activity suggestions for 75 participants over 36 weeks, with decisions twice daily and weekly policy updates.Its objective is cumulative weekly CAE; the full 12-item measure is the target, while a short 3-item measure supplies partial information.Source: §3 · pp. 2–3

3.1 Causal-DAG-Guided RL Design

  • The DAG compresses observed history into a state intended to preserve information relevant to future mediators, weekly states, and outcomes.The state includes CAE posterior mean and standard deviation, estimated perceived utility, mediator summaries, and available contextual variables.Source: §3.1 · p. 3
  • CAE and perceived utility are estimated separately from incomplete observations, using survey responses, mediator emissions, and engagement information.CAE is updated from a full survey when available or from short-survey and mediator information; perceived utility uses a point estimate, with its model parameter fixed from MRT data.Source: §3.1 · pp. 3–4
  • The working DAG represents a potential benefit pathway through anticipated affect and physical activity and a potential burden pathway through engagement and perceived utility.Weekly bottleneck states and intermediate mediators motivate value bridging, potential-based shaping, and retrospective redistribution of the weekly reward across decisions.Source: §3.1 · p. 4

3.2 Online Personalization with RLSVI

  • RLSVI learns a participant-specific suggestion policy, initializing its Bayesian value-function prior from historical ADAPT MRT data while retaining randomized exploration.A parsimonious linear action-value representation limits model complexity in the data-scarce setting.Source: §3.2 · p. 4
  • The candidate algorithms retain the RLSVI backbone while varying discounting, bottleneck-value specification, and reward construction.An ensemble approximates the posterior probability that suggesting has greater value than not suggesting, with action probabilities clipped to preserve a minimum chance of both actions.Source: §3.2 · pp. 4–5

4 Pre-Deployment Evaluation

The pre-deployment evaluation uses prior ADAPT MRT data to build and validate a simulator, then compares RL variants over the planned 36-week trial horizon. Reward redistribution improves on random sending most clearly in the strongest-signal environment.

4.1 The ADAPT MRT Data

  • The MRT supplies both simulator models and participant-specific RL priors; pre-deployment priors exclude the simulated participant’s own MRT data.The MRT’s twice-daily activity-suggestion schedule and collected measures closely correspond to the forthcoming factorial trial, though some variables use surrogates.Source: §4.1 · p. 5, §4 · p. 5

4.2 Simulation Environment Design

  • The simulator represents CAE, perceived utility, and other time-varying variables with distinct models, including participant-level residual sequences that preserve temporal dependence.CAE processes use Bayesian hierarchical linear models; perceived utility is a latent state inferred from observed mediators and emissions; fitted residuals use an AR(1)-style bootstrap.Source: §4.2 · p. 5

4.3 Experiments and Results

  • The evaluation first checks simulated MRT trajectories, then compares candidate RL variants by cumulative CAE over 36 weeks using 200 trials of 75 simulated participants.Participants are sampled with replacement from 28 MRT participant models, with independently generated stochastic trajectories for repeated draws.Source: §4.3 · p. 5, RL Variants Comparison · p. 6
  • 89.9–97.7% of observed MRT measurements across the modeled processes fall within pointwise 95% simulation intervals for the average participant.The reported perceived-utility mediator coverage averages app page views, Fitbit wear, and daily survey completion.Source: §4.3 · p. 5, Table 1 · p. 5
  • RLSVI with γ = 0.9 outperforms γ = 0.5 across panels, while bottleneck-value parameterization does not improve on standard value propagation and is slightly worse under weak signal.The paper attributes the discount comparison to γ = 0.5 placing little weight on outcomes more than two weeks ahead, and the bottleneck result to additional parameters estimated from sparse participant data.Source: RL Variants Comparison · p. 6
  • Only stacking learned reward redistribution on potential-based shaping improves on random send; the gain is clearest at STE ≈0.8.At STE ≈0.8, the proximal activity-suggestion → four-hour step-count → CAE pathway is amplified, strengthening the slot-level learning signal.Source: RL Variants Comparison · p. 6

5 Discussion

The discussion summarizes a causal-DAG-guided RL system evaluated before deployment and describes the selected configuration and operational safeguards. It also identifies simulator data-latency and causal-model assumptions as limitations requiring further assessment.

5 Discussion

  • The work develops a causal-DAG-guided RL system for the forthcoming ADAPT factorial trial and evaluates candidate designs in an MRT-calibrated simulator.Source: §5 · p. 6

5.1 Path to Deployment

  • The final configuration uses γ = 0.9 with potential-based shaping and learned reward redistribution, and initializes deployment with informative priors from eligible ADAPT MRT participants.The deployment plan addresses missing or delayed data by using only information available at each decision and retaining the most recent validated component when an update fails.Source: §5.1 · p. 6
  • Monitoring will track data availability and latency, state construction, weekly updates, action probabilities, and fallback events for operational detection and post-trial assessment.Monday-night updates include CAE and perceived-utility estimation, empirical-Bayes variance updates, RLSVI, and reward redistribution when applicable.Source: §5.1 · p. 6

5.2 Limitations and Future Work

  • The simulator does not fully reproduce delayed availability of tracker, survey, and app data, and robustness depends on the working causal DAG and associated models.The authors identify realistic latency modeling, sensitivity analyses, and postdeployment model checking as future needs.Source: §5.2 · p. 6

Supplementary Material

The supplementary material describes the planned factorial trial, the prior ADAPT micro-randomized trial, and how MRT data inform the future RL system. The MRT provides data for simulator models and participant-specific priors, while some future-trial variables require surrogate mappings.

A.1 ADAPT Factorial Trial Design

  • The ADAPT factorial trial randomizes participants across ADAPT and Intrinsic Motivation factors; approximately half of the 150 cardiac rehabilitation patients are expected to receive RL personalization.The trial follows participants for approximately 36 weeks, and the role of the Intrinsic Motivation factor is outside this paper’s scope.Source: §A.1 · p. 9
  • The baseline phase collects CAE and perceived-utility measures, and participants enter the initial micro-randomization phase after meeting a Fitbit-wear threshold.Baseline CAE is the mean of 12 items; the baseline phase lasts about a week but may be extended until participants accumulate at least seven days with eight hours of Fitbit wear per day.Source: §A.1 · p. 9

A.2 ADAPT MRT Design

  • The MRT repeatedly randomizes activity suggestions twice daily and collects longitudinal activity, psychosocial, survey, and app-engagement data relevant to the future RL system.The MRT also randomizes morning salience messages and planning prompts once daily; other listed components are delivered without micro-randomization.Source: §A.2 · p. 9

A.3 MRT-to-Factorial Trial Variable Correspondence

  • MRT data estimate simulator-generating models and construct informative priors for participant-specific RLSVI, while Table 2 summarizes mappings from future-trial targets to MRT surrogates.Source: §A.2 · p. 9, §A.3 · p. 9

A.4 Data Preprocessing and Standardization

The appendix describes MRT data preprocessing and standardization, alongside a summary of how the causal DAG informs online RL design. Preprocessing includes variable-specific transformations and distinct handling of missing outcomes and predictors.

A.4 Data Preprocessing and Standardization

  • The analysis sample retained 28 participants after exclusions for insufficient wearable or survey data and prespecified CAE ceiling criteria.The source data included 43 participants completing the 12-week active phase; eight were excluded for availability criteria and seven more for CAE ceiling criteria.Source: §A.4 · p. 9
  • Calendar variables are rescaled, step counts are log-transformed, and page views use hurdle models because they have many zero values.Source: §A.4 · p. 9
  • Continuous variables are z-standardized using analysis-sample means and standard deviations, with scaling constants and empirical support limits retained for reproducibility and online use.Source: §A.4 · p. 9
  • Ordinal survey variables on a fixed 1–7 scale use scale-based normalization to [−1, 1] rather than empirical z-score standardization.Source: Table 2 · p. 10
  • Missing outcome values are excluded during model fitting, while missing predictor values are replaced with their mean.Source: §A.4 · p. 10

B Causal-DAG-Guided Online RL Design

  • The causal-DAG-guided RL design covers partially observed weekly CAE and perceived-utility states, parsimonious decision-time states, weekly bottlenecks, and intermediate mediators for delayed credit assignment.Source: B Causal-DAG-Guided Online RL Design · p. 10

B.1 Partial Observability

The design represents partially observed weekly CAE and perceived utility as beliefs combined with mediator histories and context, using the working DAG to shorten delayed-credit paths and distribute weekly learning signals across decisions. Participant-specific RLSVI is initialized with historical MRT data and updated with each participant’s trial data, with a linear Q-function and discounting across weekly boundaries.

B.1 Partial Observability

  • A particle filter maintains beliefs about weekly CAE when it is unobserved, using transition, observation, and mediator models; survey completion is modeled through perceived utility rather than as a direct CAE measure.The working assumption is that, conditional on perceived utility and observed history, survey completion adds no information about latent CAE.Source: §B.1.0.0.1 · pp. 10–11
  • The fitted perceived-utility approximation assigns zero weights to both weekly check-in terms, with daily check-in, page-view, wear, and autoregressive signals absorbing their information.The weights are estimated offline from the historical MRT and held fixed during the factorial trial.Source: Table 3 · p. 13, §B.1.0.0.2 · p. 13

B.2 State Construction

  • The RL state combines latent-state estimates, recency-weighted mediator histories, and current context, including only measurements observed by decision time.The mediator streams encode hypothesized pathways to future CAE or perceived utility; context includes recent steps, intervention burden, engagement, and activity.Source: §B.2 · pp. 13–14

B.3 DAG-Guided Delayed Credit Assignment

  • The bottleneck-value variant changes value-function factorization without changing the scientific reward, shortening cross-week information propagation.It links a week-level value based on the low-dimensional weekly state to decision-time values.Source: §B.3 · p. 14, §B.3.0.0.1 · p. 15
  • Within-week redistribution allocates a selected weekly reward across 12 decisions using DAG-motivated mediator information and model-based attribution of daily mediators.The method decomposes once-daily mediators affected by both daily actions, then fits decision-level predictions to sum to the weekly reward; page views enter directly as an engagement readout.Source: §B.3.0.0.3 · pp. 15–16

C Participant-Specific Online Learning with RLSVI

C.1 Linear Q function

  • Rewards are undiscounted within weeks, while γ discounts rewards across weekly boundaries.Source: §C.1 · p. 16
  • The linear Q-function representation limits model complexity in the participant-specific, data-scarce setting.Source: §C.1 · p. 16

C.2 MRT-Informed Prior

The appendix describes how MRT data inform priors and how the participant-specific RL policy selects actions and updates weekly. It also specifies shared noise-variance estimation and the weekly update schedule.

C.2 MRT-Informed Prior

  • MRT data initialize Gaussian priors through pooled ridge-regression means and across-participant coefficient variability, with Q-function priors fitted offline for each reward construction and discount setting.For held-out evaluation, priors exclude the simulated participant’s own MRT data; eventual trial priors will use all eligible MRT data.Source: §C.2 · p. 17

C.3 Action Selection

  • The action-selection subroutine combines ensemble voting with clipped assignment probabilities, then samples an activity-suggestion action and returns its assignment probability.The ensemble contains randomized Q-functions.Source: §C.3 · p. 17

C.4 Weekly RLSVI Update

  • Each Monday night, the participant-specific RLSVI ensemble is updated using completed transitions through the prior week, temporal-difference targets, and temporally correlated perturbations.The alternatives described in Section D can replace the reward and, where applicable, the TD structure.Source: §C.4 · pp. 18–19

C.5 Empirical-Bayes Noise-Variance Updates

  • Empirical-Bayes noise variances are updated from marginal likelihoods, with one shared estimate per particle-filter working model and an ensemble-averaged target for the RLSVI regression.The bottleneck-state variant instead estimates variance per ensemble member and averages those estimates; Stage-1 and Stage-2 redistribution regressions retain MRT-estimated fixed variances.Source: §C.5 · pp. 19–20

C.6 Weekly Update Schedule

  • After an initial week of independent 0.5-probability assignments, online updates begin Monday night of week 2 and recur weekly, with Monday decisions using the most recently available policy.At the weekly update, the algorithm refreshes perceived utility, CAE beliefs, required variance and redistribution models, and the RLSVI ensemble.Source: §C.6 · pp. 20–21

D RL Variants Evaluated in Pre-Deployment Experiments

The pre-deployment experiments compare candidate online RL configurations using a closed-loop simulator calibrated from the historical ADAPT MRT, with participant-level models of time-varying activity, affect, app utility, engagement, and partially observed states. Local stability diagnostics assess long-horizon simulations under extreme suggestion regimes and after treatment-effect calibration.

D.1 Baseline Configuration

  • The baseline online algorithm updates beliefs, constructs action-specific features, selects suggestions, and performs weekly RLSVI updates.It begins with uniform randomization in week 1 and places weekly updating between Monday decisions, with later within-week decisions using the updated learner.Source: §D.1 · p. 21

D.2 Combined Variants

  • The base online RL configuration uses γ = 0.9, no bottleneck-state TD loss, and no reward shaping; variants change γ, TD loss, and reward design against two baseline policies.The baselines are never sending suggestions and sending independently with probability 0.5 at each decision time; the reward designs include return-invariant shaping, with or without redistribution.Source: §D.2 · p. 22, Table 4 · p. 22

E ADAPT Simulation Environment

  • The closed-loop simulation testbed, built from the historical ADAPT MRT, evaluates candidate online RL algorithms before deployment in the ADAPT factorial trial.It combines models of the CAE-related process, latent perceived utility and engagement, and additional time-varying states to generate trajectories under a specified policy.Source: E ADAPT Simulation Environment · p. 22

E.1 Variables to Model

  • The simulator generates activity and affect mediators, weekly CAE and latent app utility, engagement and survey emissions, and additional time-varying states.The historical MRT modeled decisions across seven days, whereas the forthcoming trial schedules suggestions Monday through Saturday.Source: §E.1 · pp. 22–23

E.2 CAE-Related Process

  • Bayesian hierarchical linear models jointly fit step count, anticipated affect, weekly CAE, and short CAE, retaining participant-specific dynamics while borrowing information across sparse MRT trajectories.Weekly CAE summaries use exponentially decaying weights for within-week step counts and anticipated affect, with prespecified imputation when required values are unavailable.Source: §E.2 · pp. 23–24

E.3 Perceived-Utility Process

  • Perceived utility is modeled as a participant-specific latent state evolving with engagement mediators and weekly emissions; the joint likelihood represents survey nonresponse linked to modeled check-in completion.Participant-specific fits use a two-stage estimation procedure: pooled initialization on a coarse grid, followed by separate fits on a finer grid.Source: §E.3 · pp. 24–25, §E.3.0.0.1 · p. 25
  • Perceived-utility fitting evaluates gain under extreme suggestion and weekly-check-in settings, penalizes configurations near or above unit gain, and constrains residual scales to [0.03, 10].A lighter arctanh2(a1) penalty discourages a1 from approaching its box constraint.Source: §E.3.0.0.1 · p. 26

E.4 Additional Time-Varying Variables

  • Remaining time-varying variables are fit with participant-specific ℓ2-penalized regression, using cross-validation or a fixed unit penalty when observations are insufficient; daily active status uses recent active-day frequency and weekday/weekend status.The daily active-status model also includes an intercept.Source: §E.4 · p. 26
  • Prior step-count and suggestion-interaction predictors use past-seven-day summaries, with weekday/weekend and morning/afternoon indicators.The step-count summary is an exponentially weighted average of same-slot two-hour counts; the suggestion predictor is the fraction of delivered walking suggestions interacted with.Source: §E.4 · p. 26

E.5 Stability Diagnostics

  • Long-horizon simulations are screened for feedback parameterizations that could drive the process into clipping boundaries, with diagnostics repeated after treatment-effect calibration.Source: §E.5 · p. 26
  • For each participant, perceived-utility local gain combines persistence and mediator pathways and must satisfy |gE| < 1 under never- or always-suggest regimes and both weekly-check-in carry-over values.Source: §E.5 · p. 26
  • Weekly CAE models require |gY| < 1 under never- and always-suggest regimes, accounting for autoregression and step-count and anticipated-affect mediator pathways.Mediator derivatives include baseline and action-interaction terms.Source: §E.5 · p. 26

E.6 Generating Data from the Testbed

The testbed generates longitudinal trajectories from fitted participant models under candidate policies, adding participant-specific residual structure and cross-component dependencies. The paper defines a standardized reference-policy signal, varies benefit and burden pathways to create alternative environments, and describes a two-step simulator validation.

E.6 Generating Data from the Testbed

  • Trajectories follow the selected participant model’s temporal ordering under the evaluated policy, with participant-specific AR(1) residual bootstraps and partial pooling for sparse primary-outcome streams.Initial states come from observed MRT histories; perceived utility starts at E1 = 2.Source: §E.6 · pp. 26–27
  • Continuous variables are generated from fitted conditional means plus simulated residuals and clipped to supports derived from MRT training data; binary variables use fitted Bernoulli probabilities.Weekly perceived-utility items are mapped to the allowable 1–7 scale, while page views use a hurdle model for occurrence and positive intensity.Source: §E.6 · pp. 26–27
  • The generator explicitly links AM next-four-hour steps to PM prior-two-hour steps and suggestion interaction to page-view occurrence; the step-count residual link uses λ = 0.30, calibrated from 1,709 aligned MRT pairs.AM and PM page-view occurrence use separate offset-logistic calibrations, and other dependence propagates through simulated history and shared latent states.Source: §E.6 · p. 27

E.7 Standardized Testbed Effect

  • The standardized testbed effect compares a simulator-specific high-performing reference policy with never-suggest, standardizing each type’s improvement by its own trajectory-level variation.The reference is fit with offline conservative Q-learning and evaluated on independent test seeds; its complete weekly CAE information and lack of the online learner’s probability-clipping constraint make STE an upper-reference signal metric.Source: §E.7 · pp. 27–28

E.8 Simulation-Environment Variants

  • The vanilla testbed has population STE 0.479, while calibrated low- and high-signal variants have population STEs 0.275 and 0.813, respectively.The low-signal variant strengthens the burden pathway; the high-signal variant strengthens the benefit pathway, while other model components remain unchanged.Source: §E.8 · pp. 28–29

F Simulator Validation

  • Simulator validation first examines fitted-model residual dependence, then checks whether the connected simulator reproduces observed MRT trajectories.Source: F Simulator Validation · p. 29

F.1 Component-Model Diagnostics

Component diagnostics find little systematic mean discrepancy, but residuals retain participant- and process-specific temporal and cross-variable dependence. These findings motivate preserving residual variation and dependence in the stochastic generator.

CAE-related Gaussian model fit

  • The fitted residuals generally fluctuate around zero across four CAE-related components, but their dispersion and occasional large deviations differ by participant and component.The diagnostics then assess whether this remaining variation contains temporal or cross-variable dependence.Source: §F.1 · p. 29

Bernoulli component fit

  • Bernoulli models closely reproduce event rates for page views, Fitbit wear, daily check-ins, and suggestion interaction, with larger differences for weekly check-in completion and active-day status.Largest participant-level absolute discrepancies were 0.011, 0.010, 0.018, and 0.046, respectively, for the first four processes.Source: §F.1 · p. 29

Auxiliary continuous component fit

  • Continuous-component mean residuals remain near zero, while participant-specific residual dispersion persists and some low page-view errors reflect very few positive observations.Mean residual ranges were −0.067 to 0.038 for prior-two-hour steps, −0.026 to 0.034 for positive page-view intensity, −0.019 to 0.059 for helpfulness, and −0.022 to 0.041 for pleasantness.Source: §F.1 · p. 29

Temporal residual dependence

  • Residual temporal dependence remains beyond fitted conditional means and probabilities, varies in direction and magnitude across participants and processes, and means independent residual draws inadequately represent within-participant dynamics.Weekly CAE residual correlations were negative for all 21 participants with reportable values, ranging from −0.728 to −0.017.Source: §F.1 · p. 30

Cross-variable residual dependence

  • AM step-count residuals correlate positively with same-day PM step-count residuals for 25 of 27 participants, and AM suggestion-interaction and page-view-occurrence residuals are positive for all 27 with reportable correlations.Weekly and daily cross-variable associations vary substantially in sign and magnitude; the recurring decision-time relationships motivate corresponding dependence mechanisms in the generator.Source: §F.1 · p. 30

Residual dependence in the stochastic generator

  • The generator carries first-order serial dependence through participant-specific residual sequences, with partial pooling when residual information is limited, and represents two recurring cross-component relationships explicitly.AM next-four-hour step count informs same-day PM prior-two-hour step count, and suggestion interaction informs page-view occurrence.Source: §F.1 · p. 30

F.2 Simulator-Level Validation

The simulator is checked against MRT trajectories and then assessed over the planned 36-week horizon. It reproduces broad participant-specific patterns and remains numerically stable, while some localized changes and boundary concentrations remain.

Main trajectory reproduction

  • Participant-averaged 95% interval coverage was 92.7% for weekly CAE, 89.9% for next-four-hour step count, and 93.8% for anticipated affect; coverage averaged 97.7% across three engagement mediators.The mediator-specific coverages were 97.4% for page-view occurrence, 97.4% for Fitbit wear, and 98.4% for daily check-in completion.Source: §F.2 · p. 30, Figure 7 · p. 49, Figure 9 · p. 51
  • The simulator preserves broad between-participant differences and longitudinal patterns more consistently than localized changes or extreme observations.Examples include lower CAE for participant 18, higher CAE for participants 2 and 16, and broad declines for participants 9 and 25.Source: §F.2 · p. 30

Supporting trajectory reproduction

  • Supporting checks show broad participant-level patterns in short CAE, positive page-view intensity, assessment completion, usefulness, and pleasantness, with page-view comparisons less informative when positive occasions are few.Agreement is stronger for participant-specific levels and broad longitudinal patterns than for every local change over time.Source: §F.2 · p. 30, Figure 8 · p. 50, Figure 12 · p. 54

Long-horizon stability

  • All 8,400 simulated trajectories completed with finite outcomes over 36 weeks, although boundary concentration remained for latent perceived utility in participant 18 and low FourSC values in participant 22.The long-horizon assessment used never-send, always-send, and Bernoulli(0.5) suggestion policies.Source: §F.2 · p. 31, Figure 18 · p. 60, Figure 19 · p. 61

G Experimental Design and Additional Results

The evaluation compares seven RL configurations with random sending across the MRT-calibrated and alternative environments. It uses matched participant types and simulation randomness to assess cumulative weekly CAE over the planned horizon.

G Experimental Design and Additional Results

  • Policies are compared across the vanilla MRT-calibrated testbed and alternative simulation environments, with 200 replications per environment-policy combination.Candidate policies share sampled participant types within each replication and use common random numbers for exogenous stochasticity whenever possible.Source: G Experimental Design and Additional… · p. 63
  • Each simulated participant is followed for 36 weeks, with week 1 using 0.5-probability micro-randomization and weekly policy updates beginning in week 2.Non-learning reference policies follow the same observation process and schedule but do not update from accumulating data.Source: G Experimental Design and Additional… · p. 63

G.1 Evaluation Metrics

  • The primary criterion is cumulative weekly CAE over the 36-week final horizon, with the primary policy contrast computed within replication against never-send.The reported summary uses the Monte Carlo mean and its standard error.Source: §G.1 · p. 63

G.2 Additional Results

The additional results show that RL variants differ in how they allocate suggestions between morning and afternoon slots. Shaping with redistribution can produce slot-specific behavior, including for an illustrative participant with stronger fitted morning effects.

G.2 Additional Results

  • With γ = 0.9, standard RLSVI stays near 0.5 in both slots, while γ = 0.5 keeps morning sending near 0.5 and lowers afternoon sending to approximately 0.42 across environments.The morning–afternoon asymmetry changes little as the intervention signal strengthens.Source: §G.2 · p. 63, Figure 21 · p. 64
  • Shaping plus redistribution reallocates sending across slots with overall sending relatively stable: approximately 0.53 in the morning and 0.47 in the afternoon in vanilla and STE ≈0.8 environments.Little separation is observed under STE ≈0.2; the reported pattern is consistent with redistribution's advantage becoming more pronounced as the intervention signal strengthens.Source: §G.2 · p. 63, Figure 21 · p. 64
  • For illustrative user 13, shaping with learned redistribution raises morning sending to approximately 0.71 by study end while afternoon sending remains near 0.5.The fitted environment favors morning suggestions; the participant's fitted suggestion effect on anticipated affect is 0.047 in the morning versus 0.007 in the afternoon.Source: §G.2 · p. 63, Figure 22 · p. 64

Related papers

Loading 2610.36019v1…