Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling

1 Introduction

• 42.6% pooled medication non-adherence in multimorbid populations motivates treatment planning that accounts for whether patients can and will follow recommendations.The introduction also notes that non-executed recommendations may consume resources, displace feasible interventions, and are associated with increased mortality.Source: §1 · p. 1
• Multimorbidity expands treatment plans and interaction risks, while patient heterogeneity includes kidney function, obesity, medication tolerance, and willingness to engage in lifestyle changes.The introduction highlights the HTN–T2DM dyad as a prevalent, consequential example.Source: §1 · p. 2
• Existing guidelines and computational approaches inadequately address patient-specific feasibility, coupled conditions, hierarchical decisions, and offline-learning overestimation bias.The proposed framework infers latent willingness and capacity from longitudinal BMI trajectories and uses cooperation-aware action masking as a feasibility constraint.Source: §1 · p. 2
• FAHOC separates therapeutic strategies into high-level options and disease- and intervention-specific factored policies, with retrospective evaluation on an approximately 50,000-patient multi-hospital cohort.The framework is evaluated through off-policy evaluation, guideline-concordance analysis, and stratification.Source: §1 · p. 2
• The model uses an SMDP because therapeutic strategies persist across encounters while medication adjustments occur visit by visit.Encounter data are temporally regulated into standardized intervals before leakage-free preprocessing.Source: §1 · pp. 2–3
• Held-out evaluation combines offline policy evaluation, clinician-action agreement, patient trajectories, and a Δ-concordance surface, followed by field-expert or clinician review.The review considers field literature, guidelines, and limitations of the accessible EHR.Source: §1 · p. 3

Patient preference and adherence

• Patient adherence is associated with attitudes, beliefs, and prior engagement, while a four-year trial reports better outcomes among adherent lifestyle-intervention participants.Reported outcomes include weight loss, BMI reduction, and glycemic and systolic blood-pressure improvements.Source: §2 · p. 3
• Prior preference-aware AI work includes learning a mixture of community-facing and provider-facing signals, but patient preference remains an open challenge in clinical RL.The review also cites work emphasizing autonomy, treatment burden, and personal health goals in recommendation systems.Source: §2 · pp. 3–4

Multimorbidity Treatment Planning

• Existing sequential multimorbidity RL approaches commonly lack a mechanism that constrains treatment recommendations by patient preference.The review describes limitations in flat joint-action policies, condition-specific agents, and approaches without hierarchical decision structures.Source: §2 · p. 4
• No existing computational framework is identified as addressing HTN–T2DM with both hierarchical policy structure and patient preference as a structural constraint.The review cites measurable biomarkers, shared treatment trade-offs, and disease burden as reasons the dyad is a consequential modeling target.Source: §2 · p. 4

Offline RL and Hierarchical Reinforcement Learning

• Offline clinical RL faces inflated Q-values for out-of-distribution actions; CQL regularizes action values and DDQN separates action selection from evaluation.The review notes particular concern for rare treatment combinations and sparse-reward clinical settings.Source: §2 · p. 4
• Hierarchical RL separates a high-level controller that selects temporally extended sub-policies from a low-level controller for primitive actions.The review connects hierarchy to clinicians' strategy-then-adjustment reasoning and notes its potential value in sparsely represented EHR action spaces.Source: §2 · p. 4
• The option-critic architecture learns intra-option policies and termination functions rather than relying on manually designed sub-goal structures.In this study, learned options correspond to single-target and multi-target treatment strategies.Source: §2 · pp. 4–5
• Prior healthcare HRL examples use monolithic within-option actions, while factored representations can decompose clinically distinct action dimensions.The review also describes linear Q-function decomposition as a way to assign independent Q-heads to treatment subdimensions.Source: §2 · p. 5

3 Model formulation

• The model defines an SMDP with states, factored actions, options, transitions, rewards, and discounting; the options and their multi-visit models supply its semi-Markov structure.The primitive state-action-transition-reward process alone is equivalent to a standard MDP.Source: §3 · p. 5
• Patient state representation uses an 18-feature vector selected with expert feedback to reflect biomarkers and other treatment-relevant information.The feature set is summarized in Table 1.Source: §3 · p. 5, Table 1 · p. 5
• Cooperation is inferred from BMI trajectories, and BMI intervention actions are assigned only to cooperative patients who are overweight or obese at that visit.The cooperation label is computed after train-only imputation; BMI actions are inferred because they are not explicitly recorded in most EHRs.Source: §3 · p. 6
• The factored action space has 18 joint actions across T2DM, HTN, and BMI intervention dimensions, enabling lower-dimensional Q-functions and per-component credit assignment.The medication adjustment dimensions use {-1, 0, 1}, while the BMI intervention dimension uses {0, 1}.Source: §3 · p. 6
• Two options represent single-target management and simultaneous T2DM–HTN management, while factored actions specify medication adjustments within the active strategy.The high-level policy chooses an option across encounters, and the intra-option policy selects factored actions visit by visit.Source: §3 · p. 6
• Option durations are random and state-dependent, and the model requires a minimum two-transition commitment before reassessing an assigned strategy.The option partition is based on clinical state rather than demonstrated actions to avoid data leakage.Source: §3 · pp. 6–7
• The transition kernel represents visit-level state changes as a function of factored actions and is approximated from observed EHR trajectories.Options compose these primitive transitions into treatment regimes at a coarser temporal scale.Source: §3 · p. 7
• The reward combines QALY gain, potential-based reward shaping, and improvement or worsening terms, then clips the total to [-1, 1].The reward uses SBP, A1C, and age-related state features.Source: §3 · p. 7

4 Structural Properties

• The structural analysis aims to bound the gap between learned and optimal treatment policies and characterize attainable-value differences between cooperative and non-cooperative patients.The paper states that proofs are provided in Appendix I.Source: §4 · p. 7

4.1 Factorization Error Analysis

• Under bounded rewards, the optimal Q-function is uniformly bounded by the geometric sum of the maximum absolute immediate reward.The factorization-error analysis also assumes factored actions and regularity conditions for reward and transition behavior across action domains.Source: §4.1 · pp. 7–8

1. Q∗: True optimal Q-function (fixed point of T),

• The reward-interaction residual vanishes quadratically with within-option action dispersion under the Hessian bound, and is exactly zero for this study’s outcome-based reward.Source: §1 · pp. 9–10
• The factored Q approximation error decomposes into reward-factorization error and additive-architecture error; the former is zero under this study’s clinical reward.The remaining error reflects cross-domain value interactions induced by transition dynamics.Source: §1 · pp. 9–10
• The additive-architecture error is small when cross-domain coupling between T2DM and HTN treatments is limited.This error arises because the additive architecture cannot represent transition-induced cross-domain value interactions.Source: §1 · p. 10

4.2 Cooperation Value Dominance

• Cooperation is inferred from longitudinal BMI and gates BMI actions: eligible cooperative high-BMI patients have 18 feasible actions versus 9 for non-cooperative patients.In the study, 32.1% of transitions met all three BMI-beneficial criteria.Source: §4.2 · pp. 10–11
• Allowing BMI co-treatment weakly improves optimal value and strictly improves it when BMI-beneficial states are reached with positive probability.Source: §4.2 · pp. 10–11

• FAHOC combines option-critic strategy selection, factored treatment actions, and action masking to learn preference-compliant policies in a combinatorial clinical action space.Conservative Q-learning regularizes intra-option heads, while a multi-term termination loss augments the option-critic gradient.Source: §5 · p. 11
• A shared state encoder feeds a high-level critic for Single-Target or Multi-Target options, per-option disease-specific Q-heads, and termination heads governing strategy persistence.Source: §5 · p. 11, Figure 2 · p. 12
• The weighted Q composition assigns 0.4 each to T2DM and HTN and 0.2 to BMI, giving BMI an independent gradient signal.These weights reflect the stated relative clinical priority of glycaemic and blood-pressure control over BMI management.Source: §5 · p. 11
• Training uses Double DQN targets with option-utility blending, masks inadmissible actions, and updates termination heads separately from the shared encoder.The method also clips targets and gradient norms to address instability under offline distribution shift.Source: §5 · p. 12

6.1 Data and Preprocessing; 6.2 Training Convergence

• 47,298 patients formed the final cohort; validation action accuracy rose from 66.9% at epoch 1 to a peak of 75.3% at epoch 60.The cohort came from five Southeast U.S. hospitals, and training used 33,108 patients, with separate validation and test sets of 7,095 patients each.Source: §6.1 · p. 13, §6.2 · p. 13

6.3 Clinician Action Agreement

• 69.8% overall action accuracy exceeded the 5.6% random baseline, while BMI recall was 99.98% and option-selection accuracy was 53.8%.Source: §6.3 · p. 13, Table 3 · p. 14

6.4 Off-Policy Evaluation

• 0.669 DR policy value exceeded the −0.133 observed clinician baseline on the held-out test set.WIS had a wider interval ([−0.972, −0.102]), which the authors associate with importance-sampling variance when the deterministic agent diverges from clinician actions.Source: Table 4 · p. 15, §6.4 · p. 15

6.5 Illustrative Analysis of Optimal Policy Behavior

• In four test trajectories, the agent maintained stable medication strategies, avoided clinician titration oscillations, and recommended BMI reduction only for cooperative patients.The examples include both Single-Target and Multi-Target options.Source: §6.5 · pp. 15–16, Figure 3 · p. 16

6.6 Patient Preference Masking Compliance

• 100% safety compliance across 89,556 test transitions came with zero mask violations; among eligible patients, BMI precision was 92.2%, recall 99.98%, and consistency 99.9%.Source: §6.6 · p. 16

6.7 Guideline Concordance Analysis

• Across most of the SBP/A1C threshold grid, guideline-concordance difference ranged from −0.03 to +0.14; above both reference thresholds, it reached +0.14.The validation surface deviated by less than 0.01 from the test surface.Source: Figure 4 · p. 17, §6.7 · p. 17

6.8 Stratified Analysis: Uncontrolled vs. Controlled Trajectories

• DR value was 0.889 versus clinician reward ≈0.000 for uncontrolled trajectories, and 0.511 versus −0.042 for controlled trajectories.Safety compliance was 100% in both strata.Source: §6.8 · pp. 17–18, Table 5 · p. 18

6.9 Comparison with Baseline DDQN and Clinician Action Agreement

• HRL’s overall agreement gain over flat DDQN was +0.6%, with component gains of +0.9 pp for T2DM, +1.6 pp for HTN, and +1.9 pp for BMI.Source: Table 6 · p. 18, §6.9 · pp. 18–19

Preference as a constraint that binds

• Across 89,556 held-out transitions, action masking produced zero preference violations, while cooperation classification reached F1 = 95.9%.The policy learned when to recommend BMI reduction despite having no BMI term in its reward.Source: §7 · p. 19

Structure without sacrifice

• Against standard DDQN, the framework showed no loss of QALY and accuracy advantages of HTN +1.66% and BMI for cooperative patients +1.9%.The comparisons use the same state, action space, reward, and training data.Source: §7 · p. 19

The hierarchy also operates as designed

• The hierarchy selected latent clinical intent above chance at 53.8%, and learned termination probability ¯β ≈0.2; options were initialized from documented clinical thresholds.The discussion reports that options avoid collapse and produce multi-encounter strategy persistence.Source: §7 · pp. 19–20

The architecture additionally needed to address a difference in timescales

• The factored BMI head, age-stratified QALY reward with γ = 0.97, and persistent option strategy address BMI’s slower effects across encounters.For two illustrated patients, BMI reduction was recommended earlier than clinician practice; standard DDQN had 1.9% lower BMI recommendation accuracy in Table 6.Source: §7 · p. 20

Converging offline evidence of improvement

• 0.669 DR estimate of policy value versus −0.133 clinician baseline, supported by guideline concordance and stratified analyses, provides converging retrospective evidence of improvement.The guideline concordance advantage is reported at and above ACC/AHA 2017 and ADA 2025 thresholds; DR is described as a conservative lower bound.Source: §7 · p. 20
• The advantage was larger for uncontrolled disease (DR = 0.889) than controlled disease (DR = 0.511), and the authors call for prospective validation.The guideline concordance analysis is presented as an external assessment unrelated to the reward.Source: §7 · p. 20

Preference-aware treatment planning utilizes wasted capacity for usable care

• F1 = 95.9% identification of patients open and capable of lifestyle modification from standard EHR data can help direct limited lifestyle-intervention resources.The authors contrast this with planning pharmacological alternatives upfront for others, rather than discovering non-engagement after resources are spent.Source: §8 · p. 20

Concentrated policy return on uncontrolled patients stratum informs sequence of deployment

• DR = 0.889 among patients with above-average disease condition versus DR = 0.511 among controlled patients suggests a deployment sequence prioritizing less controlled groups.The passage frames this as concentrating policy return under budget limits.Source: §8 · p. 20

Mandatory constraints require architectural enforcement instead of incentive schemes

• Across 89,556 held-out transitions, architectural masking produced zero preference violations, supporting rigid constraints for requirements that must not be traded off.The passage contrasts structural feasibility constraints with reward penalties, which make non-compliance statistically less likely.Source: §8 · p. 21

Offline evaluation serves as a economical stage gate

• Offline evaluation estimated policy value at 0.669 against a clinician baseline of −0.133 before patient exposure, offering a stage gate before prospective validation.The pipeline includes off-policy value estimation, agreement with clinician practice records, and guideline concordance.Source: §8 · p. 21

9 Conclusion

• FAHOC incorporates patient preference as a structural feasibility constraint in hierarchical RL for multimorbidity management.The conclusion says the architecture generalizes naturally, while noting that this study evaluated the HTN-T2DM pair.Source: §9 · p. 21
• The framework infers cooperation from longitudinal BMI trajectories, enforces preference during preprocessing and training, and uses clinically informed options for temporally coherent treatment regimes.Its theoretical framework includes cooperation-value dominance and a global factorization error bound.Source: §9 · p. 21
• The BMI-based cooperation label depends on trajectory length and imputation quality, the two-option hierarchy limits expressivity, and offline evaluation cannot substitute for prospective validation.The conclusion identifies prospective validation in a live clinical setting as the most important future direction.Source: §9 · p. 21
• The framework is positioned as a recommendation layer for clinician review rather than an autonomous prescriber.The passage notes that a richer option and intervention set defined collaboratively with clinicians could support an EHR-integrated policy.Source: §9 · p. 21
• The analyzed de-identified EHR dataset is not publicly available; access is subject to institutional review board approval.The codebase is stated to be released upon acceptance.Source: §9 · p. 21

Appendix A Reward Function

• The reward combines QALY change with short-term shaping and biomarker improvement bonuses or worsening penalties.QALY change is based on age-group-specific utility and A1C/SBP values; the bonus applies to improving biomarkers above target, while penalties address specified sharp worsening.Source: Appendix A · p. 25

Appendix B Data processing medications encoded for HTN and T2DM intensity

• Table 8 lists medications for HTN and T2DM by class as encoded in the model.The supplied caption identifies the table’s scope but does not provide medication entries.Source: Table 8 · p. 26

Appendix C Training Loss and Targets

• Training uses Double DQN low-level targets and a high-level target that incorporates termination probability; termination loss adds bias, entropy, and quadratic-anchor regularizers.The termination objective encourages ending an option when it underperforms its option-value baseline; its anchor target is ¯β=0.30.Source: Appendix C · pp. 26–27

Appendix D Cooperation (state feature) and BMI recommendation (action) inference algorithm

• The BMI cooperation procedure assigns a per-patient label from BMI trajectory patterns, and the BMI action procedure assigns a recommendation based on visit index, cooperation, and BMI category.For example, the cooperation procedure uses a downward trend for overweight or obese patients with at least three visits; the action procedure sets the first-visit BMI action to zero.Source: Appendix D · p. 28

Appendix E Solution Algorithms

• The solution algorithms compose factored action values, construct clinical action masks, initialize a stratified replay buffer, and train the hierarchical DDQN with option-critic.The algorithm description also identifies separate termination-head updates and a combined low- and high-level critic backward pass.Source: Appendix E · pp. 28–31

Appendix F Delta Guideline Concordance Function

• The guideline-concordance appendix defines intensity-aware actions from biomarker thresholds and medication intensity, and Figure 5 tracks training losses, validation accuracy, and termination probability.The figure caption states that the dashed red line marks the target ¯β ≤0.5.Source: Appendix F · p. 31, Figure 5 · p. 32

Appendix G Training Curves

• 75.3% validation action accuracy was reached at epoch 70, then stabilized near 74% at convergence.T2DM and HTN component accuracies converged to approximately 83% and 82%, respectively.Source: Appendix G · p. 32
• The low-level intra-option Q-loss fell from 0.1226 to 0.0587, the high-level option-value loss from ≈0.0013 to 0.0006, and termination loss from 0.1279 to 0.0004.Average termination probability remained below 0.3.Source: Appendix G · p. 32

Appendix H Factored Action Hierarchical Option-Critic (FAHOC) with Action Masking Network Architecture

• FAHOC’s shared state encoder feeds option-value, factored intra-option, action-mask, and termination heads.The two options are Single-Target and Multi-Target; the mask assigns −∞ to BMI action 1 for non-cooperative patients or those with normal BMI.Source: Figure 6 · p. 33, Appendix H · p. 34

Appendix I.1

• The optimal Q-function is bounded in magnitude by Rmax/(1−γ) under discounted rewards.The proof uses the discounted sum of future rewards and a geometric-series bound.Source: Structural Properties: FAHOC Error Bound · pp. 34–35

Global Factorization Error Bound

• The additive factored Q-function’s approximation error is bounded using reward error and cross-domain interaction terms under the stated assumptions.The total bound combines the component bounds; the proof identifies cross-domain remainder terms as the source of architectural error.Source: Global Factorization Error Bound · pp. 36–37

Appendix I.2 Cooperation Value Dominance

• Because every policy feasible without cooperation remains feasible with cooperation, the optimal value under cooperation cannot be lower.Strict superiority additionally requires a reachable state with an available action whose expected next-state value is higher.Source: Appendix I.2 · p. 37, Conditions for Strict Superiority · p. 38

Conditions for Strict Superiority

• Under the stated strict-superiority condition, cooperative patients have higher optimal expected value than non-cooperative patients.The proof constructs a policy that differs from the non-cooperative optimum only at a reachable state, where it selects the beneficial action.Source: Conditions for Strict Superiority · p. 38

Appendix J.1

• The interaction estimator uses second-order mixed finite differences on empirical next states, aggregating estimates across action-domain pairs.It uses 414,880 training transitions globally and 500 bootstrap resamples of 80% of transitions for percentile 95% confidence intervals.Source: Estimating the Transition… · p. 38
• Under Assumption 4, the finite-difference estimator approximates the mixed derivative with O(hk + hℓ) discretization error and Op(n−1/2) sampling error.The action grid is finite and observed at all four vertices, so the stated estimator has no additional discretization bias beyond empirical mean estimation.Source: Estimating the Transition… · p. 38
• BMI was the worst-case dimension for all three action-domain pairs, with worst-case base actions (1, 1, 1), (0, 2, 0), and (2, 1, 0).Bootstrap confidence intervals had coefficients of variation below 12%; the global-to-binned gap was attributed to finite-sample variance within bins rather than genuine state-dependence.Source: Estimating the Transition… · p. 39, Table 10 · p. 39

Computing the Bound

• The worst-case bound’s bootstrap 95% CI was [490.3, 700.5].This is a supremum over state–action pairs; the reward-level bound is complementary and tighter when εr is small.Source: Computing the Bound · p. 39

Appendix K Strict superiority criteria level count

• The cooperative, overweight-or-obese, uncontrolled profile covered 190,904 transitions (32.1%) and 18,761 patients (39.7%).Coverage was 32.1%, 32.7%, and 31.5% across the three splits.Source: Appendix K · p. 39, Table 11 · p. 39
• The subset empirically confirmed condition (1), establishing that Proposition 1 holds across nearly 40% of patients and one-third of all encounters.Source: Appendix K · p. 39