PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Chemotherapy Control under Tumor Heterogeneity via Reinforcement Learning

Bereket Sitotaw Kidane, Md Samiul Haque Motayed, Shuo Wang

arXiv:2609.12264v1cs.LGcs.SYeess.SY

TL;DR

Tumor heterogeneity and resistance complicate patient-specific model-based chemotherapy control across diverse populations. The paper trains and benchmarks closed-loop DQN and TD3 dosing policies, finding higher mean tumor reduction with TD3 and tighter dosing consistency with DQN across a virtual cohort. The simulations assume full observation of tumor subpopulations, leaving sparse and noisy clinical measurements as a boundary for translation.

  • Problem

    Prior oncology RL studies had limited cohort-level generalization under heterogeneity and did not address partial observability in practice.

    Source: §I · p. 2

  • Method

    The paper trains closed-loop DQN and TD3 policies on a heterogeneous tumor model and benchmarks them against a PMP-derived open-loop solution.

    Source: §I · p. 2, §III · p. 3

  • Results

    84.07% vs. 82.78% mean tumor reduction for TD3 relative to DQN (p = 0.0078), while DQN has a tighter total-drug-dosage 95% confidence interval (±0.23 vs. ±0.40 for TD3).

    Source: §IV · p. 5

  • Takeaways & Limitations

    The simulations show a cohort-level efficacy–consistency trade-off: TD3 has higher mean tumor reduction, while DQN yields tighter dosing consistency.

    Source: §VI · p. 6

    All results assume full observation of the 21 tumor subpopulations, whereas clinical measurements are typically sparse and noisy.

    Source: §V · p. 6

Abstract

from arXiv · show

Designing effective chemotherapy regimens is hindered by tumor heterogeneity and drug resistance, which complicate the deployment of patient-specific model-based optimal control across diverse populations. We develop and compare closed-loop deep reinforcement learning (DRL) dosing policies with continuous (TD3) and discrete (DQN) action spaces trained on a high-dimensional heterogeneous tumor model. The DRL policies are benchmarked against a Pontryagin's Maximum Principle (PMP)-derived open-loop benchmark. We assess generalization under parametric heterogeneity using a 100-patient virtual cohort with plus or minus 10 percent uniform perturbations in growth and drug-sensitivity parameters. Across this cohort, TD3 achieves higher average tumor reduction, while DQN yields tighter inter-patient dosing consistency, revealing a clear efficacy-consistency trade-off in this study. Our simulations assume full observation of all tumor subpopulations; translation to sparse and noisy clinical measurements will require partial-observability formulations and/or state estimation. Overall, the results show that simulation-trained DRL can learn state-dependent feedback dosing policies that complement open-loop optimal control benchmarks.

I. INTRODUCTION

Tumor heterogeneity and evolving drug resistance complicate metastatic cancer treatment and motivate adaptive dosing. The paper develops and benchmarks deep reinforcement-learning policies, finding a cohort-level efficacy–consistency trade-off while assuming full tumor-state observation in simulation.

I. INTRODUCTION

  • Intratumor heterogeneity lets resistant subpopulations emerge dynamically, making tumor composition a moving therapeutic target.The introduction links aggressive elimination of drug-sensitive cells to competitive release, which can allow preexisting resistant cells to flourish.Source: §I · p. 1
  • PMP and direct optimal-control methods provide rigorous frameworks, but patient-specific model requirements and repeated replanning limit their practical use under uncertainty.PMP shooting can be sensitive to initialization, while high-fidelity personalized models may be infeasible.Source: §I · p. 1
  • Earlier oncology RL studies often used simplified or nominal models and lacked scalable high-dimensional methods, optimal-control benchmarks, or cohort-level generalization.The paper identifies partial observability as an additional practical challenge for prior policies.Source: §I · pp. 1–2
  • TD3 achieves higher mean tumor reduction, while DQN delivers greater consistency across a 100-patient virtual cohort.The study evaluates generalization under parametric heterogeneity and assumes full observation of tumor subpopulations.Source: §I · p. 2
  • The paper derives rewards from an optimal-control cost functional and compares DQN and TD3 against a PMP benchmark on a heterogeneous model.Its simulations use a 100-patient cohort; sparse and noisy clinical measurements remain outside the full-observation setup.Source: §I · p. 2

II. PROBLEM FORMULATION AND OPTIMAL CONTROL

The paper formulates chemotherapy for heterogeneous tumor subpopulations as an optimal-control problem, with tumor burden and drug exposure represented in the objective. PMP supplies a nominal open-loop benchmark, but repeated solves and parameter sensitivity constrain its use for replanning and uncertainty sweeps.

II. PROBLEM FORMULATION AND OPTIMAL CONTROL

  • The model represents a heterogeneous tumor as n subpopulations with resistance traits x_i, where 0 is maximally sensitive and 1 maximally resistant.Drug dosage u(t) is bounded between 0 and 3, and the continuum-of-traits PDE is approximated by a finite-dimensional ODE system.Source: §II · p. 2
  • The objective minimizes cumulative tumor burden and drug exposure, with γ penalizing drug exposure as a toxicity proxy.The terminal and running tumor burdens are weighted by α and β, respectively.Source: §II · p. 2
  • PMP yields necessary optimality conditions through a Hamiltonian and costate system, solved as a two-point boundary-value problem by shooting.The terminal costate condition is λ(T) = α; shooting produces a boundary–interior control profile.Source: §II · p. 2
  • Repeated PMP solves for replanning or large uncertainty sweeps are less convenient, and parametric uncertainty can make PMP sensitive to model parameters.The high-dimensional HJB alternative is computationally intractable because of the curse of dimensionality.Source: §II · pp. 2–3

III. DEEP REINFORCEMENT LEARNING FRAMEWORK

The DRL framework casts chemotherapy as an MDP and learns dosing policies from simulated trajectories. It compares discrete DQN with continuous TD3 while assuming full observation of all 21 tumor subpopulations.

III. DEEP REINFORCEMENT LEARNING FRAMEWORK

  • The MDP uses tumor subpopulations as state, drug dose as action, and negative incremental treatment cost as reward, including a terminal tumor-burden penalty.State transitions follow the tumor model, and episodes begin from prescribed or cohort-sampled initial conditions.Source: §III · p. 3
  • The study assumes full observation of all 21 tumor subpopulations, while identifying sparse, noisy biomarkers as motivation for future POMDP or state-estimation approaches.Source: §III · p. 3
  • DQN provides discretized dosing and TD3 fine-grained continuous control; both learn action values from simulated trajectories under parametric variation and are compared with PMP.The cohort-level evaluation covers 100 virtual patients.Source: §III · p. 3

A. Deep Q-Network (DQN) for Discrete Control

DQN approximates the optimal action-value function with a neural network and stabilizes training with replay and a delayed target network. The supplied training descriptions also contrast its discrete-action update with TD3’s actor–critic procedure.

A. Deep Q-Network (DQN) for Discrete Control

  • DQN approximates Q*(s, a) with a neural network and uses experience replay and a delayed target network to stabilize learning.Transitions are collected with an ϵ-greedy policy, and replay sampling breaks temporal correlations.Source: A. Deep Q-Network (DQN) for Discrete… · p. 3
  • DQN computes a one-step target from the frozen target network and minimizes squared temporal-difference error over replay-buffer mini-batches.The training loop periodically copies the online network parameters to the target network.Source: A. Deep Q-Network (DQN) for Discrete… · p. 3
  • TD3 adds noise to actor actions, updates twin critics using a minimum target estimate, and delays actor and target updates.The algorithm description specifies Polyak target updates.Source: A. Deep Q-Network (DQN) for Discrete… · p. 3

B. Twin-Delayed DDPG (TD3) for Continuous Control

TD3 provides continuous dose control through a deterministic actor and two critics. Its target update uses clipped double-Q learning, target-policy smoothing, and delayed policy updates.

B. Twin-Delayed DDPG (TD3) for Continuous Control

  • TD3 learns a deterministic actor policy and two critic networks for continuous dose control.The actor maps states to actions; the critics estimate action values.Source: B. Twin-Delayed DDPG (TD3) for… · p. 3
  • Its target update combines clipped double-Q learning, target-policy smoothing, and delayed policy updates.These techniques are used to mitigate Q-value overestimation.Source: B. Twin-Delayed DDPG (TD3) for… · p. 3
  • Bounded noise smooths the target action, while the lower of two target-critic values supplies the conservative estimate.The noise is clipped to [−c,c], and the target uses the minimum of the two critics.Source: B. Twin-Delayed DDPG (TD3) for… · p. 3

IV. NUMERICAL RESULTS

In the 100-patient cohort, TD3 achieved higher mean tumor reduction than DQN, while DQN showed lower outcome variability. On the nominal model, TD3 slightly exceeded the computed PMP benchmark in the discretized simulation; the authors caution this is not evidence of surpassing the continuous-time optimum.

IV. NUMERICAL RESULTS

  • Evaluation benchmarks DQN and TD3 against PMP on a nominal model, then tests one trained policy per agent across 100 patients with perturbed growth and drug-sensitivity parameters.The cohort perturbations use multipliers drawn from U[0.9, 1.1] in a 21-dimensional ODE model.Source: §IV · p. 3
  • The nominal PMP benchmark uses a bang-singular control profile and reports a resulting tumor trajectory and initial-to-final tumor distributions.Its figure also presents Hamiltonian derivatives associated with the optimality conditions.Source: Figure 1 · p. 4, §IV · p. 4
  • 89.1% vs. 87.7% tumor reduction: TD3 slightly exceeded the computed PMP benchmark in the discretized nominal simulation, not the continuous-time optimum.The PMP benchmark achieved 87.7% tumor reduction on the nominal patient model.Source: Figure 2 · p. 4, §IV · p. 4
  • 84.07% vs. 82.78% mean tumor reduction favored TD3 over DQN (p = 0.0078), while DQN had tighter dosage variability (±0.23 vs. ±0.40).TD3 also had lower final tumor burden (1.59 vs. 1.72 units).Source: Figure 3 · p. 4, §IV · p. 5
  • The Hamiltonian-derivative check assesses alignment with necessary optimality conditions, but the authors state it is not proof of optimality.The check reconstructs costates from simulated states and controls before calculating Hamiltonian derivatives.Source: Figure 3 · p. 4, §IV · p. 5
  • Both adaptive policies mitigated resistant rebound in simulations; Fig. 4 shows cohort tumor-burden trajectories and corresponding dose profiles.The figure distinguishes median, best-case, and worst-case outcomes; DQN doses appear as step functions.Source: Figure 4 · p. 4, §IV · p. 5

V. DISCUSSION

Across the modeled 100-patient cohort, TD3 achieves higher mean efficacy while DQN yields lower outcome variance, revealing an efficacy–consistency trade-off. The discussion relates this difference to policy behavior and notes that full observation limits clinical translation.

V. DISCUSSION

  • TD3 achieves higher mean efficacy (greater average tumor reduction, p = 0.0078), while DQN exhibits lower outcome variance across the 100-patient cohort.The authors frame the resulting efficacy–consistency trade-off as illustrative rather than a clinical recommendation.Source: §V · p. 5, Table IV · p. 5
  • DQN’s quantized actions align more closely with bang–singular structure and have lower-magnitude costate excursions, while TD3 makes smoother, earlier adjustments to evolving tumor composition.The reported average relationship is |λDQN| < 1.2∥λPMP∥; TD3’s additional flexibility may improve mean performance under parameter variation while respecting dose bounds.Source: §V · p. 5
  • One offline training phase takes approximately two hours on a standard CPU and produces a state-feedback policy with negligible online evaluation cost.Unlike a learned policy evaluation, PMP must be re-solved for closed-loop replanning or large multi-scenario uncertainty sweeps.Source: §V · pp. 5–6
  • The simulations assume full observation of all 21 tumor subpopulations, an idealization that motivates future POMDP formulations or state-estimation-based feedback for sparse, noisy clinical measurements.The discussion also identifies safety-filtering mechanisms as a possible future direction for treatment constraints.Source: §V · p. 6

VI. CONCLUSION

The conclusion presents a DRL framework for adaptive chemotherapy and evaluates DQN and TD3 against a PMP-derived benchmark, including generalization across a 100-patient virtual cohort. In these simulations, TD3 achieves higher mean tumor reduction while DQN provides tighter dosing consistency; future work will address partial observability.

VI. CONCLUSION

  • The study introduces a DRL framework and benchmarks closed-loop DQN and TD3 policies against a PMP-derived open-loop benchmark on a high-dimensional heterogeneous tumor model.It also evaluates empirical generalization under parametric heterogeneity across a 100-patient virtual cohort.Source: §VI · p. 6
  • In the simulations, TD3 achieves higher mean tumor reduction, whereas DQN yields tighter dosing consistency.The authors identify partial-observability formulations and state-estimation-based feedback as future work.Source: §VI · p. 6
Loading 2609.12264v1…