PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavioral Cloning Outperforms Entropy-Regularized RL: Critic-Driven Failure of Actor-Critic Methods on Adaptive Tumor Treatment

Aleksandar Dimitrov, Giacomo Spigler

arXiv:2609.06667v1cs.LG

TL;DR

Learned tumor-dosing policies are often evaluated against historical or heuristic comparators, which cannot establish whether they recover a near-optimal schedule. The paper builds a numerical reference from optimal-control structure and tests cloning and actor-critic learning against a sustained-cure criterion. Cloning reproduces cure, but actor-critic fine-tuning collapses the policy, with critic action rankings implicating misvaluation.

  • Problem

    Evaluations against historical or heuristic comparators show relative performance but cannot establish whether learned dosing policies recover a near-optimal schedule.

    Source: §1 · p. 2

  • Method

    The authors study a three-population tumor-control model and construct a numerical reference controller with the bang-bang and singular-arc structure indicated by optimal-control analysis.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    BC achieved a 100% durable cure rate, while every tested actor-critic configuration reached 0% final sustained cure; the collapsed critic ranked its action above the curative action in 96% of curative-trajectory states.

    Source: §1 · p. 2, §4.4 · p. 7

  • Takeaways & Limitations

    Where a theory-derived controller is available and its assumptions are appropriate, it should serve as the reference for evaluating learned policies.

    Source: §5 · p. 9

    The clinical value of the adaptive-therapy-like equilibrium depends on the treatment objective and model assumptions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Adaptive dosing requires policies that reduce tumor burden without excessive toxicity. Learned dosing policies are typically judged against historical or heuristic comparators, which cannot show whether a policy has found the best behavior available. We instead study a three-population tumor-control ODE in which optimal-control analysis fixes the form of a good schedule -- bang-bang dosing punctuated by a singular arc -- and construct a numerical controller of that form as a proxy for near-optimal behavior. Judged against this reference under a sustained-cure criterion -- 200 consecutive days below 5% carrying capacity -- Soft Actor-Critic (SAC) trained from scratch never reaches cure. Behavioral cloning (BC) of the reference reproduces it (100% sustained cure, 30/30 seeds), but SAC fine-tuning of the cloned policy destroys it across five entropy coefficients, and TD3 and BC-regularized SAC fail identically; the pattern persists under multiplicative pharmacokinetic action noise. Along curative trajectories the post-collapse critic ranks the collapsed-policy action above the reference action in 96% of states, concentrated in the maintenance phase, and the policy settles into a non-curative adaptive-therapy equilibrium. The reference is what makes this legible: against a heuristic comparator the fine-tuned policy would read as a competent controller rather than a failure.

1 Introduction

The paper asks whether learned dosing policies can recover a near-optimal tumor-treatment schedule, rather than merely outperform historical or heuristic comparators. Against a reference controller, it finds that behavioral cloning succeeds while actor-critic fine-tuning destabilizes the cloned policy.

1 Introduction

  • Maximum tolerated dosing can release drug-resistant cells from competition, while adaptive therapy preserves sensitive cells to suppress resistant cells over time.Adaptive therapy trades immediate tumor reduction for long-term control.Source: §1 · pp. 1–2
  • Evaluations against historical or heuristic comparators can show relative improvement but cannot establish whether a learned policy recovers a near-optimal schedule.The paper frames this as a gap in prior evaluations of learned treatment-scheduling policies.Source: §1 · p. 2
  • The reference schedule follows a bang-bang pattern punctuated by a singular-arc maintenance dose, as indicated by optimal-control analysis.The authors construct a numerical controller of this form as a proxy for near-optimal behavior.Source: §1 · p. 2
  • Behavioral cloning of the Pontryagin-derived teacher achieved a 100% durable cure rate, but SAC fine-tuning destabilized it across all tested entropy coefficients.TD3 and BC-regularized SAC showed the same collapse, and the collapsed critic preferred its action over the curative action in 96% of states along the curative trajectory.Source: §1 · p. 2
  • The paper identifies a sustained-cure metric, actor-critic fine-tuning behavior, and critic Q-value analysis as central contributions.The metric is intended to expose specification gaming in standard tumor-RL cure criteria.Source: §1 · p. 2

2 Related Work

Prior work spans adaptive therapy and optimal-control analysis of tumor treatment, alongside RL applications in treatment scheduling. The paper distinguishes its evaluation by using an optimal-control-derived reference rather than relying only on historical or heuristic comparisons.

2 Related Work

  • The cited strategy aims to suppress resistance rather than continuously maximize cell kill, and it has also been evaluated in a small prostate-cancer study.Optimal-control analysis, particularly Pontryagin’s Minimum Principle, can yield bang-bang controls and, in some models, singular arcs.Source: §2 · p. 3
  • Prior RL treatment-scheduling studies commonly compare learned policies with historical, clinical, or heuristic comparators, which cannot test recovery of a known near-optimal schedule.Applications include HIV, sepsis, and chemotherapy.Source: §2 · p. 3
  • SAC is an entropy-regularized off-policy actor-critic method, while TD3 removes entropy and DAPG-style methods use demonstrations to regularize policy learning.Maximum-entropy RL has a formal robustness interpretation for certain disturbances to dynamics and rewards.Source: §2 · p. 3
  • The study evaluates multiple seeds and algorithmic variants because deep-RL results can be seed-sensitive.RLPD and Cal-QL are identified as relevant future baselines outside the study’s scope.Source: §2 · p. 3

3 Methods

The methods combine a three-population tumor-control model and a stricter sustained-cure criterion with a numerically tuned, optimal-control-structured teacher. The study clones that teacher, fine-tunes the clone with actor-critic methods, and probes critic action values along curative trajectories.

3.1 Environment and sustained-cure metric

  • TumorEnvV2 models sensitive cells, resistant cells, and accumulated toxicity, and defines cure as 200 consecutive days below 5% of carrying capacity with resistant cells a minority of residual tumor.The standard five-day threshold can be satisfied by a transient sensitive-cell crash; the model uses selective drug action and mutation from sensitive to resistant cells.Source: §3.1 · pp. 3–4

3.2 Pontryagin-derived teacher and behavioral cloning

  • The three-phase teacher waits at zero dose, crashes at maximum dose subject to toxicity, then maintains at dose 0.25 with a regrowth guard.Its structure follows optimal-control analysis, but its constants were tuned rather than derived, so it serves as a proxy rather than a certified optimum.Source: §3.2 · p. 4
  • Behavioral cloning trains an MLP matching the SAC actor architecture on 500 teacher episodes, about 158,000 transitions, then evaluates on 20 held-out seeds.Matching architectures lets fine-tuning changes be attributed to policy weights rather than representation differences.Source: §3.2 · p. 4

3.3 SAC fine-tuning and interventions

  • The interventions vary SAC entropy coefficients, test TD3 and BC regularization, and add multiplicative pharmacokinetic action noise with σ = 0.20.SAC from scratch uses a larger learning rate and a 100,000-step budget; teacher robustness is also tested under noise and randomized initial conditions.Source: §3.3 · pp. 4–5

3.4 Critic Q-value analysis

  • Critic analysis compares the minimum twin-critic estimates for teacher and collapsed-policy actions at states along the teacher’s curative trajectory.The evaluation applies the critic’s [−1, 1] action scaling and repeats the analysis over ten randomized initial conditions.Source: §3.4 · p. 5

4 Results

Behavioral cloning matches the teacher’s sustained-cure performance, but actor-critic learning repeatedly destroys it. The collapsed critic favors its own action on curative trajectories, especially during maintenance, while the policy settles into non-curative control.

4.1 Behavioral cloning cures; actor-critic fine-tuning collapses

  • 100% sustained cure for the teacher and BC at fine-tuning onset fell to 0/30 after 30k SAC updates; scratch SAC recorded no cure across three seeds.The BC+SAC result is the same run after fine-tuning, and every final episode ended in timeout.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • Across 15 SAC fine-tuning runs, all five entropy coefficients fell from 100% peak cure to 0% within 1,000 gradient steps.The five curves coincide; changing α across four orders of magnitude did not prevent collapse.Source: §4.1 · p. 6, Figure 1 · p. 6
  • With σ = 0.20 multiplicative pharmacokinetic action noise, the five α settings still collapse from 40% to 0% cure (n = 15).The BC initialization peaks at 40% cure in this stochastic setting.Source: §4.1 · p. 7

4.2 Critic misvaluation along curative trajectories

  • 96% of teacher-trajectory states had a higher estimated value for the collapsed-policy action (mean gap +1.26), with the largest gap during singular-arc maintenance.Across ten randomized initial conditions, mis-ranking occurred in 94.8% ± 1.4% of states; the mean gap was +1.83 during maintenance versus +0.35 during drug-free waiting.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 The failure is a maintenance trap, and BC does not inherit the teacher’s robustness

  • The collapsed policy begins with a low non-zero dose and remains non-curative for 700 days near S ≈ 30% K and R ≈ 3 × 10^5.The teacher and BC-only policies instead follow the wait-crash-maintain structure.Source: §4.3 · p. 7
  • Under Gaussian observation noise, BC reaches 5% cure at σ = 0.05 and 0% beyond, versus teacher cure of 100% through σ = 0.10 and 90% at σ = 0.20.Thus BC does not inherit the teacher’s observation-noise robustness in this setting.Source: §4.3 · p. 7

4.4 Ablation overview

  • Every tested actor-critic configuration reached 0% final sustained cure; TD3 and BC-regularized SAC collapse despite removing entropy or adding imitation anchoring.DAPG-style SAC collapses for every tested β, including β = 10.0, where the anchor term is comparable in magnitude to per-step rewards.Source: §4.1 · pp. 6–7, §4.4 · p. 7

5 Discussion

A curative reference makes actor-critic collapse legible as failure rather than plausible chronic control; critic misvaluation is consistent with the collapse but is not proven to cause it.

5 Discussion

  • Against historical or heuristic baselines, the reference reveals actor-critic fine-tuning converging to chronic control instead of a curative trajectory.The critic analysis associates collapse with mis-ranking teacher and collapsed-policy actions, particularly near the singular arc, but does not by itself prove critic-mediated drift.Source: §5 · p. 8
  • BC preserves the teacher’s behavior under clean observations but neither benefits from tested fine-tuning nor inherits the teacher’s observation-noise robustness.It serves as a diagnostic initialization, not a deployment solution.Source: §5 · p. 8
  • Observation perturbations are distinct from the dynamics and reward disturbances considered in maximum-entropy RL robustness results.Source: §5 · p. 9
  • The adaptive-therapy-like equilibrium is interpretable as an alternative to cure, but its clinical value depends on the treatment objective and model assumptions.Source: §5 · p. 9
  • Where a theory-derived controller’s assumptions are appropriate, it should serve as a reference; RLPD and Cal-QL are relevant future baselines.Source: §5 · p. 9
  • The evidence comes from one ODE model and parameterization, three seeds per sweep condition, and a tuned approximation rather than a certified optimum.Broader replication, additional algorithms and parameterizations, and clinically informed settings remain necessary.Source: §5 · pp. 9–10

6 Conclusion

Against a near-optimal reference, SAC from scratch did not discover cure, BC reproduced sustained cure, and actor-critic fine-tuning collapsed to non-curative maintenance. The paper concludes that durable outcome criteria and theory-derived references reveal failures that heuristic comparisons may miss.

6 Conclusion

  • Against a near-optimal reference, SAC from scratch did not discover cure, BC reproduced sustained cure, and SAC, TD3, and BC-regularized SAC fine-tuning collapsed to non-curative maintenance.Source: §6 · p. 10

Disclosure of Interests.

Loading 2609.06667v1…