Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning

1 Introduction

• Clinical agents must acquire and interpret information before acting because treatment benefits and risks depend on the patient, and multiple treatment plans can remain defensible.Existing benchmarks often provide curated scenarios with decision-relevant information already supplied; treatment choices can reflect tradeoffs among efficacy, toxicity, preferences, and other constraints.Source: §1 · p. 1
• Retrospective records do not specify the information to acquire at each intermediate state or the inference to make, and observed treatment actions need not be uniquely correct.The paper therefore frames supervision as reward over a sequence of information acquisition, inference, and first-line treatment selection.Source: §1 · p. 2
• The paper treats clinical reward verifiability as a continuum, based on how reliably the environment can determine whether behavior meets clinical criteria.Its central question is how to design strongly verifiable RL environments when patient state is partially observed and clinical reward is only partially verifiable.Source: §1 · p. 2
• The paper formulates oncology treatment planning as interactive RL under partial observability, requiring agents to acquire patient-specific information before recommending treatment.Its patient-specific, process-aware reward assesses information gathering, treatment selection, and alignment with documented clinical rationale.Source: §1 · p. 2
• The RL-trained Llama-3.1-8B policy achieves higher treatment-planning reward than GPT-5.2 in the same patient environment, retrieval interface, turn budget, and evaluation procedure.The paper suggests domain-specific environments can help smaller models acquire specialized sequential decision-making behavior not exhibited by larger models zero-shot.Source: §1 · p. 2
• The policy improves overall reward while reducing queries, and its information-seeking trajectories, rationales, and recommendations align with oncologists’ documented processes.A preliminary clinician comparison covers only a small number of cases and is treated as suggestive, not evidence of clinician-level performance.Source: §1 · p. 2

Interactive clinical agents and sequential clinical decision-making

• Many interactive clinical RL environments target diagnosis, whereas this work places the patient state under partial observability and targets evidence acquisition and interpretation before first-line treatment.Prior systems include multi-turn patient dialogue agents, diagnosis-focused information seeking, and simulated encounters.Source: §2 · p. 2

Reinforcement learning under partially verifiable clinical feedback

• Unlike guideline-focused approaches such as GatorOnco, this work uses a terminal but process-aware reward grounded in patient-specific clinician notes across the full interactive trajectory.The reward covers information acquisition, treatment selection, and clinical reasoning because intermediate actions and terminal treatments may lack unique ground-truth labels.Source: §2 · p. 3

3 Clinical Problem Formulation

• Colon cancer treatment planning requires integrating disease, biomarker, treatment-history, functional-status, comorbidity, preference, and treatment-intent information under incomplete information.Oncologists iteratively acquire and interpret evidence until sufficient information supports an initial management strategy.Source: §3 · p. 3
• The paper rewards agents against documented per-patient decision trajectories rather than explicit guideline adherence, since observed practice can deviate from guidelines and multiple options may be reasonable.The trajectory provides evidence of one realized decision, not a unique trace of information gathering and reasoning with stepwise labels.Source: §3 · p. 3

4 Methodology for Clinical Reinforcement Learning Environment Design

• The system has a trainable clinical agent, a deterministic action parser, a frozen patient-record mediator, and a frozen judge in an interactive patient environment.The agent begins with a one-line summary, retrieves patient-specific information through investigation actions, or ends with a treatment recommendation; the judge scores the completed trajectory.Source: §4 · p. 3, Figure 1 · p. 4

4.1 Data & Cohort Construction

• The dataset comprises 950 adult colon cancer patients from de-identified longitudinal UCSF EHRs, split by patient into 665 training, 143 validation, and 142 test cases.Documented first-line plans, clinical rationales, and oncologists’ information-acquisition processes inform the process-aware reward model.Source: §4.1 · pp. 3–4

4.2 Patient Representation and Environment State

• The environment represents each patient’s documented record as an incomplete proxy for clinical state and assumes no clinical time elapses or physiology changes during a trajectory.The patient record is structured around history of present illness, oncologic profiles, and physiological data.Source: §4.1 · p. 3, §4.2 · p. 4

4.3 Clinical Agent Action Space, Structured Action Parser & Frozen Patient-Record Mediator

• The agent’s responses pass through a deterministic parser and frozen patient-record mediator; top-1 retrieval without an abstention threshold can return a nearest field when the requested variable is absent.This creates traceable semantic retrieval error rather than stochastic observation noise.Source: §4.3 · p. 5

4.4 Reward Model Construction

• The reward uses a clinician reference withheld from the agent and scores information gathering, regimen accuracy, and reasoning alignment at trajectory termination.Information gathering penalizes premature recommendations and inefficient over-investigation; clinically equivalent regimens may receive substantial credit.Source: §4.4 · p. 5
• The clinical reward also subtracts one-time penalties for parser errors and failure to provide a treatment recommendation before the turn budget expires.The treatment reference is a signal rather than an exact verifier because other clinically defensible treatments may exist.Source: §4.4 · p. 5

4.5 Objectives and Algorithms: GRPO and GDPO

• GDPO normalizes each clinical reward component separately before combining component advantages, addressing GRPO’s loss of distinct reward profiles after scalarization.Both methods use multiple trajectories for the same patient to estimate relative quality without a separate value function.Source: §4.5 · p. 6

5 Results

• The evaluation measured Mean@3 and Best@3 across three generations per patient, using average and highest scalar clinical reward, respectively.The test set comprised 142 held-out patients.Source: §5 · p. 6
• GDPO-trained Llama achieved the highest Mean@3 reward (0.6085), exceeding GPT-5.2 (0.5121) among evaluated policies.The evaluation used 142 held-out patients with three generations per case.Source: §5 · p. 6
• Qwen3.5-9B Mean@3 rose from −0.0142 to 0.1994 with GRPO and 0.3429 with GDPO; GLM-4-9B-0414 rose from −0.0163 to 0.5313 and 0.5105.For Llama-3.1-8B-Instruct, GRPO and GDPO increased Mean@3 from 0.3388 to 0.5712 and 0.6085, respectively.Source: §5 · p. 6
• Against GPT-5.2, Llama’s relative advantage was concentrated in treatment selection, while GPT-5.2 remained stronger on information gathering and reasoning.Treatment rewards were 3.366 for GRPO-Llama and 2.791 for GDPO-Llama, versus 2.702 for GPT-5.2; information-gathering and reasoning values are reported as Mean@3 total scalar rewards.Source: §5 · p. 6
• During Llama training, treatment reward increased more strongly than information-gathering and reasoning reward, while GRPO showed a later-stage reward collapse.Information-acquisition behavior changed with RL, but not uniformly across policy backbones.Source: §5 · p. 6
• On 34 shared held-out cases, GDPO-Llama averaged 0.6000 scalar reward versus 0.573 for oncologists, 0.544 for GPT-5.2, and 0.537 for GRPO-Llama.Four oncologists produced trajectories in the same environment, with recommendations required within five turns.Source: §5 · p. 6

6 Discussion & Conclusion

• The environment models oncology treatment planning as interactive reinforcement learning from retrospective records, with patient-specific information gathering before a recommendation and process-aware rewards.The reward decomposition makes information acquisition, treatment selection, and reasoning observable as distinct parts of the policy trajectory.Source: §6 · pp. 6–7
• The specialized Llama policy outperforms both its base policy and GPT-5.2 in overall treatment-planning reward while improving across information, treatment, and reasoning axes.The conclusion also reports more selective information acquisition.Source: §6 · p. 7
• During training, clinical reward rises as the GRPO-trained Llama policy issues fewer queries, while treatment reward improves faster than information-gathering and reasoning rewards.These patterns are consistent with more selective information acquisition and show that reward axes can improve at different rates.Source: §6 · p. 7
• Observable decision trajectories create an opportunity to study expert-policy variation, since policies may reach the same recommendation through different information-gathering and reasoning paths.Retrospective data aggregated across clinicians may represent multiple practice patterns rather than a single expert policy.Source: §6 · pp. 7–8
• Physician identity alone cannot establish clinician practice phenotypes because clinicians treated different patient sets, confounding subgroup differences with case mix.Phenotypes would require comparison of decision trajectories across comparable clinical states while accounting for patient differences.Source: §6 · p. 8
• The environment’s fidelity is limited by a deterministic mediator and reconstructed patient records that approximate actual encounters, which vary across time, conditions, specialties, and systems.The authors also report only one RL training run per configuration due to resource constraints.Source: §6 · p. 8
• The current scope is first-line treatment selection; treatment modifications and greater fidelity to realistic patient representations remain future work.The authors also identify further mediator development as future work.Source: §6 · p. 9

• The cohort linked adult colon-cancer patients to chemotherapy episodes to focus on patients receiving their oncology care within UCSF.Eligible episodes were linked to cancer-registry concepts and chemotherapy episode records; collected notes came from GI Medical Oncology encounters with MD clinicians.Source: A Data & Cohort Construction · p. 12
• Eligible plans required a documented definitive regimen and excluded patients identified as enrolled in a clinical trial.After filtering, GI Medical Oncology notes were used to reconstruct each patient's clinical trajectory and structured representation.Source: A Data & Cohort Construction · p. 12
• Clinical observations remained irregularly sampled, missing values were not imputed, and no explicit administrative or clinical censoring was applied.Observation windows were defined relative to the cancer index date, chemotherapy episode date, and treatment-plan date.Source: A Data & Cohort Construction · p. 12
• The final dataset contains 950 patients, partitioned into 665 training, 143 validation, and 142 test patients.Source: A Data & Cohort Construction · p. 12

B Patient Representation and Environment State

• The patient representation is an incomplete, potentially noisy proxy for the underlying clinical state; the simulated trajectory assumes no clinical time elapses and agent actions do not change physiology or generate counterfactual outcomes.The patient record is initially hidden but available through retrieval, while a one-line patient summary is available at the start.Source: B Patient Representation and… · pp. 12–13

C.1 Action Space

• At each turn, the agent generates a natural-language response that is parsed into an investigate or recommend action, with recommendation terminating the episode.The response may include free-form reasoning before the parser extracts the structured action.Source: §C.1 · p. 13, §C.2 · p. 14

C.2 Action Parser

• The deterministic parser executes valid structured actions, consumes a turn for malformed output, and screens queries that request clinical inference instead of documented variables.Accepted queries can execute independently even when other queries in the same investigation are rejected.Source: §C.2 · pp. 13–14

C.3 Frozen Patient-Record Mediator

• The frozen mediator uses query and document embeddings to select patient-record fields and returns the selected field with its full documented value.The record is flattened into retrievable fields; queries in an investigation are ordered and passed through the mediator as a batch.Source: §C.3 · p. 14

D Policy Optimization Algorithms

• GRPO optimizes expected terminal return, while GDPO adds a second normalization across rollouts in the optimizer batch.Source: D Policy Optimization Algorithms · pp. 14–15

• Training used a five-turn interaction budget, at most 512 generated tokens per turn, and groups of eight trajectories per patient.LoRA adapters were applied to attention and feed-forward projection matrices; the remaining network was frozen and loaded with 4-bit quantization.Source: E Experimental Setup · p. 16
• The environment used a frozen Qwen3-Embedding-0.6B mediator and GPT-5.2 judge throughout training and evaluation.Source: E Experimental Setup · p. 16
• The highest-Mean@1 validation checkpoint was selected for each configuration, and test data was not used for checkpoint selection.Each of the 665 training patients was processed once, with three inner optimization steps reusing each patient's sampled trajectories.Source: E Experimental Setup · p. 16

• Figure 3 illustrates how the clinical agent retrieves information, explains its retrieval reasoning, and infers from the retrieved information.Source: Figure 3 · p. 16
• Figure 4 plots total scalar reward and average queries, with dark curves showing 15-step moving averages and faint curves showing unsmoothed trajectories.Each curve represents one training run.Source: Figure 4 · p. 17

G Reinforcement Learning using Different Policy Backbones

• For Qwen3.5-9B, GDPO raises Mean@3 from −0.0142 to 0.3429, compared with 0.1994 for GRPO.Source: G Reinforcement Learning using… · p. 17
• For GLM-4-9B-0414, GRPO raises Mean@3 from −0.0163 to 0.5313, compared with 0.5105 for GDPO.Source: G Reinforcement Learning using… · p. 17
• Both backbones improve on each reward axis during training, as shown in Figures 5 and 6.Source: G Reinforcement Learning using… · p. 17
• Information, treatment, and reasoning are scored from 0–5, while completeness and format are scored from 0–1.Source: Table 4 · p. 17

H Quantitative Analysis of Information-Acquisition Behavior

• Behavioral measures were extracted from held-out test trajectories for 142 patients and compared pairwise by patient after averaging three sampled trajectories.Source: H Quantitative Analysis of… · pp. 17–18
• 4.65 fewer queries per case and 2.09 fewer unique fields retrieved accompany a 0.63 increase in information-gathering reward for GRPO versus base Llama.Source: H Quantitative Analysis of… · p. 18
• 2.22 more queries and 2.65 more unique fields retrieved accompany a 0.83 increase in information-gathering reward for GDPO versus base Llama.Source: H Quantitative Analysis of… · p. 18
• Both RL-trained Llama policies issue approximately 27 to 34 fewer queries and retrieve approximately 10 to 14 fewer unique fields per case than GPT-5.2.Source: H Quantitative Analysis of… · p. 18
• The information-gathering reward favors decision-relevant variables and penalizes unnecessary or redundant investigation, including gathering after sufficient evidence is available.Source: H Quantitative Analysis of… · p. 18

I Exploratory Analysis Across Primary Oncologist Subgroups

• GDPO’s median paired reward differences versus GPT-5.2 were 0.067, 0.044, and 0.044 for Providers A, B, and C, respectively; only Provider A was significant.For Provider A, the mean paired difference was 0.100 (95% CI [0.036, 0.169], Holm-corrected p = 0.018). No evidence showed that the GDPO–GPT-5.2 reward difference varied across providers (Kruskal–Wallis H = 0.464, p = 0.793); GDPO absolute rewards did vary (H = 8.585, p = 0.014), but these comparisons may reflect differences in patient case mix.Source: I Exploratory Analysis Across Primary… · p. 18, Figure 7 · p. 20, Table 6 · p. 20

Clinician Decision Extraction Prompt

• The clinician decision-extraction prompt asks for first-line therapy and dose intensity while reconstructing the chronological information-gathering and decision pathway.It separates observations, actions, and conclusions, and instructs the extractor not to infer undocumented facts.Source: Clinician Decision Extraction Prompt · pp. 21–22

Patient Representation Prompt

• The patient-representation prompt structures clinical information while prohibiting extraction of future treatment plans or recommendations.It directs the extractor to use injected structured demographics as primary, overriding them only when notes explicitly correct an error.Source: Patient Representation Prompt · p. 26
• The patient summary captures cancer context and chemotherapy-relevant comorbidities without including a treatment plan, using a specified one-line format.Its purpose is to help a downstream model recognize the clinical context and safety constraints; the output is specified as JSON.Source: Patient Representation Prompt · p. 27

Judge Model Prompt

• The grader scores regimen accuracy, reasoning alignment, and information gathering, accounting for clinical equivalence, decision timing, and query efficiency.Its rubric prioritizes the primary regimen over trial drugs and does not score dosage or cycle counts.Source: Judge Model Prompt · p. 28

Clinical Agent Prompt

• The clinical agent queries a retrieval layer for single recorded facts, sequencing basic findings, histology, staging, and relevant molecular data before treatment selection.The retrieval layer cannot interpret findings or judge eligibility; the prompt also limits queries to information relevant to the immediate decision.Source: Clinical Agent Prompt · pp. 30–31