PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

Humza Nusrat, Kundan Thind

arXiv:2610.37081v1cs.AIphysics.med-ph

TL;DR

Radiotherapy planning agents are often evaluated by dose, which does not reveal whether they recognize impossible requests or make appropriate submission decisions. OpenGray tests decisions on tasks with known answers and audits clinical planning logs; high aggregate accuracy coexisted with specific decision failures.

  • Problem

    Dose-focused evaluations of LLM planning agents do not assess whether requests can be met or whether agents should submit plans or request review.

    Source: §1 · p. 1

  • Method

    OpenGray adds tasks with known correct decisions to a planning benchmark and applies the same outcome measures to clinical-agent planning logs.

    Source: §1 · p. 1

  • Results

    Above 90% decision accuracy in both LLM-based systems coexisted with missed overlap contradictions and feasible submissions that missed hard goals.

    Source: §4 · pp. 4–5

  • Takeaways & Limitations

    Planning agents should be evaluated on decisions as well as dose and given an explicit action for stopping and requesting review.

    Source: §4 · p. 5

    The benchmark covered one head-and-neck dataset, two contradiction types, and 30 evaluation patients, so findings describe those tasks and the two tested LLMs.

    Source: §4 · p. 5

Abstract

from arXiv · show

Background and purpose: Radiotherapy planning agents based on large language models (LLMs) decide when to submit a plan or request review but are evaluated mainly by dose. This study developed OpenGray, a benchmark of planning tasks with known correct decisions. Materials and methods: OpenGray adds tasks with known answers to the OpenKBP-Opt head-and-neck dataset, including two contradictions with dose goals proven incompatible and a feasible control with a stored plan meeting every hard goal. Four systems, from a rule-based planning script to a full LLM agent, were evaluated on 30 patients, the LLM-based ones with a commercial (Terra) and a local (Ministral 3 8B) LLM. Decisions, achieved dose, problem recognition, and technical failures were recorded and also read from logs of 39 stereotactic radiosurgery runs of SAGE, a clinical LLM agent. Results: With Terra, the script preceded by an LLM review and the full agent had decision accuracies of 95.4% and 92.1%, respectively, yet escalated only 50% (95% confidence interval, 25-75%) and 25% (10-40%) of overlap contradictions, which required arithmetic on the anatomy. All four systems submitted every feasible control, but 24% of these plans missed a hard goal that the stored plan had met. In four SAGE runs, one goal failed at every attempt; the agent identified it in every reply but had no action for stopping. Conclusions: Favorable aggregate scores concealed specific decision failures. Planning agents should be evaluated on their decisions as well as their dose and given an explicit action for stopping and requesting review.

1. INTRODUCTION

Dose-based evaluation does not reveal whether a planning agent recognizes an impossible request or should submit a plan versus escalate. OpenGray evaluates decisions on tasks with known answers and applies the same outcomes to clinical planning logs.

1. INTRODUCTION

  • Dose-based comparisons cannot show whether a request is impossible or whether an agent should continue, submit, or request review.For flawed requests, judging the decision requires knowing the correct answer in advance.Source: §1 · p. 1

2.1. Benchmark data and planning functions

  • OpenGray uses 100 head-and-neck cases—10 for development, 30 for evaluation, and 60 for score calibration—and routes planning actions through a harness that returns case summaries and dose.The benchmark decisions are known, unlike those for logged clinical requests.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2. Tasks with known answers

  • Six tasks cover routine planning, note and missing-structure errors, two contradictions, and a difficult feasible control verified by a stored plan meeting every hard goal.The contradictions arise either from stated dose values or from anatomy-dependent overlap arithmetic.Source: §2.2 · p. 2

2.3. Planning systems

  • Four systems ranged from a rule-based controller through arithmetic checks and one-time LLM review to a full LLM agent; both LLM systems used Terra and Ministral under a shared policy.The policy directed escalation for missing structures and demonstrated incompatibilities, and otherwise required planning within budget.Source: §2.3 · p. 3

2.4. Outcomes

  • Evaluation recorded final decisions, dose-goal attainment, recognition of the actual problem, and technical failures.Decision accuracy was calculated on the four scored tasks; submitted plans were assessed against hard dose goals.Source: §2.4 · p. 3

2.5. Planning logs of a clinical agent

  • The same outcomes were reconstructed from logs of 39 SAGE stereotactic radiosurgery runs, although the correct decision for each clinical request was unknown.The logs recorded each request, proposed objective changes, and resulting dose.Source: Figure 1 · p. 2, §2.5 · p. 3

3. RESULTS

With Terra, high aggregate decision accuracy concealed poor recognition of overlap contradictions, while feasible plans and clinical runs exposed dose-goal and stopping failures. The findings support evaluating decisions alongside dose and providing explicit stopping and review actions.

3.1. Decision accuracy

  • 95.4% interpreter and 92.1% full-agent decision accuracy with Terra coexisted with overlap escalation rates of 50% and 25%, respectively; arithmetic checks escalated 9 of 10.Ministral 3 8B missed the same contradictions, with interpreter and full-agent accuracies of 31.7% and 43.5%.Source: §3.1 · p. 4

3.2. Dose-goal attainment on the feasible controls

  • 76% of feasible-control plans met every hard goal, although every system submitted those controls; increasing the optimization budget did not correct the six deficient controls.Their stored plans had met every goal, and the shortfall was attributed to the controller-selected objectives.Source: §3.2 · p. 4

3.3. The SAGE planning logs

  • 35 of 39 SAGE runs met every goal, but in four runs a goal failed at every attempt and the agent had no action for stopping or requesting review.A reply-format failure also prevented dose calculation in attempts 9 and 10 of Patient 27’s initial run.Source: §3.3 · p. 4

4. DISCUSSION

  • Overlap contradictions required arithmetic on the anatomy, which the LLM-based systems missed despite having the necessary case-summary numbers.Their instructions permitted infeasibility escalation only after incompatibility was demonstrated; otherwise, the fallback was to plan and report unmet goals.Source: §4 · pp. 4–5
  • The aggregate score obscured task-specific failures: missing-structure and feasible-control tasks made up 60% of the score, while overlap tasks made up only 9%.The controller scored 60.0% while missing every contradiction, supporting reporting decision accuracy by task alongside achieved dose.Source: §4 · p. 5
  • SAGE recognized failing goals in every reply but lacked a stopping action, supporting an explicit action for stopping and requesting review, with a rule and necessary information.The full benchmark agent similarly reported unmet goals rather than escalating, despite having escalation available.Source: §4 · p. 5
  • OpenGray tested one head-and-neck dataset, two contradiction types, and 30 evaluation patients, so its results describe those tasks and the two LLMs tested.Because benchmark tasks can be tuned to once public, future versions will require conflict types withheld from developers.Source: §4 · p. 5
  • The clinical log audit covered one agent at one institution and could not establish whether the four unresolved runs could have met their goals.Plans were optimized without machine constraints, so feasible controls would need re-verification under deliverability constraints.Source: §4 · p. 5

CRediT authorship contribution statement

The contribution statement names the authors’ roles, and the accompanying statements report funding and data availability.

CRediT authorship contribution statement

  • Humza Nusrat and Kundan Thind are credited with conceptualization, methodology, and writing; Nusrat is also credited with formal analysis.Both are credited with writing the original draft and reviewing and editing.Source: CRediT authorship contribution statement · p. 5

Funding

  • The authors report no funding sources for this work.Source: Funding · p. 6

Data availability statement

  • The OpenKBP and OpenKBP-Opt datasets are publicly available, while clinical logs and patient-to-plan linkage records are retained privately and excluded from the deposit.The statement says the OpenGray software and benchmark data are available in Zenodo, but the supplied passage does not complete that sentence.Source: Data availability statement · p. 6

S1 PLANNING ENVIRONMENT, TASK DEFINITIONS, AND EXECUTION

The supplement describes the planning interface, task policy, PlanScore, and the systems and schedules used in development experiments.

S1.1 Planning interface and task policy

  • OpenGray’s harness executes requested planning functions and records each step; development tracks vary task presentation and feasibility, with distinct policies for scoring escalation and submission.The nine functions include setting objectives, optimizing fluence, comparing dose with goals, submitting plans, and escalating tasks.Source: S1.1 Planning interface and task policy · p. 8

S1.2 Dose goals and PlanScore

  • PlanScore combines hard-goal attainment, soft-goal violation, and reference-relative sparing, while rejection gates can set the score to zero.Hard and soft dose-valued goals use a 0.05 Gy tolerance; missing other hard goals reduces the score without automatically rejecting the plan.Source: S1.2 Dose goals and PlanScore · p. 8

S1.3 Development systems and comparison schedules

  • Development comparisons included scripted systems, arithmetic checks, and LLM-based systems, with schedules and model conditions varying across tracks.The Terra, Sonnet, and GLM interpreter used one model call; some T2 results used historical rule-withheld conditions, so those results cannot rank models under common disclosure conditions.Source: S1.3 Development systems and… · pp. 8–9

S1.4 Reserved-cohort protocol and execution

The reserved evaluation used a prespecified patient-weighted decision-accuracy policy, alongside development results on rejection-rule disclosure and revised goals.

S1.4 Reserved-cohort protocol and execution

  • The reserved evaluation compared four systems on 30 patients, with two fresh repetitions for language systems and patient-weighted decision accuracy as the primary policy outcome.The policy required escalation for missing structures and certified contradictions and submission for feasible controls; technical failures were retained rather than replaced.Source: S1.4 Reserved-cohort protocol and… · p. 9

S2.1 Rejection-rule disclosure and planning performance

  • Disclosing rejection rules increased T1 PlanScore by 0.281 for Terra, 0.194 for Sonnet, and 0.146 for GLM.Among successful episodes, rejection-gate failures fell from 46/150 with rules withheld to 7/149 with rules disclosed.Source: S2.1 Rejection-rule disclosure and… · p. 9

S2.2 Response to revised goals

  • On T3, the tightened goal was attained in 30/30 controller and interpreter episodes, 29/30 Terra episodes, 27/30 Sonnet and GLM episodes, and 24/30 heuristic episodes.Mean scores ranged from 0.855 for the heuristic to 0.887 for Terra.Source: S2.2 Response to revised goals · pp. 9–10

S2.3 Altered inputs and explicit checks

Development experiments tested altered inputs, anatomical contradiction recognition, fluence-complexity objectives, response ratings, and robustness.

S2.3 Altered inputs and explicit checks

  • The full Terra agent escalated all six missing-structure tasks but submitted all six conflicting-note tasks without a conflict flag; the interpreter escalated both sets.The interpreter and preflight system exceeded the heuristic’s matched T4 grade by 0.161 and 0.159, respectively.Source: S2.3 Altered inputs and explicit checks · p. 10

S2.4 Anatomical information and contradiction recognition

  • The full-summary interpreter escalated 25/25 overlap contradictions versus 0/25 with the reduced summary, but summary conditions also differed in case version and beam geometry.Arithmetic checks escalated all 15 certified development scenarios and none of the 10 feasible controls; their mean T5 grade was 0.882.Source: S2.4 Anatomical information and… · p. 10

S2.5 Fluence-complexity objectives

  • Across 240 complexity experiments, mean PlanScore fell by 0.061–0.528 relative to corresponding results without added complexity objectives, and normalization could violate the intended fluence limits.Only 82/240 final normalized fluences met the sum-of-positive-gradients limit, while 234/240 met the beamlet-weight limit; the changes do not demonstrate clinical deliverability.Source: S2.5 Fluence-complexity objectives · p. 10

S3.1 Sampling and review procedure

  • The response review used stratified samples designed to identify errors, not estimate their prevalence, and one medical physicist rated responses with system identities and automatic grades withheld.The development rubric credited nine harness-forced escalations, while the reserved rubric did not.Source: S3.1 Sampling and review procedure · p. 10

S3.2 Ratings

  • Across all 198 rated responses, development recognition was credited on 70/128 applicable responses and action on 82/138; reserved counts were 25/40 and 45/60.Because sampling and rubrics differed, the counts cannot be pooled; reserved action ratings agreed with the automatic endpoint on all 40 reviewed responses from the four primary families.Source: S3.2 Ratings · p. 10

S4 ROBUSTNESS OF THE DEVELOPMENT COMPARISONS

  • Development comparisons were exploratory: omitting one patient left disclosure gains positive, but the T2 Terra–controller difference changed sign.The leave-one-patient-out ranges were 0.222–0.314 for Terra’s disclosure gain and −0.0115 to 0.0082 for the T2 difference.Source: S4 ROBUSTNESS OF THE DEVELOPMENT… · p. 11

S5 CERTIFICATE PROOFS AND FEASIBLE CONTROLS

This section specifies dose-volume conventions and sufficient geometric conditions for proving that overlapping target and organ goals are incompatible. The certificates are conservative, so they establish some contradictions without claiming to detect every infeasible request.

S5.1 Dose-volume conventions

  • Percentile dose values are interpolated between adjacent order statistics of the sorted structure-dose vector.The checked cases assume uniform voxel volume, that External contains every calculated voxel, and nonnegative dose and fluence.Source: S5.1 Dose-volume conventions · p. 11

S5.2 Overlap contradictions

  • A target coverage goal and an overlapping organ limit are incompatible when their required and permitted voxel counts and dose thresholds cannot both be satisfied.The sufficient test allows extra voxels for rounding and interpolation, so it is not necessary for detecting incompatibility.Source: S5.2 Overlap contradictions · p. 12
  • For pt_248, the overlap certificate has a 342-voxel count margin and a 5.40 Gy dose margin.The example concerns 631 mandible voxels overlapping PTV56, with target and organ goals of 53.2 Gy and 47.7 Gy, respectively.Source: S5.2 Overlap contradictions · pp. 12–13

S5.3 Coverage/cap contradictions and construction eligibility

The section gives a sufficient coverage-versus-cap test for incompatibility, verifies constructed certificates and feasible witnesses independently, and reports hard-goal attainment for submitted feasible controls and routine tasks.

S5.3 Coverage/cap contradictions and construction eligibility

  • The sufficient coverage/cap contradiction follows when the required target coverage exceeds what an External maximum-dose cap permits.For pt_242, the exact threshold was 77.9476 Gy against a 77 Gy cap, a positive margin of 0.0524 Gy.Source: S5.3 Coverage/cap contradictions and… · p. 13
  • Independent geometric reconstruction verified all 39 coverage/cap and 15 overlap certificates across the development and reserved cohorts.Reserved coverage margins were 0.0524–0.5190 Gy and overlap count margins were 2–550 voxels; some constructions were excluded because no positive certificate or admissible witness was found.Source: S5.3 Coverage/cap contradictions and… · p. 13

S5.4 Feasible controls and independent dose reconstruction

  • All 25 reserved feasible witnesses met every hard goal under the same 0.05 Gy tolerance, and all 327 submitted reserved plans were independently reconstructed.Reconstruction discrepancies from stored metrics were at most 4.41 × 10^-13 Gy; the witnesses establish feasibility within the supplied fluence model, not machine deliverability.Source: S5.4 Feasible controls and independent… · p. 13

S6.1 Submitted-plan dosimetry and attempt-level outcomes

  • 76% of feasible-control attempts met every hard goal, although every system submitted every feasible control.This was 19/25 attempts for each deterministic system and 38/50 for each language system; no submitted control crossed a PlanScore rejection gate.Source: S6.1 Submitted-plan dosimetry and… · p. 13
  • Routine all-hard-goal attainment was 21/30 controller, 20/30 arithmetic/controller, 42/60 interpreter/controller, and 47/60 full-agent attempts.These all-attempt denominators include nonsubmissions; three routine attempts ended without submission.Source: S6.1 Submitted-plan dosimetry and… · pp. 13, 15

S6.2 Objective functions

The section defines objective penalties and optimizer settings, then verifies dose reconstruction, objective calculations, and solver agreement on reduced problems. Agreement depended on conditioning, and reduced-problem verification does not establish full-size convergence.

S6.2 Objective functions

  • The objective template uses weighted penalties for dose deviations; these terms do not enforce hard constraints.Optional DVH penalties select hot voxels by dose quantile, while smoothness penalizes adjacent-beamlet differences; normalization can change dose goals and complexity measures.Source: S6.2 Objective functions · p. 15
  • The default FISTA optimizer stops after 500 iterations or at relative tolerance 10^-4, defining the study's computational budget.The verification of objective implementation and reduced-problem solver agreement is distinct from convergence on full-size problems.Source: S6.2 Objective functions · p. 15

S6.3 Reconstruction and optimizer verification

  • Published-fluence reconstruction on all 10 development cases agreed with supplied dose within 4.27 × 10^-14 Gy.Synthetic objective and gradient checks met their specified tolerances, and a diagonal least-squares test had maximum fluence error of 8.56 × 10^-8.Source: S6.3 Reconstruction and optimizer… · p. 15
  • All 10 reduced-problem solver comparisons passed, with a maximum scaled objective difference of 2.57 × 10^-13 and maximum voxel-dose difference of 0.000198 Gy.Earlier series using 128 evenly spaced columns passed in only 2 and 6 of 10 cases, so agreement depended on conditioning.Source: S6.3 Reconstruction and optimizer… · pp. 15–16

S6.4 System components and available information

The systems differ in where checks and planning decisions occur and in the information available to them. An audit of SAGE logs found repeated unresolved goal failures, but the records are descriptive rather than an estimate of agent success rate.

S6.4 System components and available information

  • The controller-based systems shared a planning controller, while the arithmetic system checked for missing structures and two contradiction families before planning.The interpreter used one LLM call to escalate or delegate; the full agent chose its own planning actions, so the comparison spans different prompts and opportunities for action.Source: S6.4 System components and available… · p. 16
  • The full case summary included voxel counts, rounded overlap fractions, and distances, but omitted certificates, expected decisions, witness fluence, reference dose, and score weights.The full agent could request calculated dose-volume histograms and metrics; score-visible search was a privileged development comparator.Source: S6.4 System components and available… · p. 17

S7.1 Cohort and source records

  • The SAGE audit covered 39 runs from 31 patients, and the authors caution that it is not an estimate of agent success rate.Thirty-five runs from 29 patients met every loaded goal; four reached the attempt limit.Source: S7.1 Cohort and source records · pp. 17–18
  • In all four SAGE runs that reached the attempt limit, one goal failed at every new dose calculation, yet no log recorded stopping, escalation, or manual action.The instructions offered no such action; the audit describes these runs and is not an estimate of SAGE's success rate.Source: S7.1 Cohort and source records · p. 18

S7.2 Recorded goals, replies, flags, and execution

The analyses validate recorded SAGE goals and decisions, trace a successful iterative plan, and examine task-weighted language-system comparisons and overlap-contradiction decisions.

S7.2 Recorded goals, replies, flags, and execution

  • Across all 88 replies, the failing goals named in the rationale matched the measured failures, but “not feasible with priorities” was never written.Its condition was met in a Patient 30 refinement run; 83 of 94 attempts produced new dose calculations, and all 610 goal-value rows matched the logs.Source: S7.2 Recorded goals, replies, flags,… · p. 18

S7.3 Example of iterative planning

  • Patient 4's PTV V21Gy rose from 21.75% to 96.62% over four calculations while the other four goals passed throughout.Target and ring dose-volume parameters stayed fixed as priorities changed, but no single priority's contribution was isolated; an unreadable-reply retry added no dose state.Source: S7.3 Example of iterative planning · pp. 18–19

S8.1 Task families, weighting, and the contrast between the language systems

S8.2 Overlap contradictions task by task

  • The interpreter delegated both runs of three overlap tasks and split its decisions on four; the full agent escalated five attempts and submitted 15.The arithmetic check escalated nine overlap tasks but missed pt_312.Source: S8.2 Overlap contradictions task by task · p. 19

S8.3 Final decisions and the conflicting-note scenario

Reserved-task analyses found strong performance on conflicting-note decisions, alongside unmet hard goals in feasible-control submissions and sensitivity to alternative scoring policies.

S8.3 Final decisions and the conflicting-note scenario

  • The interpreter/controller earned credit on all 60 conflicting-note attempts; the full agent scored 59 of 60, while neither scripted system earned credit.The interpreter/controller had 31 escalations and 29 flagged submissions; the full agent's one lost attempt was a technical failure.Source: S8.3 Final decisions and the… · p. 19

S8.4 Feasible controls with unmet hard goals

S8.5 Alternative action policies

  • Requiring matching escalation reasons lowered controller accuracy from 60.0% to 25.8%; requiring all hard goals for feasible-control credit lowered every system by 6.4 points.The interpreter-minus-agent difference remained 3.33 points under the latter policy.Source: S8.5 Alternative action policies · p. 22

S9 SECOND LANGUAGE MODEL, LONGER OPTIMIZATION BUDGETS, AND FLUENCE COMPLEXITY

  • The reserved-task analyses used stored outcomes and certificates, with no model calls or plan recomputations; comparisons among these runs were not tested statistically.Source: S9 SECOND LANGUAGE MODEL, LONGER… · p. 22

S9.1 Rebuilt tasks and their verification

S9.2 Controller with longer optimization budgets

Longer optimization budgets did not resolve the same six deficient controls; local-model runs also showed decision and technical failures, and submitted fluences exceeded the reference SPG limit.

S9.2 Controller with longer optimization budgets

  • All three budgets left the same six controls deficient, with PTV_7000 D99 changing by at most 0.02 Gy, implicating the controller's objectives rather than iteration limits.Every control was submitted at each budget; stored witnesses met the same goals with at most three 500-iteration solves.Source: S9.2 Controller with longer… · p. 23, Table S17 · p. 24

S9.3 Ministral 3 8B in the two language systems

  • The interpreter's required reply format failed in 113 attempts; with format relaxed, it planned on 9 of 20 overlap attempts and escalated 6, none citing the mandible.Source: S9.3 Ministral 3 8B in the two… · p. 24
  • The full agent escalated 4 of 20 overlap attempts after one to three optimizations with plans missing hard goals, and only 9 of 32 submitted feasible controls met every hard goal.It lost 26 attempts to unreadable planning actions and two more to harness endings.Source: S9.3 Ministral 3 8B in the two… · p. 24

S9.4 Fluence complexity of the reserved submissions

  • No submitted plan met the SPG limit of 65, and the smallest value was 85.3; the full agent's median SPG was about 2.8 times that of controller-based systems.These measurements describe normalized fluences only; pre-normalization fluences were not stored.Source: S9.4 Fluence complexity of the… · pp. 24–25, Table S19 · p. 25
Loading 2610.37081v1…