PaperlayerView exact revision on arXiv

Source-linked AI summary

JET: Judge-Guided Evolution at Test Time for Agent Programs

Yao Long Teng, Jiayi Cai, Bo An

arXiv:2609.34126v1cs.SEcs.AI

TL;DR

When target rewards are unavailable, it is difficult to decide which agent-program changes to retain, and a judge must remain useful as tasks and candidate behaviors change. JET evolves a judge on labeled source trajectories and freezes it to guide target-side search. On unseen WebShop tasks, it outperforms fixed-rubric guidance in cold- and warm-start mean reward, with a larger cold-start exact-success improvement.

  • Problem

    At test time, true rewards may be unavailable, leaving open whether a judge trained on labeled source trajectories can guide program evolution on reward-hidden target tasks.

    Source: §1 · p. 1

  • Method

    JET evolves an executable judge on labeled source trajectories, selects and freezes it, then transfers its scores and diagnostics to target-side program evolution without target evaluator access or model-weight updates.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    On unseen WEBSHOP tasks, JET achieves approximately 13% higher mean reward in cold start and 4% in warm start than fixed-rubric guidance, with 36% higher cold-start exact success.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support executable judge transfer for program adaptation under evaluator-preserving task shifts.

    Source: §6 · p. 9

    Both benchmarks preserve the evaluation rule, leaving transfer to changed reward criteria or outcomes requiring unobserved information untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

An agent's executable program governs how it uses tools, processes observations, and responds to failures. Evolving this program at test time can help adaptation, but deciding which changes to retain is difficult when true rewards are unavailable. Execution traces provide evidence of agent behavior, yet interpreting that evidence requires a judge that remains useful as tasks and candidate programs change. We introduce Judge-Guided Evolution at Test Time (JET), which evolves an executable judge on labeled source trajectories, then freezes and transfers it to guide target-side program evolution. The judge supplies scores and diagnostic feedback without target evaluator access or model-weight updates. On unseen WebShop tasks, JET achieves approximately 13% higher mean reward than fixed-rubric guidance when evolution begins from an unevolved program (cold start) and 4% higher when it begins from one already optimized on source tasks (warm start), with a 36% relative improvement in cold-start exact success. An exact-judge control on PushT, where the judge reconstructs the scoring rule from observations, shows that without judge error, program search becomes the bottleneck. Analyses identify useful reward-prediction logic in the evolved code and show that better final selection alone cannot explain the gains. These results support executable judge transfer for program adaptation under evaluator-preserving task shifts.

1 INTRODUCTION

JET addresses whether an executable judge developed on labeled source trajectories can guide program evolution when target rewards are unavailable. It transfers a frozen judge to target-side search, with higher mean reward than fixed-rubric guidance on unseen WebShop tasks and an exact-judge PushT control.

Research question

  • The central question is whether a judge developed on source trajectories with known rewards can guide program evolution on target tasks whose evaluator outputs are withheld.At test time, execution behavior may be observable even when true rewards are unavailable.Source: §1 · p. 1

Executable judge evolution for test-time adaptation

  • JET evolves an executable judge on labeled source trajectories, selects it using a separate source validation split, then freezes it to provide scores and diagnostics during target program evolution.Target evaluator outputs are reserved for post-hoc evaluation, and underlying model weights remain fixed.Source: §1 · p. 2

Evaluation under evaluator-preserving shifts

  • An exact-judge PUSHT control reconstructs the unchanged geometric evaluation rule from observations, isolating program search from judge error.Both benchmarks preserve the evaluation rule across the source–target shift while tasks or dynamics change.Source: §1 · p. 2
  • Approximately 13% higher mean reward in cold start and 4% in warm start, with 36% higher cold-start exact success, than fixed-rubric guidance on unseen WEBSHOP tasks.Cold start begins from an unevolved program; warm start begins from one optimized on source tasks.Source: §1 · p. 2

Analysis of how evaluation shapes program search

  • Source-holdout controls identify executable reward-prediction rules, while primary campaign results show better final selection alone cannot explain the guidance gap.Complementary online comparisons also examine evolution beyond prompt optimization and the roles of textual diagnostics and numerical score adjustments.Source: §1 · p. 2

2 RELATED WORK

Prior work studies pipeline and prompt optimization, executable-agent search, harness generation, and learned judges or reward transfer. JET focuses on evolving executable judge software against known source outcomes and transferring it to guide persistent program rewrites.

Program optimization and heuristic learning

  • Program-optimization work spans language-model pipelines and prompts, executable agents and workflows, and code- or prompt-based improvement from evaluation feedback.The heuristic-learning perspective describes software maintenance through edits informed by rewards, tests, logs, and replays.Source: §2 · p. 3

Harness generation

  • Harness-generation work searches code using task-specific scores, trains models to generate or repair harnesses, or studies test-time evolution with unlabeled execution feedback.JET instead evolves the evaluation procedure against known source outcomes, then freezes it to guide test-time program search.Source: §2 · p. 3

Learned judges and reward transfer

  • Learned-judge work assesses outputs or trajectories, learns interactive-agent signals, or transfers reward models; JET evolves executable judge software to guide persistent rewrites across tasks.Its judge combines evidence extraction, LLM assessment, score adjustment, and textual diagnostics while model weights remain fixed.Source: §2 · p. 3

3 METHOD: EVOLVING AND TRANSFERRING THE JUDGE

JET develops a judge from labeled source trajectories, freezes it, and transfers it to target program search without querying target evaluator outputs. The judge both scores candidates and supplies feedback for subsequent rewrites.

3.1 PROBLEM SETTING

  • Target evaluator outputs are withheld during adaptation, while task instructions and deployment-visible trajectories remain available to the proposer, judge, and selection rule.The objective is mean true reward on adaptation tasks, measuring adaptation to observed tasks rather than generalization to unseen ones.Source: §3.1 · p. 3

3.2 STAGE 1: EVOLVE AN EXECUTABLE JUDGE

  • Judge evolution minimizes mean absolute error on labeled source trajectories, then uses a private validation split to select a candidate that is frozen for target adaptation.A separate source holdout is reserved for post-selection reporting; the underlying inference-model weights remain fixed.Source: §3.2 · p. 4

3.3 STAGE 2: FREEZE THE JUDGE AND EVOLVE THE AGENT PROGRAM

  • The frozen judge scores candidate programs across target tasks; eligible candidates are promoted only when their score beats the current best program.The proposer also receives trajectories and judge feedback from rejected candidates, and selection uses no target evaluator outputs.Source: §3.3 · pp. 4–5

3.4 FROM REWARD PREDICTION TO SEARCH GUIDANCE

  • Prediction error is only one measure of judge usefulness: candidate-score ordering, especially near promotion decisions, matters for guiding search.A useful judge should assign a higher score to the candidate with higher true mean reward; an optional source constraint can enforce a floor on a fixed source check set.Source: §3.4 · p. 5

4 EXPERIMENTAL SETUP

The experiments test transfer across clothing-to-non-clothing shopping and a PushT control, comparing target adaptation conditions with evaluator outputs withheld and measuring reward or coverage.

4.1 TASKS AND TRANSFER SETTINGS

  • WEBSHOP tests adaptation from clothing source tasks to non-clothing target tasks, while PushT shifts initial displacement and damping but preserves an exactly reconstructible geometric scoring rule.Held-out tasks do not guide adaptation, and sessions are disjoint across splits.Source: §4.1 · p. 5

4.2 COMPARISON CONDITIONS

  • The comparison includes no target evolution, proposer self-selection, fixed-rubric guidance, and evolved-judge guidance from cold or source-optimized warm starts.Each primary condition runs five independent 40-round campaigns; adaptive conditions share task sets and proposal budgets within each comparison and initialization.Source: §4.2 · p. 5

4.3 EVALUATION METRICS

  • WEBSHOP performance is mean true reward and PushT performance is mean final goal coverage, while judge fidelity uses MAE and reward correlation.Oracle-best and selection regret are computed within each campaign and averaged; Table 1 reports five campaigns, with selected values as mean ± SD and other columns as means.Source: §4.3 · p. 5, Table 1 · p. 6

5 RESULTS

JET outperforms fixed-rubric guidance on WebShop across cold and warm starts, and analyses show that better final selection alone cannot explain the gap. In PushT, exact scoring leaves program generation as the remaining bottleneck.

5.1 WEBSHOP ADAPTATION AND HELD-OUT TRANSFER

  • Cold-start JET reaches 0.7797 mean reward versus 0.6965 under fixed-rubric guidance; warm-start JET reaches 0.6951 versus 0.6589, with the ordering preserved on held-out tasks.On held-out tasks, cold-start evolved-judge guidance reaches 0.7915 reward versus 0.7008 under fixed-rubric guidance; warm-start reaches 0.6951 versus 0.6700.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 WHAT THE JUDGE LEARNS

  • Source-holdout MAE falls from 0.1480 to 0.1168, while partial-reward MAE drops 23.4% and 24.7% on trajectories from both auxiliary programs.The evolved judge also underestimates partial progress less, although the fixed-rubric judge remains more accurate on complete failures.Source: Figure 3 · p. 7, §5.2 · pp. 7–8
  • Deterministic evolved code achieves source-holdout MAE of 0.1153 versus 0.1168 for the full judge, while isotonic calibration reaches 0.1488.The deterministic code also has lower MAE on both cached target pools; calibration does not match evolved-judge accuracy.Source: §5.2 · p. 7
  • On a wrong-size purchase, the evolved judge credits satisfied product, color, and price requirements and identifies size as the remaining mismatch.The fixed-rubric judge substantially underestimates partial completion and raises an unsupported price concern; scores affect retention and proposer feedback, while diagnostics guide revisions.Source: Figure 4 · p. 8, §5.2 · p. 8

5.3 SELECTION QUALITY AND CANDIDATE GENERATION

  • Better final selection closes only about 9.7% of the original mean reward gap; most of the advantage reflects programs generated during search.Across intermediate rounds, evolved-judge selection has zero regret under fixed-rubric candidate generation and averages 8 regret rounds under evolved-guided generation, versus 18 for the fixed-rubric judge.Source: §5.3 · p. 8

5.4 JUDGE BASELINES AND COMPONENT ABLATIONS

  • Executable evolution exceeds prompt-only optimization by 0.1473 adaptation reward and 0.1074 held-out reward.Prompt refinement leaves evidence extraction and score-adjustment code unchanged, whereas executable evolution can revise them alongside instructions.Source: §5.4 · pp. 8–9
  • With scoring fixed, diagnostics yield mean adaptation reward of 0.6984 versus 0.6579 and held-out reward of 0.6869 versus 0.6608.With diagnostics withheld, retaining score adjustments raises observed mean adaptation and held-out rewards by 0.0211 and 0.0484, respectively; scores can affect both candidate promotion and later rewrites.Source: §5.4 · p. 9

5.5 PUSHT: AN EXACT-JUDGE CONTROL

  • In PushT, exact evolved-judge scoring reaches 0.7679 cold-start coverage versus 0.6199 under fixed-rubric guidance, yet mean coverage remains below one.Zero selection regret means the evolved-judge condition selects the highest-true-coverage generated policy, so further improvement requires generating better policies.Source: §5.5 · p. 9

6 CONCLUSION AND LIMITATIONS

JET transfers a frozen judge from labeled source trajectories to target-side program search without target evaluator access. The appendix details the campaign protocol and observation constraints, while the conclusion notes limits in replication, transfer scope, and computational cost.

6 CONCLUSION AND LIMITATIONS

  • The main limitation is limited campaign replication: smaller differences, including diagnostic-feedback gains, remain uncertain, and transfer to changed reward criteria or outcomes requiring unobserved information is untested.Repeated generation, execution, and evaluation add inference cost and latency; broader replication and cost analysis remain future work.Source: §6 · p. 9
  • The authors used generative AI for experimental-methodology feedback, result interpretation, readability editing, drafting, restructuring, and reference and LaTeX formatting, and take responsibility for the final content.They report not using it to develop the method or hypotheses, generate or clean data, or perform mathematical claims, proofs, translation, or qualitative data analysis.Source: AI USE STATEMENT · p. 10
  • The authors state that code and artifacts will be released; the appendix directs readers to Sections 4 and 5 for protocols and results and to Appendix A for campaign settings and replication details.Other appendices cover source constraints, judge-error analyses, campaign diagnostics, qualitative examples, offline controls, and ablations.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 INDEPENDENT CAMPAIGNS AND SEARCH SETTINGS

  • Each primary adaptation condition uses five independent campaign seeds; each campaign is a complete evolution run from a specified initialization, with the judge frozen and evaluator outputs reserved for post-hoc measurement.Dataset-shuffling seeds are distinct from campaign replication.Source: §A.1 · p. 12

A.2 WEBSHOP DATA AND EXECUTION DETAILS

  • The WEBSHOP adaptation set contains 500 non-clothing tasks, and its held-out target set has the same domain composition with disjoint session IDs and is not exposed to proposal, judging, or selection.Source data comprise 500 clothing training tasks, 300 validation tasks, and 300 holdout tasks; judge development uses fixed seed-program trajectories and separate validation selection.Source: §A.2 · pp. 12–13
  • The selected judge combines an LLM call with deterministic calibration logic and returns a continuous near-success score, a binary passed field, and textual failure, reason, and repair diagnostics.Mean near success is used for candidate promotion; the binary field does not determine promotion, while diagnostics remain visible to the proposer.Source: §A.2 · p. 13
  • The judge and promotion rule receive no target evaluator outputs; target-visible interaction information is allowed, but explicit task identifiers, hidden evaluator access, and external campaign state are prohibited.Trajectory scrubbing also removes evaluator-derived rewards, success indicators, and evaluator-only annotations while retaining task instructions and ordinary observations.Source: §A.2 · p. 13

A.3 PUSHT TASK AND PROTOCOL DETAILS

The PUSHT control reconstructs coverage from visible block and goal geometry while adapting policies to changed dynamics. The optional source constraint filters target candidates using evaluator-measured source capability, but does not guarantee target performance or per-task source non-regression.

A.3 PUSHT TASK AND PROTOCOL DETAILS

  • In PUSHT, the agent program is an executable Python control policy mapping observed state to a two-dimensional target position; final coverage is block–goal intersection area divided by goal area.The proposer and judge observe block and goal poses, while stored coverage, success, and reward fields are withheld.Source: §A.3 · p. 13
  • The evolved PUSHT judge computes final polygon-intersection coverage from visible geometry and obtains 0.0000 MAE on both training and validation splits, while the fixed-rubric judge has coverage MAE of 0.0624 and 0.0742.The policy must adapt to changed dynamics while the reconstructed coverage rule remains unchanged.Source: §A.3 · p. 13

B.1 CONSTRAINT DEFINITION

  • The optional source constraint anchors reward-hidden target adaptation to evaluator-measured source capability, but guarantees only aggregate performance on a fixed source check set—not target performance or per-task source non-regression.Its task set, metric, reference policy, and tolerance are environment-specific; source-ineligible candidates can contribute to reported selection regret.Source: §B.1 · p. 13

C DETAILED ERROR DECOMPOSITION

The error analysis compares seed and evolved judge fidelity on source splits and evaluates fixed judges on matched target trajectories. The reported trade-off is better aggregate performance by the evolved judge on partial or exact outcomes, but better recognition of true-zero cases by the fixed-rubric judge.

C DETAILED ERROR DECOMPOSITION

  • The evolved judge has lower reported training and validation MAE than the seed judge: 0.1038 versus 0.1685 on training and 0.0982 versus 0.1637 on validation.The main paper reports prediction error on a source holdout excluded from judge development.Source: C DETAILED ERROR DECOMPOSITION · p. 14
  • The matched target cross-scoring analysis scores the same 500 reward-scrubbed auxiliary WEBSHOP trajectories with both frozen judges, isolating prediction comparison on a fixed trajectory set.Table 4 covers source-holdout fidelity and target auxiliary partial-reward trajectories; Table 5 reports aggregate accuracy and error by reward regime, with lower MAE and higher Spearman ρ better.Source: C DETAILED ERROR DECOMPOSITION · p. 14, Table 4 · p. 14, Table 5 · p. 14
  • The evolved judge’s aggregate advantage comes from partial or exact outcomes, while the fixed-rubric judge better recognizes the small number of true-zero cases, a trade-off compatible with dense optimization and occasional severe false optimism.Table 5 decomposes error by ground-truth reward regime.Source: C DETAILED ERROR DECOMPOSITION · p. 14, Table 5 · p. 14

D PRIMARY CAMPAIGN DIAGNOSTICS

The campaign diagnostics distinguish retained-program rewards from the best candidate generated so far, and report additional oracle and regret measures averaged across campaigns. Hidden rewards support post-hoc analysis rather than online selection.

D PRIMARY CAMPAIGN DIAGNOSTICS

  • Figure 1’s cold-start curves average independent campaigns: panel (a) tracks retained-program rewards by round, while panel (b) tracks each campaign’s best generated candidate through that round, including rejected candidates.The searches use 40 valid evolution rounds and 500 adaptation tasks; hidden rewards are used only for post-hoc analysis.Source: D PRIMARY CAMPAIGN DIAGNOSTICS · p. 15
  • Table 6 reports additional primary-campaign diagnostics, with oracle-best reward and regret computed within each campaign’s generated candidates before campaign averaging.Success is measured on adaptation tasks, and oracle-best, regret, and exact success use hidden reward only post hoc.Source: Table 6 · p. 15, D PRIMARY CAMPAIGN DIAGNOSTICS · p. 15

E OFFLINE JUDGE CONTROL PROCEDURES

The offline controls rescore saved trajectories and compare judge predictions and selections, including across prefixes of candidate search. They also describe source-fitted calibration and a deterministic scoring variant, while noting that saved-candidate comparisons do not isolate candidate-generation effects.

E OFFLINE JUDGE CONTROL PROCEDURES

  • Each judge selects the candidate with the highest mean predicted score across 500 tasks; audited reward, oracle-best reward, and regret are computed within each campaign before averaging.The cross-scoring controls rescore saved trajectories from five primary cold-start campaigns, with the agent execution held fixed.Source: E OFFLINE JUDGE CONTROL PROCEDURES · p. 15
  • Source-fitted isotonic calibration maps fixed-rubric scores to source rewards by unweighted least-squares regression, without validation, holdout, or target labels.Scores outside the observed training range use fitted boundary values; the nonlinear mapping can change candidate rankings, and its squared-error objective differs from judge evolution’s MAE objective.Source: E OFFLINE JUDGE CONTROL PROCEDURES · pp. 15–16
  • The deterministic evolved judge scores extracted trajectory features using structural scoring and evolved postprocessing, leaving LLM-generated fields empty and making no fresh LLM judgment.The intervention retains extraction rules, scoring tables, calibration functions, and missing-field defaults.Source: E OFFLINE JUDGE CONTROL PROCEDURES · p. 16
  • Prefix regret is the gap between the best candidate’s mean and the judge-selected candidate’s mean at each search stage, averaged across five campaigns.The analysis counts positive-regret prefixes among 40 post-initialization stages and averages campaign-specific maximum regret.Source: E OFFLINE JUDGE CONTROL PROCEDURES · p. 16
  • Saved-candidate prediction and selection comparisons do not isolate the effects of promotion, diagnostics, or proposal randomness on candidate generation.The main paper reports mean positive-regret prefixes and mean campaign-specific maximum regret.Source: E OFFLINE JUDGE CONTROL PROCEDURES · p. 16

F JUDGE BASELINES AND COMPONENT ABLATION RESULTS

The ablation series varies judge-development procedures and the availability of diagnostics or numerical score adjustments while retaining the primary data and search protocol. Table 7 reports mean true rewards and standard deviations across campaigns, with held-out evaluation on separate target tasks.

F JUDGE BASELINES AND COMPONENT ABLATION RESULTS

  • The ablation series retains the primary data and search protocol while varying judge-development procedures and the availability of diagnostics or numerical score adjustments.It uses five cold-start campaigns, 20 proposer rounds, and 40 target-search rounds.Source: F JUDGE BASELINES AND COMPONENT… · p. 16
  • Held-out evaluation uses frozen selected programs on separate target tasks.The retained task sets include 500 disjoint held-out target tasks.Source: F JUDGE BASELINES AND COMPONENT… · p. 16
  • Table 7 reports mean true reward and standard deviation across campaigns for each judge baseline and component-ablation condition.The table’s caption identifies the reported measures but does not provide condition values in these passages.Source: Table 7 · p. 16

Related papers

Loading 2609.34126v1…