PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments

Zhilin Wang, Han Song, Runzhe Zhan, Jusen Du, Jiacheng Chen, Tianle Li, Qingyu Yin, Yulun Wu, Zhennan Shen, Tong Zhu, Yanshu Li, Guanjie Chen, Derek F. Wong, Yafu Li, Yu Cheng, Yang Yang

arXiv:2607.02440v1cs.AIcs.CL

TL;DR

Existing evaluations often reduce autonomous improvement to final scores or confound it with open-ended engineering. EvoPolicyGym introduces a controlled benchmark for iterative policy revision under bounded feedback, where GPT-5.5 achieves the highest Core16 score and top-two placement across all 16 environments.

  • Problem

    Existing evaluations often collapse feedback-driven improvement into final scores or confound it with open-ended software-engineering progress, limiting controlled process-level assessment.

  • Method

    EvoPolicyGym evaluates agents that repeatedly edit executable policies under a fixed interaction budget, with visible train feedback and hidden validation-selected held-out performance.

  • Results

    GPT-5.5 obtains the highest Core16 score (0.891), with nine wins and top-two placement on all 16 environments.

  • Takeaways & Limitations

    Strong autonomous policy evolution involves discovering task-appropriate mechanisms, translating feedback into code changes, and refining policies under budget pressure.

  • Takeaways & Limitations

    The trajectory diagnostics are conservative proxies rather than semantic proofs and exclude generated data files, learned weights, and unreferenced experiments.

Abstract

from arXiv · show

Autonomous agents are increasingly expected to improve executable policies through feedback, yet existing evaluations often collapse this process into a final score or confound it with open-ended software-engineering progress. We introduce Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget. We instantiate this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments that evaluates how agents iteratively improve explored policies. On the EvoPolicyGym suite, GPT-5.5 achieves the strongest aggregate rank score and top-two performance on all 16 environments. Beyond leaderboard results, EvoPolicyGym also provides trajectory-level diagnostics that distinguish how agents allocate budget, convert feedback into parametric tuning. These analyses show that strong autonomous policy evolution depends not only on isolated task wins, but on discovering task-appropriate mechanisms and refining policies under bounded feedback.

1 Introduction

The paper formalizes Autonomous Policy Evolution as controlled, feedback-driven revision of executable policies, targeting held-out generalization rather than observed performance alone. EvoPolicyGym instantiates this setting with bounded interaction, hidden validation, and trajectory-level diagnostics.

  • Motivation: The setting isolates conversion of bounded environment feedback into generalizable executable-policy improvements while retaining iterative decisions.This addresses confounders from fully open-ended engineering tasks, including evolving specifications and software-maintenance quality.
  • Autonomous Policy Evolution: Autonomous Policy Evolution evaluates agents that repeatedly revise executable decision policies using feedback from prior deployments.The observable process is a sequence of submitted policy systems and train-feedback records.
  • EvoPolicyGym: EvoPolicyGym evaluates repeated policy edits under a fixed interaction budget, server-generated sandboxed-rollout feedback, and hidden held-out generalization.The benchmark uses compact interactive environments and strict visibility boundaries with trajectory-level feedback.
  • Preliminary results: GPT-5.5 obtains the highest aggregate rank score and top-two performance on all 16 environments, while Claude Opus 4.7 leads the MiniGrid family.Experiments evaluate four harness–model agents on Core16 under a common 128-episode interaction budget.
  • Diagnostics: Trajectory-level diagnostics distinguish structural synthesis from parameter tuning and connect policy improvement to budget-conditioned evolution and audited trace revisions.Audited CarRacing and BipedalWalker traces illustrate how agents translate visible feedback into revisions.

2 Related Work

Prior work spans repository-level coding-agent evaluation, feedback-driven self-improvement, interactive-agent benchmarks, and bounded optimization. EvoPolicyGym is positioned relative to these settings by focusing on iterative policy improvement in standard reinforcement-learning environments through environment interaction.

  • From static patches to long-horizon coding-agent evaluation: Repository-level benchmarks evaluate coding agents by having them edit real repositories and validate patches against unit tests.Related systems emphasize repository navigation, tool use, and interactive debugging in realistic workflows.
  • From static patches to long-horizon coding-agent evaluation: Long-horizon software benchmarks study repeated modifications beyond one-shot patch success, including quality degradation and multi-file consistency challenges.These benchmarks address software evolution under ongoing revision, extension, and maintenance.
  • Feedback-driven self-improvement: Reflexion and Self-Refine use language-level reflection, while Voyager, Eureka, FunSearch, and AlphaEvolve iteratively improve executable artifacts.The executable artifacts include skills, reward functions, and programs.
  • Evaluation of interactive and self-improving agents: Interactive benchmarks assess tool use, state tracking, and multi-step decision making in web, operating-system, database, and workplace environments.These settings are primarily episodic and focus on task completion within single interactions, limiting their coverage of iterative improvement.
  • Bounded optimization and trajectory-level analysis for agents: Frontier-Eng studies generative engineering design under bounded feedback and computation, whereas this work examines policy improvement in standard reinforcement-learning environments through environment interaction.The distinction is between broad engineering environments and iterative improvement of decision-making policies.

3 EvoPolicyGym: A Framework for Autonomous Policy Evolution

EvoPolicyGym evaluates autonomous policy evolution as a fixed-budget loop in which a coding agent repeatedly edits and submits executable policy systems using environment feedback. Runs are scored by the held-out return of the best hidden-validation checkpoint, while trajectories diagnose how the outcome was reached.

  • Evaluation and diagnostics: A complete budget-constrained run is scored by the held-out return of its best validation checkpoint, with trajectories providing diagnostic evidence about how that outcome was reached.The server performs hidden validation selection and held-out evaluation after the run ends.
  • Executable policy systems: The policy system is an executable Python bundle with a reset/act interface that may contain helper modules, planners, memory, diagnostics, controllers, or learned parameters.The environment remains server-owned, while the submitted artifact maps observations to actions through the policy-system entry point.
  • Autonomous policy-evolution loop: The framework has a coding agent inspect feedback, edit a persistent policy workspace, and choose when and how much to submit under a fixed episode budget.The agent observes workspace state, prior server-written feedback, and remaining budget while writing workspace patches.
  • Visibility boundary: Train episodes are the only source of environment evidence during optimization, while validation and held-out evidence remain hidden until the loop ends.Observable feedback can include returns, statuses, trajectories, diagnostics, errors, frames, and videos.
  • Environment interface: The environment layer uses Gymnasium-compatible reset/step tasks through adapters that standardize schemas, reproducible initialization splits, and hidden split metadata.The design keeps interaction, visibility, budget, and artifact semantics fixed as environment coverage grows.

4 Experiments

Experiments evaluate model-and-harness systems on Core16 under a fixed 128-episode optimization budget, using validation-selected held-out returns and aggregate rank scores across heterogeneous environments. GPT-5.5 leads overall through broad coverage, while trajectory diagnostics reveal when candidate-policy improvements occur during budget consumption.

  • Experimental setup: Core16 spans 16 environments across Gym / Box2D, MuJoCo, MiniGrid, and Robotics / Driving, with 128 training episodes plus reserved validation and held-out cases.Each model is evaluated together with its coding harness.
  • Aggregate results: GPT-5.5 achieves the highest Core16 score (0.891), nine wins, and top-two placement on all 16 environments.The aggregate score is based on rank scores across heterogeneous tasks rather than directly comparable reward scales.
  • Aggregate results: Claude Opus 4.7 ranks second (0.750), with five wins and 12 top-two placements, and achieves the best MiniGrid family score (0.938).Its strengths include wins on ContinuousCar, Ant, KeyCorridor, FourRooms, and ObstructedMaze.
  • Family-level results: GPT-5.5 leads Gym / Box2D, MuJoCo, and Robotics / Driving, whereas Claude Opus 4.7 is strongest on MiniGrid.Task-family specialization explains why local wins and overall reliability do not always coincide.
  • Trajectory diagnostics: Trajectory diagnostics reconstruct best-so-far hidden-validation scores over consumed budget, with jumps marking improved policies and plateaus marking budget without improvement.These post-hoc curves distinguish early discovery from late improvement despite similar selected-checkpoint scores.

5 Mechanisms of Policy Evolution

EvoPolicyGym distinguishes structural synthesis from parametric tuning to explain how agents improve executable policies under different task demands. The diagnostics show that strong performance depends on task-appropriate machinery, targeted revisions, and feedback-guided candidate management, while the measures remain conservative proxies rather than semantic proofs.

  • Exploration modes: Structural synthesis builds task machinery such as perception, memory, planning, reward interpretation, or state abstraction, whereas parametric tuning adjusts gains, thresholds, constants, and branch-local parameters.The distinction follows the policy system’s computational structure and parameters.
  • Task demands: Synthesis-dominant tasks select richer policy bundles with more functions, branches, loops, and persistent state, while tuning-dominant tasks use smaller, more compressed controllers.The split contrasts pixel-perception and symbolic-planning environments with lower-dimensional control tasks.
  • Held-out performance: 0.98 and 1.00 are GPT-5.5’s and Claude Opus 4.7’s synthesis-dominant held-out scores, compared with 0.19 and 0.03 for MiniMax-M3 and DeepSeek-V4-Pro.MiniMax-M3 and DeepSeek-V4-Pro solve none of the three locked-door MiniGrid tasks, while tuning-dominant scores cluster more tightly at 0.67–0.99; MiniMax-M3 reaches 0.83 on tuning.
  • Edit effectiveness: 41% and 48% are the synthesis-edit rates producing new validation bests for GPT-5.5 and Claude Opus 4.7, versus 10% and 3% for MiniMax-M3 and DeepSeek-V4-Pro.Same-topology edits rarely rescue a wrong mechanism but become useful on tuning-dominant tasks once the controller family is sufficiently close.
  • Feedback-guided evolution: Successful traces connect visible failures to targeted structural edits, then use later feedback to select or roll back candidates; weaker runs churn mechanisms without escaping the wrong abstraction.Across diagnostics, higher-scoring runs associate successful structural edits with feedback-linked policy revisions on synthesis-dominant tasks.
  • Limitations: AST topology and source-bundle analysis are conservative proxies: they capture objective code changes but do not prove semantic differences or cover generated data, learned weights, or unreferenced experiments.The source-bundle boundary includes policy.py and reachable self-written helper modules, while the synthesis/tuning split remains a lens rather than a taxonomy.

6 Conclusion

EvoPolicyGym evaluates autonomous policy improvement as the evolution of executable systems under fixed interaction budgets and visible feedback. Its leaderboard and trajectory diagnostics measure stable, feedback-driven policy evolution beyond isolated task wins.

  • EvoPolicyGym evaluates agents that iteratively edit executable policies under fixed interaction budgets, learning from visible training feedback and hidden validation.
  • Strong policy evolution requires task-appropriate abstractions, mechanism-level code changes driven by feedback, and preservation of useful candidates under budget pressure.
  • Pairing leaderboard scores with trajectory-level diagnostics provides a concrete protocol for measuring stable, feedback-driven autonomous policy evolution.

A Benchmark Object and Task Suite · A.1 Run Protocol and Policy System · A.2 Core16 Suite

EvoPolicyGym evaluates autonomous improvement of executable policies through repeated edits under a fixed interaction budget, using a server-mediated run protocol and the 16-environment Core16 suite. The benchmark combines heterogeneous task families while preserving trajectory-level feedback and within-environment ranking for aggregate analysis.

  • A.1 Run Protocol and Policy System: Each run provides a live workspace, one executable policy system, and a fixed interaction budget for iterative improvement.The server stages AGENTS.md, initializes system/, exposes a local HTTP service, and keeps validation and held-out evaluation server-side.
  • A.1 Run Protocol and Policy System: The policy entry point is system/policy.py, where a top-level Policy class implements reset and act using environment-provided spaces and metadata.Agents can add modules, configurations, tests, weights, memory files, and analysis utilities under system/.
  • A.1 Run Protocol and Policy System: 128 episodes define each Core16 run, with 1–128 episodes permitted per submit and budget charged by the expanded requested case-index list.Valid submissions snapshot system/; execution-stage failures still consume budget, while request-format failures do not.
  • A.1 Run Protocol and Policy System: Completed submits write summaries containing returns, lengths, errors, timing, remaining budget, and aggregate statistics, alongside step-level trajectory records for successful episodes.Feedback may also include policy streams and environment-specific media, while submit-level and per-episode failures produce separate error records.
  • A.1 Run Protocol and Policy System: Live workspace edits have no-rollback semantics, while immutable submitted checkpoints support hidden validation and final held-out evaluation.Harmful valid edits remain visible until the agent repairs or restores them.
  • A.2 Core16 Suite: Core16 contains 16 implemented Gymnasium-compatible scenarios from four environment families, spanning control, visual driving, locomotion, symbolic partial observability, manipulation, and trafficstyle control.The suite requires mechanisms including controllers, visual abstractions, world models, phase machines, and recovery logic.
  • A.2 Core16 Suite: The four-by-four organization gives each family equal category-level weight while retaining heterogeneous policy interfaces across state-control, CarRacing, MiniGrid, and Fetch tasks.Raw returns are reported per environment and aggregated only after within-environment ranking.

B Evaluation Protocol and Agent Configuration · B.1 Visibility, Selection, and Scoring · B.2 Agent and Run Configuration

The protocol separates visible train feedback from hidden validation and held-out evaluation, selecting checkpoints by validation performance before reporting held-out returns. All leaderboard agents share a fixed interaction budget and run configuration, while rank-based aggregation and audit traces support cross-environment comparison and behavioral analysis.

  • B.1 Visibility, Selection, and Scoring: Train cases expose integer handles and in-loop feedback, while hidden validation contains 16 cases and hidden held-out evaluation contains 32 cases per environment.Agents never observe hidden case identities, trajectories, returns, or failure details during optimization.
  • B.1 Visibility, Selection, and Scoring: After the 128-episode train budget, the server selects the status == ok checkpoint with highest validation mean return, breaking ties by later submit.The selected checkpoint is then evaluated on the held-out split, whose raw mean return is reported per environment.
  • B.1 Visibility, Selection, and Scoring: Core16 aggregation ranks each agent against four reported agents plus a uniform random-policy reference within every environment.Category scores average per-environment scores across four environments, while the Core16 score averages across all 16 environments.
  • B.1 Visibility, Selection, and Scoring: Raw held-out means measure selected-policy strength per environment, whereas rank-normalized scores measure consistency under comparable interaction budgets across reward scales.Aggregate scores are analysis metrics; validation selection and held-out evaluation use raw environment returns.
  • B.1 Visibility, Selection, and Scoring: Runs record submits, budget use, validation curves, checkpoint selection, invalid transitions, score drops, complexity growth, wall time, and available token accounting.These audit traces support the behavior analysis in Section 5.
  • B.2 Agent and Run Configuration: All four leaderboard agents use the Core16 list, a 128-episode train interaction budget, hidden validation size 16, and hidden held-out size 32.The shared run configuration also specifies minimum submit size 1, maximum submit size 128, external case files, and local loopback execution.
  • B.2 Agent and Run Configuration: Claude Code-compatible runs expose Bash, Read, Edit, Write, Glob, and Grep, while the Codex run uses a persistent logical session; retries do not consume environment interactions.Retry events also leave server-side budget accounting unchanged.
  • B.2 Agent and Run Configuration: The shared run budget fixes environment-interaction comparisons, but harness differences affect context and tool traces, making token and cost statistics diagnostic rather than scored.This preserves interaction-budget comparability while excluding implementation-dependent resource measures from evaluation.

C Supplementary Analysis Diagnostics

This section expands the quantitative diagnostics behind the synthesis/tuning split, harness context traffic, and checkpoint edit-size effects. These diagnostics explain behavior without affecting validation selection, held-out evaluation, or leaderboard rank.

  • Ability diagnostics: The ability table reports per-environment values underlying the synthesis/tuning split.It provides the quantitative basis for that distinction.
  • Token diagnostics: The token table characterizes harness-level context traffic.This diagnostic focuses on how context moves through the harness.
  • Edit-size diagnostics: The edit-size plot summarizes how checkpoint-change magnitude relates to visible improvement.Together, these diagnostics explain behavior but do not affect validation selection, held-out evaluation, or leaderboard rank.

C.1 Per-Environment Relative Held-Out Performance by Task Demand · C.2 Token and Cost Accounting

Per-environment scores are normalized relative to random and best evaluated-agent anchors on held-out pools. Token accounting diagnoses optimization and implementation overhead while remaining separate from leaderboard evaluation tied to fixed interaction budgets.

  • C.1 Per-Environment Relative Held-Out Performance by Task Demand: C.1 scores are scaled from 0 for a random policy to 1 for the best evaluated agent performance on each environment.The scaling is environment-specific.
  • C.1 Per-Environment Relative Held-Out Performance by Task Demand: C.1 random anchors are measured on the same held-out pools used for per-environment evaluation.
  • C.2 Token and Cost Accounting: C.2 token accounting is diagnostic and excluded from the leaderboard score.
  • C.2 Token and Cost Accounting: C.2 accounting separates non-cached input, cache read/creation, and output tokens because cache events overlap semantically with previously supplied context.
  • C.2 Token and Cost Accounting: C.2 values are parsed from agent stream logs and reported in millions, with cache-token handling differing between Codex and Claude Code streams.For Codex, non-cached input subtracts cached_input_tokens from input_tokens; Claude Code reports cache read and cache creation separately.
  • C.2 Token and Cost Accounting: C.2 token use varies substantially across tasks and harnesses, and cache traffic can dominate non-cached input in several runs.This is especially observed when long interaction histories are carried across repeated revisions.
  • C.2 Token and Cost Accounting: C.2 accounting helps interpret optimization behavior and implementation overhead, while the leaderboard remains tied to fixed environment interaction.

C.3 Edit-Size Diagnostic

The edit-size diagnostic shows that same-hash transitions cannot improve, while small and medium edits define the common local-search regime. Larger edits can yield useful mechanism synthesis or destructive changes, with agents differing in how reliably they consolidate gains into validation-best checkpoints.

  • Edit-size regimes: Same-hash transitions produce zero improvement by construction, while small or medium edits form the common local-search regime.These edit bins describe the dominant local-search behavior across adjacent submissions.
  • Edit-size regimes: GPT-5.5 and Claude Opus 4.7 improve on a meaningful share of larger structural edits by introducing mechanisms and then consolidating them.Their qualitative traces align larger-edit improvements with mechanism introduction followed by consolidation.
  • Edit-size regimes: MiniMax-M3 obtains some large gains from rewrites but consolidates them less reliably, whereas DeepSeek-V4-Pro less often turns large edits into validation-best checkpoints.The comparison concerns the reliability of converting large structural changes into validated improvements.
  • Edit-size regimes: Large edits may represent useful mechanism synthesis, destructive rewrites, or interface repair depending on the surrounding run context.The figure treats the association between edit size and improvement probability as diagnostic rather than causal.

D Policy Mechanism Case Studies

Successful submitted policies are compact, stateful programs that convert observations into task-specific abstractions, controllers or planners, and recovery mechanisms. Across CarRacing, HalfCheetah, ObstructedMaze, and FetchPush, visible feedback supports mechanism-level refinement of these policies.

  • CarRacing: CarRacing policies use road-mask lookahead, curvature and edge warnings to control speed and steering, with explicit recovery when visual confidence drops.They convert pixels into near-, mid-, and far-road centers and close the loop with recovery toward the last track.
  • HalfCheetah: HalfCheetah policies synthesize compact periodic gaits, then apply clipping and posture-based amplitude scaling while tuning phase, amplitude, and frequency.The policy stores gait parameters and scales actions when root height exceeds a safety threshold.
  • ObstructedMaze: ObstructedMaze policies maintain an egocentric symbolic map, update pose from actions, and use BFS planning for exploration, object handling, doors, and blockers.The same planner supports frontier exploration, toggling doors, pickup, drop, and obstacle clearing.
  • FetchPush: FetchPush policies use geometric phase control to move behind an object, lower to pushing height, and drive toward beyond the goal, adding clearance when needed.The clearance phase addresses cases where the gripper begins on the wrong side of the object.
  • Cross-task pattern: Across four examples, useful policies are small stateful programs that combine task abstractions with controllers or planners and recovery logic for feedback-exposed failures.This mechanism-level pattern corresponds to synthesis and parametric-tuning behavior in the main analysis.
Loading 2607.02440v1…