PaperlayerView exact revision on arXiv

Source-linked AI summary

PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

Yuhao Zhan, Bingxiang He, Zecong Tang, Chaojun Xiao

arXiv:2608.14441v1cs.AI

TL;DR

Existing evaluations rarely test whether agents can recover when changing execution conditions invalidate a previously successful design. PACE-Bench evaluates this capability through executable physical designs and simulator feedback, finding that current agents remain far from saturation.

  • Problem

    Existing evaluations do not test whether agents can adapt a previously successful design after execution-environment changes cause it to fail.

  • Method

    PACE-Bench evaluates iterative revision of executable physical designs across 144 source-to-target pairs spanning six physics domains, with fixed goals and interfaces and diagnostic feedback.

  • Results

    Current agents remain far from saturation: Reflexion with Qwen3-14B succeeds on only 35.9% of full-benchmark pairs.

  • Takeaways & Limitations

    Simulator-grounded reflection is more reliable than unverified self-revision, while mechanism redesign remains the central bottleneck over parameter inference.

  • Takeaways & Limitations

    Findings come from controlled 2D Box2D simulations and may not transfer to 3D environments or real robots with noisy sensing and actuation.

Abstract

from arXiv · show

Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change. To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. Each pair links a source environment to a mutated target environment with the same goal and interface. A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. We compare ten self-evolving methods from four paradigms. The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9\% of full-benchmark pairs, while GPT-5.5 solves 66.7\% of the Statics subset under the full budget. Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. Data and code are available at https://github.com/thunlp/PACE-Bench.

1 Introduction

PACE-BENCH addresses the untested problem of adapting executable physical designs after controlled environment changes, evaluating whether agents can recover when mutations invalidate source-environment solutions. Results show the benchmark remains unsaturated and that simulator-grounded reflection is more reliable than unverified revision, while mechanism redesign—not parameter inference—limits performance.

  • Motivation: Existing evaluations largely use fixed execution rules, task streams, or transfer settings, leaving adaptation after an environment change untested.These benchmarks assess experience accumulation, generalization, or transfer rather than revising an existing design after physical conditions shift.
  • Benchmark: PACE-BENCH evaluates self-evolving adaptation by mutating physical parameters so a code-driven design that succeeds in a source environment must be revised for a target environment.Physical changes include friction, material strength, and dynamics, while the design remains executable and its goal is preserved.
  • Results: 35.9% of full-benchmark pairs are solved by Reflexion with Qwen3-14B, while GPT-5.5 solves 66.7% of the Statics subset under the full budget.These results indicate that the benchmark remains far from saturated across model scales and selected subsets.
  • Findings: Simulator-grounded reflection outperforms unverified self-revision, whereas memory can anchor agents to early designs and broad tree search often fails to convert exploration into convergence.Code-similarity and nine-category error analyses associate these behaviors with Design Fixation, Stagnation, and undirected Exploration.
  • Findings: Revealing exact physical changes does not raise the performance ceiling, indicating that mechanism redesign (“know how”) is the bottleneck rather than parameter inference (“know what”).The result frames adaptation as redesigning how a system works, not merely identifying which parameters changed.

2 Related Work

Existing self-evolving benchmarks evaluate feedback-driven improvement, knowledge and skill transfer, or generalization across modified tasks and environments. None tests whether agents can adapt a design after the environment changes while the goal and interface remain fixed.

  • Self-Evolving Benchmarks and Physical Design: Existing benchmarks assess performance improvement over task streams, including long-term performance, efficiency, and stability.They also examine whether learned knowledge and skills transfer to related tasks.
  • Self-Evolving Benchmarks and Physical Design: Other benchmarks modify prompts, code, or agent harnesses to test generalization to held-out tasks and heterogeneous environments.
  • Self-Evolving Benchmarks and Physical Design: None tests whether an agent can adapt its design when the environment changes while the goal and interface stay fixed.

3 The PACE-BENCH Benchmark

PACE-BENCH evaluates whether agents can adapt source solutions to mutated target environments that preserve the task goal and interface but alter physical conditions. Its 144 source-to-target pairs span six physics domains and test both hidden-parameter inference and mechanism redesign under structured feedback and a 20-attempt budget.

  • Benchmark structure: Each base task defines one source environment and four mutated stages sharing the same goal and permitted primitive APIs while differing in physical conditions.The environments are parameterized by terrain and physics, and solutions are Python programs evaluated in a Box2D sandbox.
  • Adaptation protocol: Agents must adapt a source solution that is invalid in the target without being told which physics changed, using feedback to diagnose changes and redesign the mechanism.The target remains solvable, while the source solution is explicitly excluded from the target solution set.
  • Adaptation protocol: 20 attempts constrain iterative adaptation, with each candidate receiving constraint-violation fractions, task scores, and diagnostic reports from the target environment.The feedback includes examples such as peak joint force and failure timestamp, while the score distinguishes constraint satisfaction from task progress.
  • Dataset construction: 144 pairs come from 36 base tasks across six physics domains, with difficulty increasing from Stage 1 through Stage 4.Each domain contributes six base tasks, and each base task yields four source-to-target pairs.
  • Benchmark scale: 1,051-token prompts average 7 hard constraints and 8 primitive APIs, while mutated parameters increase from 2 at Stage 1 to 10 at Stage 4.The benchmark contains 180 evaluation environments, 273 hard constraints, 292 primitive APIs, and 945 stage-level parameter mutations.

4 Experiments

Across dynamic-physics tasks, Reflexion performs best among self-evolving methods but reaches only 35.9% Pass@2, while larger models and extra information provide uneven gains. Failure analyses show memory anchoring, nonconvergent exploration, and task-category difficulty remain central barriers.

  • Overall Performance: 35.9% Pass@2 is achieved by Reflexion + Qwen3-14B, the strongest configuration, leaving the full benchmark far from saturated.GPT-5.5 performs better than Qwen3-14B on Statics, yet still fails one third of pairs.
  • Overall Performance: ToT reaches 23.4% Pass@2 at 4B, or 2.0× Vanilla, but its advantage erodes with scale while CodeEvolve declines from 10.7% to 5.3%.Memory-augmented methods anchor search to early designs, whereas broad search explores without converging.
  • Failure Analysis: 47.1% Design Fixation affects ACE, whereas ToT has 0.3% Fixation but 50.2% Stagnation, illustrating anchoring versus exploration without convergence.CodeEvolve shows the opposite pattern: 66.3% Fixation and 0.0% Exploration.
  • Failure Analysis: 58.2% Design Fixation occurs in Control, while Exotic Physics is most solvable and Dynamics hardest; category effects exceed model-scale effects.The 4B→14B gain ranges from +16.3 points in Exotic Physics to +1.9 points in Kinematics.
  • Information Interventions: Exact change values improve all 4B configurations by +2.6 to +8.4 points, but five of six larger-model changes remain within ±5 points.Video feedback helps memory-based methods at larger scales but harms context-based methods, including Vanilla 14B dropping from 40.9% to 18.2%.

5 Conclusion

PACE-BENCH evaluates code-driven adaptation across 144 source-to-target environment pairs spanning six physics domains, requiring agents to revise executable designs under hidden mutations and simulator-grounded feedback. The benchmark remains unsaturated, with most self-evolving methods offering limited gains over Vanilla or underperforming it.

  • Benchmark design: PACE-BENCH contains 144 solvable but non-trivial source-to-target environment pairs across six physics domains.The pairs include systematic hidden environment mutations and simulator-grounded feedback.
  • Adaptation setting: Within 20 attempts, agents must infer environmental changes and revise executable code-driven designs for the target physics.The adaptation process uses simulator-grounded feedback.
  • Findings: Frontier LLMs remain unsaturated on the benchmark.The passage characterizes the benchmark as not yet saturated for frontier language models.
  • Findings: Most self-evolving methods deliver limited gains over Vanilla or underperform it.This conclusion compares the methods against the Vanilla baseline.

Limitations

PACE-Bench’s controlled 2D simulations, fixed 20-attempt budget, and selectively scoped evaluations limit conclusions about transfer to real-world settings, sustained adaptation, and broader method or domain coverage.

  • Transferability: PACE-Bench uses controlled 2D Box2D simulations, so findings may not transfer to 3D environments or real robots with noisy sensing and actuation.Future work should extend the benchmark to richer simulators and real hardware.
  • Evaluation horizon: A fixed 20-attempt budget limits the interaction horizon, leaving sustained adaptation untested.The benchmark therefore does not evaluate adaptation over longer interaction horizons.
  • Evaluation scope: Due to computational cost, frontier-model, parameter-disclosure, and visual-feedback evaluations cover only selected domains and methods.These evaluation settings do not provide full-benchmark coverage.

Ethical Considerations

PACE-Bench evaluates physical intelligence in simulated, synthetic tasks without human subjects, personal data, or real-world physical consequences. The authors acknowledge potential dual-use implications of automated mechanical design and promote responsible, transparent, reproducible development.

  • PACE-Bench evaluates physical intelligence in LLM agents using synthetic simulation tasks.
  • The benchmark involves no human subjects, personally identifiable information, or safety-critical deployment.
  • The authors encourage responsible development because automated mechanical design may have dual-use implications.
  • Public release supports transparency and reproducibility while keeping all tasks free of real-world physical consequences.

A Dataset Construction Details … A.4 Human Verification

PACE-Bench combines automated module auditing, staged difficulty escalation, physics-grounded diagnostic feedback, and independent human verification to construct and validate source-to-target adaptation tasks. Its design preserves information asymmetry while requiring agents to discover changed physics through interaction and failure analysis.

  • A.1 Module Auditing: Seven automated audit rules check cross-module consistency and constraint completeness across each task’s environment, evaluator, feedback, prompt, stages, and renderer modules.The audit checks coherence of physical parameters, constraints, and success criteria, including hardcoded constraint values such as mass budgets and force limits.
  • A.1 Module Auditing: Visible variables are disclosed precisely, while invisible variables are scrubbed and the Uniform Suffix U provides only generic awareness that shifts may exist.This information asymmetry forces agents to discover environment-specific physics through interaction.
  • A.2 Difficulty Escalation: Qwen3-4B serves only as a difficulty check, triggering harder mutations when it achieves s(x) = 100 on either of two independent runs.Mutations can widen gaps, increase gravity, or tighten force limits; the target reference solution is rerun and revised until it scores 100 before reevaluation.
  • A.3 Feedback Design: Each task’s feedback.py converts raw simulation metrics into physics-grounded diagnostics with numerical quantities, timestamps, and locations.Reported quantities include peak joint forces and torques, collision impulses, structural deformation, and constraint margins.
  • A.3 Feedback Design: Feedback uses current environment thresholds, identifies failures without prescribing fixes, and accounts for distinct simulation phases.A diagnostic can state that a joint broke at step 42 because torque exceeded its limit by 35%.
  • A.4 Human Verification: Two graduate-level physics or engineering authors independently audited three categories each, manually cross-checking all 36 tasks and LLM-generated modules against executable behavior.For every task, auditors executed the reference solution in both source and target environments and recorded s = 100.
  • A.4 Human Verification: Forensic failure analysis evaluates Qwen3-4B on all four target environments and uses its consistently failing execution logs as diagnostic evidence.An LLM analyzes JSON logs across temporal chronology, spatial margins, load and stress distribution, energy flow and loss mechanisms, and constraint satisfaction.

B Error Taxonomy Judging Criteria

Failed runs are judged by a priority-ordered classifier that assigns exactly one error type, prioritizing hard failures before trajectory-level patterns and using budget exhaustion as a fallback. Hard-failure labels are defined by score, constraint, structural, and numerical-instability criteria.

  • Classification procedure: Each failed run receives a single error type from a priority-ordered classifier, with the first matching condition determining the label.The priority sequence begins with hard failures, followed by trajectory-level patterns, and ends with Budget Exhaustion as fallback.
  • Error categories: Hard failures are checked before trajectory patterns: Catastrophic Collapse, Constraint Violation, Structural Failure, and Numerical Instability.The listed trajectory-level patterns are Design Fixation, Late Convergence, Stagnation, and Exploration.
  • Judging criteria: Catastrophic Collapse is assigned when the best score is ≤−60, while Constraint Violation requires any hard-constraint flag in the best attempt.Structural Failure indicates joint or beam breakage, and Numerical Instability includes NaN, Inf, or solver-detected numerical failure.
  • Judging criteria: Budget Exhaustion is the fallback label for runs that make progress but exhaust all available attempts.This fallback follows the hard-failure and trajectory-pattern checks in the classifier priority order.

C Method Adaptation Details … D.4 Run Reliability and Cost Tradeoff

PACE-Bench adapts diverse self-evolving methods to a common Box2D feedback-and-revision protocol, while its implementation, cost, validity, and reliability analyses quantify practical constraints. Two independent runs generally support reproducible conclusions, but additional runs provide limited marginal value relative to their computational and API costs.

  • C Method Adaptation Details: All methods revise Python programs from Box2D diagnostic feedback within a shared 20-attempt budget, using one backbone model for auxiliary calls.The protocol preserves full autonomy across reflection, memory induction, and rule extraction variants.
  • C Method Adaptation Details: Reflexion accumulates up to three FIFO diagnoses, while beam search retains top-b states and generates n candidates per state.Beam search defaults to b = 3 and n = 2, with rounds constrained by (b × n + 1) × rounds ≤ max_iterations.
  • D.1 Implementation: The benchmark uses 60-FPS Box2D simulations, up to 10,000 steps per attempt, two independent runs per environment pair, and Previous-One + Best history truncation.The truncation retains the latest attempt and highest-scoring historical attempt to avoid context overflow.
  • D Supplementary Experiment Results: The supplementary analyses separately examine VLM video feedback, disclosed mutated variables, adjacent-attempt code similarity, and Vanilla from-scratch results.These analyses cover error-taxonomy shifts, code evolution across transitions, and results by model and target environment.
  • D.2 Cost: Approximately 9,500 trajectories and up to 190K LLM calls consume 4,200 A100 GPU-hours, while one extra local-model run would require about 2,100 GPU-hours.The constrained API study totals 1,440 calls and approximately $2,000, with an extra run adding approximately $1,000.
  • D.3 Result Validity and Denominators: 139.20 average valid instances per method-model cell, with standard deviation 4.67, explain why Pass@2 values need not be exact multiples of 1/144.Incomplete executable solutions are excluded because of truncation, parse failure, sandbox runtime crash, or context overflow.
  • D.4 Run Reliability and Cost Tradeoff: 86.6% pass/fail agreement and a zero median score difference across two full-benchmark runs indicate that most outcomes are not run-specific.Adding a third Kinematics run changes only 5.1% of pair-level pass decisions, suggesting limited marginal value from another repeat.

D.5 Larger-Model Results … E.2 Per-Attempt Similarity

Under a five-attempt budget, larger frontier models achieve more success and partial progress, while additional attempts widen scaling differences. Supplementary analyses show that performance improves monotonically with interaction budget and that ACE produces increasingly stable code revisions over time.

  • D.5 Larger-Model Results: Five-attempt evaluation compares six frontier LLMs with Qwen3-4B/8B/14B on the 24-pair Statics subset using Vanilla.Because this budget differs from the main protocol, comparisons are valid only within this setting.
  • D.5 Larger-Model Results: 0% Pass@2 is achieved by Qwen3-4B/8B/14B and MiniMax-M2.7 under the five-attempt budget.Qwen3’s Score@2 nevertheless rises from 0.0 to 6.4 with scale.
  • D.5 Larger-Model Results: 36.4% Pass@2 is achieved by GPT-5.5, leading the nonzero-success ranking ahead of Gemini-3.1-Pro, Claude-Opus-4.7, DeepSeek-V4-Pro, and Kimi-K2.6.Score@2 and Pass@2 produce the same ranking among models with nonzero success.
  • E Supplementary Analysis: No model succeeds with only one attempt (k = 1) when evaluated using Pass@2@k and Score@2@k.These metrics are aggregated across all methods after truncating each trajectory to k attempts.
  • E.1 Convergence Analysis: Performance rises monotonically with the interaction budget after k = 1.The analysis evaluates truncated trajectories using Pass@2@k and Score@2@k.
  • E.1 Convergence Analysis: The gap between model scales widens with more attempts, suggesting that larger models extract more value from additional interaction.This result comes from metrics aggregated across all methods.
  • E Supplementary Analysis: 0.52→0.75 and 0.69→0.92 are the first-to-last-five-transition mean-similarity changes for Vanilla and ACE, respectively.ACE remains higher overall, with global mean 0.77 versus 0.68.
  • E.2 Per-Attempt Similarity: ACE submits identical consecutive code at transitions 8 and 20, whereas Vanilla alternates between near-fixation and more changing revisions.Vanilla reaches similarity 0.963 at transition 12 in the reported case study.

E.3 Category-Level Analysis · F Prompts

Category-level results show a stable difficulty hierarchy and sharply varying scaling gains, while the benchmark prompts define exhaustive auditing, forensic feedback, and constrained difficulty escalation procedures.

  • E.3 Category-Level Analysis: Exotic Physics is consistently most solvable, while Dynamics remains hardest across model scales; scaling gains are largest in Exotic Physics (+16.3) and Fluid (+12.2).The smallest 4B→14B gain occurs in Kinematics (+1.9).
  • E.3 Category-Level Analysis: Statics produces diverse exploration but 34.1% Exploration without convergence, as nearly balanced structures can still collapse without a refinement gradient.Statics tests equilibrium reasoning; its reported radicality is 0.311.
  • E.3 Category-Level Analysis: Kinematics has the smallest scaling gain (+1.9) and 44.5% Budget Exhaustion, indicating combinatorial linkage-topology search as a likely bottleneck.Kinematics tests linkage topology design.
  • F Prompts: The benchmark construction prompts cover task auditing, diagnostic-feedback design, and difficulty escalation across the corresponding pipelines.These prompts are documented in §3.3 and Appendix §A.
  • F.1 Module Auditing Prompt: The module-auditing prompt requires exhaustive line-by-line violation reporting and checks cross-module consistency, invisible variables, constraint completeness, mutation synchronization, and suffix uniformity.It requires numeric task constraints and visible variables in prompt.py while prohibiting numeric invisible background-physics values there.
  • F.2 Feedback Optimization Prompts: The feedback workflow first performs forensic analysis across six diagnostic dimensions, then implements feedback.py to report limits, margins, chronology, severity, and task-specific failures without prescribing fixes.Rules require retrieving every limit from metrics.get(), avoiding hardcoded thresholds, and preserving existing defaults and pass/fail logic.
  • F.3 Difficulty Escalation Prompt: Difficulty escalation mutates only existing variables in stages.py and agent.py, preserves solvability by reference solutions, and verifies source passage with target failure across all four target environments.The workflow also runs a mock evaluation pipeline to catch runtime errors.

G Full Task Specifications

This section specifies 36 tasks across source descriptions and four mutated target environments, spanning statics, equilibrium, and dynamic locomotion challenges. The specifications vary physical parameters such as mass, forces, friction, gravity, wind, damping, and geometric constraints across stages.

  • Task specification overview: Tables 17–22 specify all 36 tasks, pairing each source task description with physical mutations in four target environments.The section presents detailed source-to-target task specifications.
  • Statics & Equilibrium: Statics tasks include vehicle bridges, seismic towers, cantilevers, pivot-balanced structures, shelters, and block stacking.Examples include S_01 through S_06, with increasingly varied structural and environmental conditions.
  • Statics & Equilibrium: Bridge mutations combine tighter structural limits with changing gaps, gravity, wind, joint forces, and torques across stages.For example, Stage-4 uses gap_width = 26.0, gravity = (0, -28.0), and wind_force = (-45.0, -8.0).
  • Statics & Equilibrium: Pivot-balance mutations progressively alter gravity, friction, obstacles, wind strength, allowable angle deviation, and balance duration.Stage-4 requires max_angle_deviation_deg = 0.5 and balance_time = 180.0 under gravity = (0, -60.0).
  • Dynamic tasks: Dynamic tasks include a motor-driven walker, vertical wall climber, object manipulation, and ground-cart pushing scenarios.The listed mutations adjust traction, joint limits, damping, object properties, gravity, and structural constraints.
Loading 2608.14441v1…