PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Experimentalist Agents

Abhranil Chandra, Sankaran Vaidyanathan, Utsav Dhanuka, Varun Gandhi, Scott Niekum

arXiv:2606.29315v1cs.AIcs.LG

TL;DR

LLM agents struggle when novel tasks require active experimentation rather than recall or reasoning alone. HExA learns reusable skills from interaction and improves performance on the Interphyre physics benchmark, reaching up to 77% success versus 2% for the same Claude Sonnet 4.6 model.

  • Problem

    LLM agents have limited evidence for solving novel, instance-specific tasks that require active experimentation beyond recalling general knowledge.

  • Method

    HExA is a training-free in-context reinforcement-learning framework that designs experiments, distills reusable skills from interaction, and reinjects them through an external skill bank.

  • Results

    HExA improves frontier and open-weight models on Interphyre, achieving up to 77% success for Claude Sonnet 4.6 versus 2% without HExA.

  • Takeaways & Limitations

    Active experimentation with in-context skill evolution helps agents make progress on unfamiliar, long-horizon tasks and transfer skills to unseen levels.

  • Takeaways & Limitations

    The current evaluation is confined to 2D physics, so HExA remains unvalidated in other experimentation-centric domains.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to take actions in the real world and support human decision-making, yet most agents rely on parametric knowledge, fixed post-training data, retrieval, or search. This paradigm breaks down in novel domains and for sophisticated queries that cannot be answered from prior knowledge alone. Knowing the laws of physics, for instance, does not by itself enable LLMs to answer queries or complete long-horizon tasks in a complex physical system. To address this, we introduce Hierarchical Experimentalist Agents (HExA), an in-context self-improvement framework to learn from active experimentation. HExA iteratively designs and refines query-relevant experiments, learns a reusable library of composable skills from experience, and integrates experimental evidence to answer queries or take actions. HExA is training-free, compatible with any black-box model, and does not require external supervision, oracles, or offline data. To evaluate active experimentation, we introduce Interphyre, a tool-calling benchmark built on the PHYRE 2D procedural physics environment, where agents propose interventions and test hypotheses through simulation APIs. Experiments show that current LLM agents struggle in these settings, especially on the hardest levels of Interphyre. Claude Sonnet 4.6 achieves only 2% success, while HExA improves the same model to up to 77% success. HExA also improves open-weight models and outperforms agentic baselines such as ReAct and Reflexion. Moreover, using only skills learned from easier levels and transferred without active experimentation, HExA achieves 44% success, demonstrating the reusability and generalization of its learned skills. Overall, HExA shows that learning through active experimentation can help agents discover useful knowledge, acquire reusable skills, and make efficient progress on novel long-horizon tasks.

1 Introduction

The introduction argues that parametric knowledge and reasoning alone are insufficient for complex, novel tasks requiring instance-specific evidence. It proposes HExA, which uses active experimentation, hierarchical skill learning, and transfer to enable agents to make progress across new environments and harder levels.

  • Motivation: Novel environments can defeat agents because their dynamics, constraints, or solution strategies were absent from training, while applying known principles to specific instances may require experimentation beyond recall.The paper uses experimental physics as an example: knowing physical laws does not by itself solve a particular experimental problem.
  • Motivation: Active experimentation lets agents explore, gather information, propose falsifiable hypotheses, test them through interaction, and reason over what the environment confirms or refutes.The introduction frames this process as learning like scientists rather than relying solely on pretrained knowledge.
  • HExA: HExA extends in-context learning by interacting with simulators, testing hypotheses, and reflecting on successful and failed observations to distill reusable skills.The framework treats the simulator as a tool and gathers query-relevant information through deliberate experimentation rather than passive observation.
  • HExA: Hierarchical skill distillation enables later skills to build on earlier ones and supports higher-order skills for learning from interaction, including generalization to harder structurally novel levels without target-level experimentation.These skills capture both solutions for individual instances and procedures for making consistent progress across new seeds and configurations.
  • Contributions: HExA formalizes experiment-centric in-context learning for tasks requiring active experimentation to gather query-specific evidence and introduces Interphyre as a benchmark for this setting.The setting covers both complementing parametric knowledge and addressing entirely novel, unseen domains.

2 Related Works

Prior work improves LLM agents through parametric or in-context adaptation and augments language models with reusable procedural skills. However, existing skill-learning methods typically depend on external feedback signals that are unavailable in context-learning scenarios without automatic feedback.

  • Learning from Interaction: Parametric fine-tuning can improve agent decision-making but incurs high computational costs and risks catastrophic forgetting or over-specialization.These limitations motivate a shift toward in-context agents as a more efficient alternative.
  • Skills for LMs: Natural-language skills capture reusable procedural knowledge for augmenting language models at inference time, with validation across agent tasks including coding and web tasks.Related approaches include generator–verifier co-evolution, failure-driven refinement, and trajectory distillation into structured skill folders.
  • Skills for LMs: Existing skill-learning methods typically rely on external feedback signals, including execution feedback, ground-truth comparison, or task-completion rewards, to evaluate and improve skill quality.Such signals are unavailable in context-learning scenarios without automatic feedback.

3 Hierarchical Experimentalist Agents: Learning via Experimentation

HEXA is a training-free, in-context reinforcement-learning framework that improves agents through active experimentation without updating model weights. It iteratively retrieves relevant experience, runs scored episodes, and distills reusable skills and mistakes into a persistent external skill bank.

  • Framework: HEXA improves across episodes by storing interaction experience in a persistent external skill bank rather than updating model weights.The bank contains structured natural-language strategies and mistakes that are re-injected into the agent’s context, making the framework compatible with any tool-using model.
  • HEXA Loop: Each round retrieves relevant bank entries, runs x skill-conditioned episodes, scores their trajectories, and evolves the bank from the resulting batch.The first round may start empty or use a warm start; subsequent rounds repeat the retrieve–act–score–evolve cycle.
  • Trajectory Reward: Trajectory rewards in [−1, +1] combine task outcome with efficiency, favoring fast successes and penalizing failures.The reward is designed around qualitative categories that an LLM evolver can use, such as fast success, slow success, exploratory failure, and early exit.
  • Skill Distillation: Strategy skills are distilled by contrasting high- and low-reward trajectories into records containing principles, application conditions, examples, source seeds, and reward scores.These records capture insights gained from testing hypotheses through experimentation and can be composed for later episodes.
  • Mistake Distillation: Mistake records extract failures into descriptions, root causes, and corrective actions, while failed trajectories can also yield partial skills from individually correct reasoning steps.For example, the system can identify fixation on one launch point and recommend switching x zones after repeated local failures.

4 Experiments and Results on Interphyre

INTERPHYRE evaluates whether agents can discover solutions through programmatic interaction, intermediate observations, and reusable experimentation across procedurally generated 2D physics tasks. HExA improves performance and exploration efficiency by distilling experiments into evolving skill banks, with gains extending to weaker models and unseen levels.

  • Benchmark design: INTERPHYRE requires agents to discover unknown solutions by inspecting scenes, testing interventions, observing intermediate dynamics, and reusing experience across procedurally generated levels.The benchmark provides a programmatic interface for deliberate interaction and evidence gathering in a controlled 2D physics domain with continuous actions and long-horizon causal structure.
  • Overall performance: HExA improves every tested open-weight model on every level, including Qwen-2.5-3B from 8.0 →24.0% on down_to_earth and GPT-OSS-120B from 0.0 →54.0% on catapult.Other reported gains include Qwen-2.5-7B from 62.0 →72.0% on down_to_earth and 18.0 →34.0% on two_body_problem.
  • Exploration efficiency: HExA’s active skill acquisition and reuse improves exploration efficiency, while Reflexion requires more trials because it lacks a persistent reusable skill bank.The evolving skill bank supports structured exploration even when reward labels are removed.
  • Reward ablations: 64% versus 72% shows that reward-free skill banks underperform reward-labeled banks on down_to_earth for Qwen-2.5-7B, with 26% versus 34% on two_body_problem.Without reward labels, success and failure trajectories are treated equally and the LLM evolver updates the bank using its existing knowledge.
  • Skill transfer: 44.0% versus 2.0% with the base model and 8.0% with REACT shows that skills transferred from easier levels improve Claude Sonnet on catapult without active experimentation.Transfer from down_to_earth to falling_into_place also yields a +12% success-rate gain for Qwen 7B.

5 Conclusion, Limitations, & Future Works … A.3.1 Trajectory Reward

HExA is a training-free, in-context reinforcement-learning framework that improves LLM agents through active experimentation, hierarchical skill evolution, and cross-task reuse without parameter updates or external supervision. The appendix details its problem formulation, actor–evolver architecture, reward design, and implementation.

  • 5 Conclusion, Limitations, & Future Works: The overall contribution combines training-free experimentation, hierarchical skill distillation, query-relevant trajectory gathering, and skill reuse without parameter updates, offline data, or external supervision.The framework is paired with Interphyre, a physical reasoning benchmark for learning through tool-call experimentation.
  • 5 Conclusion, Limitations, & Future Works: HExA’s evolved skill banks transfer zero-shot to unseen levels, improve frontier and open-weight models, and achieve strong gains against ReAct, Reflexion, and gradient-based GRPO under matched interaction budgets.The transferred skills can support progress without target-level interaction, and discovered strategies become usable by the next episode through context.
  • 5 Conclusion, Limitations, & Future Works: HExA is designed as a general experiment-centric algorithm operating on interaction trajectories, scalar rewards, and natural-language skills rather than environment-specific assumptions.Its evaluation is currently confined to 2D physics through Interphyre.
  • A.2 Problem Formulation: The framework targets solve-rate maximization without parameter updates by retrieving high-reward skills and common-mistake records from a naturally evolved language-based skill bank.Each trajectory receives a scalar reward in [−1, +1] that captures outcome and efficiency.
  • A.3 The HExA Framework: HExA uses an actor–evolver loop in which an LLM generates tool-interaction trajectories and an evolver distills them into reusable skills.Retrieved skills are injected into the actor’s context at the start of each episode.
  • A.3 The HExA Framework: HExA improves policies through context augmentation rather than weight updates, while its skill bank compresses and curates cross-episode experience.The framework operates on a single frozen model and does not require pretraining on a task distribution.
  • A.3.1 Trajectory Reward: Trajectory rewards combine success and efficiency, assigning higher values to faster solutions so the evolver can prioritize more informative behavior.The specified success rewards are +1.00 for t ≤3, +0.75 for t ≤7, +0.50 for t ≤15, and +0.25 for success by t ≤T.
  • A.3.1 Trajectory Reward: Failure penalties are asymmetric because extensive exploration before failure provides richer interaction records for extracting partial skills and diagnosing mistakes.Not all trajectories are equally informative: a fast solve and an extensively explored failure provide different evolutionary signals.

A.3.2 Skill Distillation from Experimentation on Self-Proposed Hypotheses … A.3.5 Initialisation, Update Regimes, and Full Algorithm

HExA distills successful and failed experimental trajectories into reusable skills, mistakes, and partial skills, stores them in a bounded task-indexed bank, and retrieves selected knowledge for the actor. Its configurations vary by bank initialization and update strategy, with Off2On + Evolution performing best across settings.

  • A.3.2 Skill Distillation from Experimentation on Self-Proposed Hypotheses: At each round, the evolver separates actor trajectories into successful and failed subsets and performs two-phase knowledge distillation.The trajectory subsets are annotated with scalar rewards.
  • A.3.2 Skill Distillation from Experimentation on Self-Proposed Hypotheses: Contrastive extraction compares high- and low-reward trajectories to produce 4–6 strategy skills with principles and applicability predicates.Each skill includes a short title, a 2–3 sentence mechanism principle, and a condition describing when it applies.
  • A.3.2 Skill Distillation from Experimentation on Self-Proposed Hypotheses: Failure-focused extraction records mistakes with errors, root causes, and corrective strategies while recovering partial skills from correct reasoning within failed trajectories.These partial successes provide learning signals unavailable from successful trajectories alone.
  • A.3.3 Skill Bank: The skill bank stores reusable skills and recurring mistakes by task family, labels skills by source-trajectory reward, and caps each family at Mmax skills and Nmax mistakes.Skills from efficient solves receive higher rewards, while partial skills from failed trajectories receive lower rewards but can preserve valuable discoveries.
  • A.3.4 Skill Retrieval and Injection: At episode start, the retriever selects the top-M skills by reward and top-N task-family mistakes, injecting them into the actor’s system prompt as structured context G.The actor receives this context alongside the task description and tool documentation and may use or disregard individual entries.
  • A.3.5 Initialisation, Update Regimes, and Full Algorithm: HExA varies along initialization and update axes: Offline and Off2On warm-start from N0 trajectories, whereas Pure Online begins with K0 = ∅.Evolution updates the existing bank with new trajectories, while Iterative Replacement distills only the current round and discards prior skills.
  • A.3.5 Initialisation, Update Regimes, and Full Algorithm: Crossing the two axes yields five variants, and Off2On + Evolution works best across all settings and is the paper’s default HExA variant.The five variants include Offline, two Off2On regimes, and two Pure Online regimes.

A.4 Cross-Task Skill Transfer

HExA transfers reusable physical and procedural principles across task families by synthesizing a target-task skill bank from source banks without target-task trajectories. The transferred bank supports zero-shot execution and can initialize subsequent within-task refinement.

  • Cross-Task Skill Transfer: HExA synthesizes a target-task skill bank from one or more source banks by exploiting shared physical or procedural primitives.Recurring principles include momentum conservation, lever mechanics, and collision geometry; no target-task trajectories are required.
  • Cross-Task Skill Transfer: The evolver selects structurally relevant source skills, re-grounds their principles in the target scene, and rewards transferability, especially when multiple source banks corroborate a skill.The process uses the target family’s textual description and calibrates reward according to how directly each principle transfers.
  • Cross-Task Skill Transfer: The synthesized bank is injected through the RETRIEVER mechanism, enabling zero-shot target-task attempts without target-task trajectories.The actor benefits from cross-task skills despite having seen no trajectories from the target family.
  • Cross-Task Skill Transfer: Transferred skills and within-task evolution are complementary: the synthesized bank can initialize a later HExA run as K0 for task-specific refinement.This combines transferred knowledge with subsequent adaptation to the target task.

A.5 Design Principles … B.2.2 Two Body Problem — compute_relative_positions()

HExA is designed as a domain-agnostic, reward-function-agnostic, in-context framework whose hierarchical skill bank drives adaptation without weight updates. The appendix specifies shared and level-specific tools for querying scenes, simulating interventions, diagnosing outcomes, and computing geometry relevant to puzzle solutions.

  • A.5 Design Principles: HExA’s generic tool interface and text-to-JSON evolver keep domain-specific tools and task prompts outside the framework architecture.The framework is instantiated on physics puzzles but is not architecturally tied to that domain.
  • A.5 Design Principles: HExA can guide skill creation without environment rewards and can incorporate any available reward function or additional feedback.This reward-agnostic design broadens applicability to domains lacking explicit environment rewards.
  • A.5 Design Principles: HExA adapts entirely in context through its evolving skill bank, without parameter updates to either the actor or evolver.This supports closed-source API models while avoiding fine-tuning cost and catastrophic-forgetting risks.
  • A.5 Design Principles: Hierarchical skills abstract low-level tool calls into high-level strategies and are iteratively revised, merged, and built upon using previously distilled skills.This structure compresses lengthy trajectories into reusable principles while allowing later evidence to refine earlier knowledge.
  • B Tool Descriptions: The appendix partitions agent tools into shared tools available on every level and level-specific tools exposed only for corresponding puzzles.This partition defines the tool interface used by the REACT agent.
  • B.1 Shared Tools: Shared tools expose structured scene state, validate and run full or partial physics simulations, return contact logs, and submit final placements.Scene queries include object geometry, world bounds x, y ∈[−5, 5], and level-specific success conditions; full simulations run up to 2,000 steps at 60 Hz.
  • B.2.1 Down to Earth — compute_gap_analysis(): compute_gap_analysis() reports platform edges, ball diameter, both gap widths, fit feasibility, and a larger-viable-gap recommendation or warning.It analyzes whether the green ball can pass around the platform separating it from the floor.
  • B.2.2 Two Body Problem — compute_relative_positions(): compute_relative_positions() returns ball geometry, separations, contact distance, and whether the blue ball lies LEFT or RIGHT of the green ball.These measurements guide red-ball placement to push the green ball into contact with the blue ball.

B.2.3 Catapult — Four Analysis Tools … B.2.8 Tipping Point — compute_tipping_point_analysis()

The paper presents query-specific analysis tools that expose scene geometry, predict contacts, trace trajectories, and compute mechanisms across six physics puzzles. These tools provide strategy-neutral measurements and targeted guidance for simulation-based reasoning.

  • B.2.3 Catapult — Four Analysis Tools: Catapult exposes four complementary tools for surveying geometry, tracing object kinematics, probing green-ball trajectories, and predicting first contact.The tools support strategy-neutral scene inspection, full or truncated simulations, lightweight trajectory sampling, and pre-simulation contact checks.
  • B.2.3 Catapult — Four Analysis Tools: simulate_with_trace returns extrema, angular motion, and up to 15 relevant contact events to verify lever mechanics and green-ball success.Per-object summaries include peak_y, min_y, v_max, ∆pos, peak angular speed ωmax, and angle range [θmin, θmax].
  • B.2.4 Falling Into Place — compute_intercept_setup(): The intercept tool computes ball and jar positions, required platform-edge crossing, gap-centre x-coordinate, and estimated fall time in seconds and 60 Hz steps.It also identifies the platform supporting the green ball and the direction required for its travel.
  • B.2.5 Basket Case — compute_basket_analysis(): The basket analysis reports object geometry, green-ball-to-basket distance, and a recommended LEFT/RIGHT push direction to deflect the ball away.Reported geometry includes the basket opening half-width and purple-ground position when present.
  • B.2.6 Pass the Parcel — get_ramp_center(): The ramp-centre tool infers ramp, basket, platform, angle, and bound geometry to guide red-ball placement and downstream interactions.It returns the ramp centre coordinates (x, y) and positions of the top and bottom baskets.
  • B.2.7 Cliffhanger — compute_cliffhanger_analysis(): The cliffhanger analysis identifies bar and platform geometry, the closer edge the bar must fall past, and the falling-gap width on that side.It also reports the bar’s endpoints, environmental heights, and distances from its bottom point to both platform edges.
  • B.2.8 Tipping Point — compute_tipping_point_analysis(): The tipping-point analysis computes bar, basket, and wall geometry, the wall’s side and distance, approximate tip angle, and suggested LEFT/RIGHT direction.The angle is treated as rigid-stick rotation about the base, using arcsin(∆x/L) when ∆x < L and N/A otherwise.

C Baseline Implementation Details … C.4 GRPO: Weight-Update Baseline

The appendix specifies reproducible implementations for REACT, Reflexion, DIRECT, and GRPO baselines on the INTERPHYRE tool API. These methods differ in interaction, reflection, static-scene reasoning, or weight updates, while sharing defined evaluation and compute protocols.

  • C Baseline Implementation Details: The baselines use open-weight models with vLLM and Huggingface on up to 4 A100 GPUs, while Claude Sonnet uses the Anthropic API.Together with the system prompts and tool descriptions, these details are sufficient to reproduce every baseline number reported in Section 4.
  • C.2 Reflexion: K=2 Trial Wrapper: Reflexion reflections identify the attempted strategy, diagnose concrete kinematic, geometric, or procedural failure causes, and propose one specific alternative action or strategy.The reflector receives the full trajectory and final observation, with long observations truncated to their first and last 300 characters.
  • C.3 Direct: One-Shot, No-Tool Baseline: DIRECT removes interaction and reasoning loops, making one scene inspection and one committed placement without read tools, probes, or retries.It uses a single Claude Sonnet 4.6 call, parses one final (x, y, r) placement, and scores success with the same simulator predicate as other methods.
  • C.4 GRPO: Weight-Update Baseline: GRPO fine-tunes Qwen-2.5-3B-Instruct on down_to_earth and two_body_problem, serving as the only evaluated baseline that updates model weights.It uses VERL-TOOL and is included purely as a sample-efficiency comparison for Section D.
  • C.4 GRPO: Weight-Update Baseline: GRPO trains for 10 epochs, comprising 500 gradient steps over 50 training seeds, with four rollouts per step and binary environment rewards.The non-skilled and static-skilled variants differ only in max_prompt_length, which is increased from 2048 to 3072 for the skill-bank prefix.
  • C.4 GRPO: Weight-Update Baseline: GRPO evaluates non-skilled prompts against static-skilled prompts that prepend the frozen skill bank produced by HExA after its final training pass.Results are reported as greedy validation success rate (mean@1, 50 validation seeds) at each epoch boundary, while compute comparisons use unique seeds seen.

C.5 Seed Ranges per (method, level) Cell

Table 8 specifies reproducible seed ranges for every method–level cell, with all cells using 50 seeds and within-level comparisons sharing identical evaluation instances. HExA separates warm-start, online-round, and cross-level transfer seeds to preserve paired and zero-shot comparisons.

  • Seed allocation: All method–level cells use exactly 50 integer seeds, passed to the simulator’s level generator to randomize object positions, masses, and dimensions.The seed ranges are listed in Table 8 for full reproducibility.
  • Within-level comparisons: Within each level, REACT, Reflexion, and HExA are evaluated on the same 50 seeds, making reported comparisons paired on identical instances.HExA additionally reserves seeds 1–5 for its offline warm-start bank K0; these seeds are excluded from evaluation.
  • Online-round protocol: In HExA’s online rounds, each attempted batch uses fresh seeds, with k=3 or k=5 per round, before the evolver updates the bank from those trajectories.Thus, each attempt uses a bank not yet distilled on the seed being attempted, though earlier same-level seeds trained it.
  • Cross-level transfer: For cross-level transfer, the synthesized target bank and target-level REACT baseline use the same 50 seeds, while synthesis sees no target-level trajectory.The resulting Table 13 comparison is paired on identical instances and strictly zero-shot at the trajectory level.

D Additional Details of Experiments and Results … E.2 REACT system prompt (catapult seed 45)

Additional experiments show that HExA improves open-weight agents, can outperform direct RL when interaction budgets are limited, and transfers across levels. A catapult case study illustrates the contrast: REACT fails after 25 iterations, whereas HExA succeeds in 6 iterations using an evolved skill bank.

  • D Additional Details of Experiments and Results: HExA improves every tested open-weight model-level pair, lifting weaker agents through a general in-context skill-evolution mechanism.The reported gains are not limited to frontier-model agents or Claude Sonnet’s prior knowledge.
  • D Additional Details of Experiments and Results: 24.0% and 14.0% are Qwen-2.5-3B’s HExA scores on down_to_earth and two_body_problem, up from 8.0% and 6.0%.For Qwen-2.5-7B, HExA raises down_to_earth from 62.0% to 72.0% and two_body_problem from 18.0% to 34.0%.
  • D Additional Details of Experiments and Results: 54.0% is GPT-OSS-120B’s HExA score on catapult, compared with 0.0% under the REACT baseline.This is the largest open-weight gain reported in the passage.
  • D Additional Details of Experiments and Results: With sufficient training, GRPO eventually dominates HExA on exact environment reward: down_to_earth reaches 100% by epoch 5 for GRPO and epoch 6 for GRPO w. skills.On two_body_problem, GRPO peaks at 56% and plateaus around 40–50%, while GRPO w. skills converges to 100% by epoch 4.
  • D Additional Details of Experiments and Results: When the sample budget is small, HExA’s in-context skill evolution is preferred; static skills also dip to 6% on DTE and 0% on TBP at epoch 1 before recovering.The static-skilled prompt is approximately 2200 tokens versus approximately 1400 tokens without injected skills, requiring re-adaptation.
  • E Working Example: ReAct vs. HEXA on Level Catapult of InterPhyre: On catapult seed 45, REACT fails after 25 iterations, while HExA succeeds in 6 iterations at final placement (x=0.3, y=0.9, r=1.5).Both agents use Claude Sonnet 4.6 as the actor; the example reproduces their prompts, transcripts, and HExA’s round-14 evolving skill bank.
  • E.2 REACT system prompt (catapult seed 45): The REACT prompt requires stepwise Thought/Action/Observation cycles, one tool per step, and an initial predict_first_contact call before tracing the green ball.It defines success as green_ball contacting blue_ball and permits up to 25 iterations, with simulations resetting the environment.
  • E.1 Sparsified simulation frames (catapult seed 45): In the accepted HExA simulation, the heavy red ball lands on the catapult arm, flings the green ball over the ceiling-blocker, and the green ball settles against the blue ball.REACT’s 25-iteration run never produces a placement bringing the green ball into contact with the blue ball.

E.3 REACT trajectory (catapult seed 45, 25 iterations, FAILURE)

The REACT agent iteratively adjusted red-ball drop parameters to launch the green ball toward the blue ball, but every tested simulation failed. Its attempts included correcting an initial miss, increasing impact strength, and refining placement near the catapult tip, without achieving the target trajectory.

  • Initial targeting: The initial drop at (1.5, 2.0) first contacted the ledge rather than the gray_platform catapult arm.The placement was valid, but the predicted first contact occurred at step 62 with the ledge.
  • Initial targeting: A corrected drop at (1.0, 1.0) contacted the gray_platform at step 28, but the simulation still reported Success: False.The predicted approach speed was 2.006, and the green ball reached only y=0.67 before falling back.
  • Impact-strength refinement: Increasing the red ball to radius 0.8 and dropping it from y=3.0 raised contact speed to 5.428, yet the green ball peaked at y=2.34 and moved right by 2.41 units without reaching the blue_ball.This second full simulation also reported Success: False.

E.4 Round-14 evolving skill bank used by HEXA · E.5 HEXA system prompt (catapult seed 45)

The Round-14 skill bank distills catapult-placement heuristics into a primary launch configuration, geometry-aware fallbacks, contact validation, and rules against ineffective micro-tuning. The E.5 prompt operationalizes these skills through simulator tools, mandatory thought/action cycles, and a seed-45 catapult strategy.

  • E.4 Round-14 evolving skill bank used by HEXA: r=1.5 is the minimum reliable radius, while r=2.0 adds negligible range at x=0.5 because arm rotation saturates.When r=1.5 fails at x=0.5, the bank recommends changing x rather than increasing radius.
  • E.4 Round-14 evolving skill bank used by HEXA: The canonical first attempt is x=0.5, y=0.4, r=1.5, producing a consistent rightward launch arc from a stable lever-arm position.The skill bank specifies deviation to x=0.3 only after ceiling hits or overlap constraints.
  • E.4 Round-14 evolving skill bank used by HEXA: The bank requires predict_first_contact before every full simulation to verify placement, confirm gray_platform contact, and catch gray_ball pivot overlaps.This cheap pre-simulation check prevents invalid or mechanically ineffective placements from consuming a full simulation budget.
  • E.4 Round-14 evolving skill bank used by HEXA: The learned rules identify x~=0.0 to 0.5 as the stable launch zone, while x<=-0.3 and x=0.7--1.5 are bifurcation zones with unpredictable trajectories.Small x shifts can substantially change landing position, so failed attempts should reset toward x=0.2--0.4 rather than continue chaotic fine-tuning.
  • E.4 Round-14 evolving skill bank used by HEXA: After a ceiling hit, the three-tier fallback first tries x=0.3, y=0.4, r=1.5, then changes height or x, and finally shifts x after repeated radius failures.The x-shift flattens the launch arc, while y=0.4 avoids the gray-ball overlap zone; persistent radius tuning should stop after 2+ variations.
  • E.5 HEXA system prompt (catapult seed 45): The E.5 prompt defines success as green-blue contact lasting at least 3 seconds and provides geometry inspection, simulation, contact-log, trajectory, partial-simulation, contact-prediction, and finish tools.It also constrains the red ball to remain inside the box, avoid initial overlaps, and use 0.1 <= radius <= 2.0.
  • E.5 HEXA system prompt (catapult seed 45): The prompt enforces one-tool-per-step Thought/Action cycles, careful observation parsing, and independent simulations for comparing candidate placements.Each simulation resets the environment, and the agent submits the final candidate through finish.

E.6 HEXA trajectory (catapult seed 45, 6 iterations, SUCCESS) · F More Background and Preliminaries · G Interphyre: Environment and Benchmark Details

The HExA trajectory solves the catapult task by using scene evidence and reusable skills to revise an initially failed intervention. The surrounding sections define tool-augmented interaction, MDP and in-context reinforcement-learning foundations, and Interphyre’s benchmark scope.

  • E.6 HEXA trajectory (catapult seed 45, 6 iterations, SUCCESS): The agent observes the scene geometry and identifies the catapult arm at y=-0.87, motivating a higher drop height.The world bounds are x in [-5, 5] and y in [-5, 5].
  • E.6 HEXA trajectory (catapult seed 45, 6 iterations, SUCCESS): It first selects the reusable placement x=0.5, y=0.9, r=1.5 and verifies valid contact with the gray_platform.The prediction reports placement_valid: True and contact_step: 10.
  • E.6 HEXA trajectory (catapult seed 45, 6 iterations, SUCCESS): The initial full simulation fails, with the green ball hitting the top_wall at step 67 and reaching y_peak=4.78.The agent interprets the ceiling hit as evidence that the trajectory must be flattened.
  • E.6 HEXA trajectory (catapult seed 45, 6 iterations, SUCCESS): Shifting x from 0.5 to 0.3 while retaining y=0.9 and r=1.5 yields success after 343 simulation steps.The final observation reports that the green ball contacted the blue ball in the basket, with Reward: 1.0.
  • F More Background and Preliminaries: Tool-augmented LLM agents interleave thoughts, actions, and observations to gather information, execute APIs, and manipulate simulators.This augmentation supports interaction with complex environments through external tools and programmatic execution.
  • F More Background and Preliminaries: The paper models agent-environment interaction as an MDP M = ⟨S, A, p, r, γ, p0⟩ with states, actions, dynamics, rewards, discounting, and initial-state distribution.At each time step, the agent observes a state, selects an action according to policy π, and receives a reward.
  • F More Background and Preliminaries: In-context reinforcement learning conditions a frozen policy on the current state and evolving context, enabling adaptation to test environments differing from pre-training environments.The action is sampled according to πθ(· | St, Ct), while the pre-trained weights θ remain frozen.
  • G Interphyre: Environment and Benchmark Details: Interphyre’s appendix specifies the environment’s design, intervention API, and benchmark limitations.The supplied passage introduces the scope of the environment and benchmark-details section without further technical detail.

G.1 Introduction … G.4.2 Event-Driven Branch Points

Interphyre is an intervention-aware 2D physics environment designed for paired counterfactual evaluation, mechanistic probing, and controlled physics generalization. Its Python-defined curriculum, shared simulator interfaces, certified solvability, snapshot/restore branching, and event-driven triggers support reproducible causal experimentation.

  • G.1 Introduction; G.2 Related Work: Interphyre addresses physics benchmarks’ lack of mid-trajectory intervention, paired counterfactual rollouts, and activation-level causal ground truth.Continuous, contact-dependent physical state requires rewinding a running simulation rather than symbolically resetting a board.
  • G.1 Introduction; G.3 Design; G.3.2 Physics Parameters as First-Class Inputs: Interphyre separates physics parameters from scene topology, allowing fixed-level and fixed-seed generalization studies that independently sweep gravity or friction.Unlike Kinetix, it does not vary parameter regimes jointly with level structure.
  • G.1 Introduction; G.4 Intervention API; G.4.1 Snapshot/Restore and Counterfactual Branching: Snapshot/restore captures complete physics state and creates independent branches with controlled perturbations sharing a bit-identical physical prefix.Branches can vary impulses, forces, parameters, or objects, enabling matched factual and counterfactual outcomes.
  • G.1 Introduction; G.3 Design; G.3.1 Levels as Python Code: Levels are extensible Python classes with composable objects, action slots, tunable physics, and user-defined success conditions that researchers can fork and register.This makes geometry, actions, and success logic ordinary code rather than fixed benchmark data.
  • G.2 Related Work: Interphyre extends prior physical reasoning and causal-evaluation environments with white-box access, parametric control, intervention infrastructure, and instrumentation for representations.Existing benchmarks generally evaluate puzzle-solving behavior or lack paired counterfactual and activation-level access.
  • G.3 Design; G.3.4 Agent Interfaces and Observation Modes: Gymnasium-compatible RL and named tool-call interfaces use one simulator backend, while physics-state, image, and combined observations make modality a controlled variable.The interfaces expose the same physics to standard RL code and LLM agents.
  • G.3 Design; G.3.3 Curriculum and Oracle Verification: The curriculum contains 25 canonical levels and 250,000 pre-certified task instances, with every seed verified by exhaustive oracle search over a discretized action grid.Oracle verification ensures evaluation failures are attributable to the agent up to action-grid resolution.
  • G.4.2 Event-Driven Branch Points: Event-driven triggers select branch points by physical conditions rather than fragile step indices, and compositional triggers support ordered or alternative event logic.The engine fires automatically when a specified event occurs, decoupling experimental design from seed-specific timing.

G.4.3 Worked Example: Strategy-Dependent Causal Structure

The worked example shows that causal structure depends on the strategy used: interventions necessary for one trajectory can be irrelevant or disruptive for another. Strategy-conditioned probing is therefore essential, because pooling across strategies can obscure causal effects.

  • Strategy-dependent causal structure: Different seeds succeed through different strategies: seed 8 uses a deflector bounce, whereas seed 5 reaches the basket directly.Both seeds succeed in the factual branch.
  • Strategy-dependent causal structure: Removing the black deflector causes seed 8 to fail but leaves seed 5 unaffected, identifying strategy-specific causal necessity.The deflector is necessary for the bounce strategy but not for the direct path.
  • Strategy-dependent causal structure: Shrinking the red action ball to radius r=0.4 makes seed 5 fail while seed 8 retains a successful deflector arc.The intervention changes seed 5’s launch dynamics without disrupting seed 8’s trajectory.
  • Strategy-dependent causal structure: The same remove_object intervention is causally necessary for one strategy and irrelevant for another, so unconditional probing can recover neither effect when strategies are balanced.Pooling opposite effects across strategies averages away causal relevance.
  • Protocol timing: The protocol snapshots when the red action ball contacts the catapult arm and branches 30 steps later after the trajectories diverge.Three branches then apply different structural modifications.

G.4.4 Perturbation Primitives as Experimental Treatments … I Evolver Prompts for HEXA and Cross-Level Transfer

The appendix defines perturbation-based experimental treatments, analyzes strategy-conditioned probing and curriculum transfer, and documents ReAct prompts and physics-simulation tools across multiple InterPhyre levels. Together, these sections specify how agents intervene, test hypotheses, and reason through tool-mediated puzzle solving while noting limitations of the current evidence and curriculum.

  • G.4.4 Perturbation Primitives as Experimental Treatments: Kinematic perturbations modify existing objects through impulses, continuous forces, velocity or position overrides, and freezing, enabling tests of dependence on specific state variables.These treatments support counterfactual questions such as whether a 5% post-contact velocity change flips an outcome.
  • G.5 Conclusion: Strategy-conditioned linear probes on Qwen3-8B residual activations predict counterfactual outcomes above chance across three levels, whereas pooling strategies averages over opposite causal effects.The result indicates that indexing representations to each seed’s oracle strategy is necessary to recover probing signal.
  • G.5 Conclusion: Fixing scene topology while varying physics parameters isolates physics transfer from structural variation, while the hand-authored curriculum supports known-solution tasks and hypothesis-driven behavioral tests.This independent parameter axis addresses a confound that Kinetix cannot separate and enables questions about transfer across physics regimes.
  • G.5 Conclusion: The current evidence is limited by probing one model family on three levels and by a 25-level hand-authored curriculum that requires additional authoring effort to scale.The oracle verification system applies to new levels without runtime modification, but broader validity across model scale, architecture, and curriculum remains open.
  • H ReAct System Prompts: ReAct receives factual scene descriptions, success conditions, placement constraints, a complete tool list, and format instructions without privileged strategy, directional, geometry, radius, or physics guidance.The prompt requires thought-action cycles, one tool call per step, careful observation parsing, and simulation-based comparison before finishing.
  • H.1 Down to Earth; H.2 Two Body Problem: The Down to Earth and Two Body Problem prompts provide state inspection, full and partial simulation, contact logging, and level-specific geometric analysis before final submission.Down to Earth adds gap analysis, while Two Body Problem adds relative-position analysis and recommended placement direction.
  • H.3 Pass the Parcel; H.5 Falling Into Place: Pass the Parcel and Falling Into Place prompts target sustained object contacts and expose tools for simulation, contact inspection, partial dynamics, ramp geometry, and intercept setup.Their goals require the green ball to contact a target for at least 3 seconds, with Falling Into Place additionally estimating platform-crossing and jar-intercept timing.
  • H.4 Catapult; H.6 Basket Case: Catapult and Basket Case prompts support strategy-neutral geometry, trajectory and first-contact checks, traced simulations, and intervention planning for deflecting or contacting target objects.Catapult exposes green-ball trajectory and object-level kinematic traces, while Basket Case frames sideways deflection as necessary to prevent the green ball from entering the basket.
Loading 2606.29315v1…