PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Experimentalist Agents

Abhranil Chandra, Sankaran Vaidyanathan, Utsav Dhanuka, Varun Gandhi, Scott Niekum

arXiv:2606.29315v1cs.AIcs.LG

TL;DR

LLM agents struggle when tasks require instance-specific evidence and active experimentation beyond prior knowledge. HExA addresses this by learning reusable skills from interaction and reusing them across tasks, substantially improving performance on physical reasoning benchmarks.

  • Problem

    LLM agents lack reliable ways to solve unfamiliar tasks requiring instance-specific evidence and experimentation beyond recalled knowledge.

    Source: §1 · p. 2

  • Method

    HExA is a training-free in-context reinforcement-learning framework that distills interaction experience into reusable skills and reinjects them across episodes.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    HExA improves every tested open-weight model on every Interphyre level, outperforming ReAct, Reflexion, and gradient-based GRPO under matched interaction budgets.

    Source: §4.4 · p. 10, §5 · pp. 13–14

  • Takeaways & Limitations

    Reusable skills transfer zero-shot to unseen levels, helping agents make progress without target-level interaction across frontier and open-weight models.

    Source: §5 · p. 14

    Evaluation is currently confined to 2D physics, and HExA depends on an LLM-based evolver whose skill quality may be bounded by the evolver’s reasoning.

    Source: §5 · p. 14

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to take actions in the real world and support human decision-making, yet most agents rely on parametric knowledge, fixed post-training data, retrieval, or search. This paradigm breaks down in novel domains and for sophisticated queries that cannot be answered from prior knowledge alone. Knowing the laws of physics, for instance, does not by itself enable LLMs to answer queries or complete long-horizon tasks in a complex physical system. To address this, we introduce Hierarchical Experimentalist Agents (HExA), an in-context self-improvement framework to learn from active experimentation. HExA iteratively designs and refines query-relevant experiments, learns a reusable library of composable skills from experience, and integrates experimental evidence to answer queries or take actions. HExA is training-free, compatible with any black-box model, and does not require external supervision, oracles, or offline data. To evaluate active experimentation, we introduce Interphyre, a tool-calling benchmark built on the PHYRE 2D procedural physics environment, where agents propose interventions and test hypotheses through simulation APIs. Experiments show that current LLM agents struggle in these settings, especially on the hardest levels of Interphyre. Claude Sonnet 4.6 achieves only 2% success, while HExA improves the same model to up to 77% success. HExA also improves open-weight models and outperforms agentic baselines such as ReAct and Reflexion. Moreover, using only skills learned from easier levels and transferred without active experimentation, HExA achieves 44% success, demonstrating the reusability and generalization of its learned skills. Overall, HExA shows that learning through active experimentation can help agents discover useful knowledge, acquire reusable skills, and make efficient progress on novel long-horizon tasks.

1 Introduction

HExA addresses the limits of parametric knowledge by enabling LLM agents to experiment, distill reusable hierarchical skills, and transfer them across novel tasks. The framework is formalized and evaluated with Interphyre, where it substantially improves performance without parameter updates or external supervision.

Motivation

  • Parametric knowledge and retrieval break down in novel environments and when applying general knowledge requires instance-specific experimentation.Knowing physical laws, for example, does not by itself solve an experimental physics problem.Source: §1 · pp. 1–2
  • Active experimentation should be combined with learning from experience because repeatedly re-exploring each related instance wastes the structure shared across a task family.The proposed approach treats agents as scientists that explore, propose falsifiable hypotheses, test them, and reason over environmental feedback.Source: §1 · p. 2

HExA

  • HExA actively tests hypotheses through simulator interaction, distills experimental trajectories into reusable skills, and injects those skills into subsequent episodes.Its hierarchical skills abstract over low-level actions, while later skills build on previously learned ones to amortize exploration across related instances.Source: §1 · pp. 2–3

Interphyre

  • Interphyre benchmarks experiment-centric reasoning through programmatic physics interventions, including scene inspection, partial simulation, contact logging, and counterfactual snapshot/restore.The benchmark evaluates whether agents can gather query-relevant evidence and consolidate skills for novel tasks.Source: §1 · p. 3

Contributions

  • +75% accuracy on the hardest Interphyre level is achieved by HExA over Claude Sonnet 4.6, which is stuck at 2%.HExA also improves smaller and frontier black-box models, while transferred skills alone yield up to +36% gains without target-level experimentation.Source: §1 · p. 3

2 Related Works

Prior work improves LLM agents through parametric or in-context adaptation and increasingly reusable skills, but often depends on costly updates, external feedback, or parameter access. HEXA addresses these limitations through training-free, in-context experimentation that self-evolves hierarchical skills.

Learning from Interaction

  • Parametric fine-tuning can improve decisions but incurs computational costs and risks catastrophic forgetting or over-specialization, motivating more efficient in-context agents.HEXA targets this challenge through in-context experimentation and self-evolution of reusable hierarchical skills.Source: §2 · p. 3

Skills for LMs

  • Natural-language skill modules support reusable procedural knowledge across agent tasks, while recent systems automate skill construction from interaction traces, trajectories, or execution feedback.Examples include AutoSkill, AutoRefine, CoEvoSkills, EvoSkill, and SkillX, which construct or refine reusable skill artifacts and packages.Source: §2 · p. 3
  • Existing skill methods typically require external feedback, ground-truth comparison, rewards, or parameter access, limiting use with closed-source models and reducing interpretability.HEXA instead discovers and evolves skills through in-context active experimentation without requiring such supervision.Source: §2 · p. 4

3 Hierarchical Experimentalist Agents: Learning via Experimentation

HEXA is a training-free, in-context framework that improves tool-augmented LLM agents through active experimentation rather than weight updates or external supervision. It distills experimental experience into a persistent, retrievable skill bank containing reusable strategies and mistakes, supporting within-task learning and cross-task transfer.

Framework

  • HEXA improves across episodes by reinjecting an externally stored skill bank into any tool-augmented LLM, without offline data, oracle supervision, external teachers, or model-weight updates.The framework learns from environment interaction feedback and can begin each task family without domain-specific reward-labeled training.Source: §3 · pp. 4–5

HEXA Loop

  • Each HEXA round retrieves relevant skills, runs actor episodes, scores their trajectories, and uses an evolver to update the skill bank from the resulting experience.The actor tests hypotheses through interventions and tool interactions, while the retriever injects top-ranked skills and mistakes into subsequent episode contexts.Source: Figure 2 · p. 4, §3 · pp. 5, 7

Trajectory Reward

  • Trajectory rewards combine outcome and efficiency, favoring fast successes while treating exploratory failures as more informative than early abandonment.HEXA uses qualitative reward bins because the evolver can reliably distinguish categories such as fast success, slow success, exploratory failure, and early exit.Source: §3 · pp. 5–6

Skill Distillation

  • The evolver extracts strategy skills by contrasting high- and low-reward trajectories, mistake records from failures, and partial skills from correct intermediate reasoning in unsuccessful episodes.Skills encode principles, applicability conditions, examples, provenance, and reward scores, while mistake records pair failure descriptions with root causes and corrective actions.Source: §3 · p. 6

Cross-task Skill Transfer

  • HEXA transfers mechanism-level abstractions across task families by removing source-specific bindings and re-grounding principles in an unseen target’s entities, constraints, and tools.The transfer procedure forbids invented target-specific coordinates and can provide zero-shot initialization or seed a subsequent within-task HEXA run.Source: §3 · p. 7

4 Experiments and Results on Interphyre

INTERPHYRE evaluates whether agents can learn through interactive experimentation in a controlled, procedural 2D physics environment. HExA substantially improves task success, efficiency, skill-bank quality, transfer, and low-data adaptation over non-experimental and RL baselines.

4.1 INTERPHYRE: A Testbed for Learning through Experimentation

  • INTERPHYRE requires agents to inspect scenes, test interventions, observe intermediate dynamics, and reuse evidence across long-horizon tasks with controllable procedural difficulty.Its structured API supports scene inspection, partial and full simulation, contact logs, geometric analysis, and intervention traces in a controlled 2D physics domain.Source: §4 · p. 7, §4.1 · pp. 7–8

4.2 Experimental Setup, Evaluations, and Baselines

  • The evaluation uses solve rate and average turns per seed across eight levels, with pass_the_parcel and catapult forming Tier 2 because parametric reasoning is insufficient.All methods are compared on identical seeds, and unsolved seeds are capped at 25 turns.Source: Table 1 · p. 8, §4.2 · p. 9

4.3 Instantiation of HEXA – Skill Initialization, Update, and Reward Labels

  • Off2On + Evolution with reward-labeled skills is HExA’s default and best-performing configuration, combining warm-started skill banks, evidence-based evolution, and environment-grounded reward labels.Reward-free ablations perform worse, and Table 4 identifies Off2On Evolving as best across the two hard levels.Source: §4.3 · p. 9, Table 4 · p. 12

4.4 Results, Analysis, and Ablations

  • 67.3 ± 9.3% on catapult versus 2.0% DIRECT, 8.0% REACT, and 21.3 ± 2.5% Reflexion; HExA also improves every tested open-weight model on every level.On catapult, GPT-OSS-120B rises from 0.0 →54.0%, while Qwen-2.5-3B and 7B improve on multiple levels.Source: §4.4 · pp. 9–10, Figure 5 · p. 10, Table 2 · p. 11
  • 14.4 ± 1.8 average iterations versus 22.9 for REACT and 21.2 ± 0.7 for Reflexion, showing HExA achieves roughly 37% fewer turns while improving success.The evolving skill bank amortizes experimentation by guiding later exploration, whereas verbal self-reflection without persistent reusable skills is less efficient.Source: §4.4 · pp. 10–11, Figure 6 · p. 11
  • HExA’s hierarchical skills transfer without target trajectories, raising Claude Sonnet on catapult to 44.0% versus 2.0% base and 8.0% REACT, while beating matched-budget GRPO.With 50 seeds, HExA reaches 24% versus GRPO’s 20% on down_to_earth and 14% versus 6% on two_body_problem.Source: Figure 7 · p. 12, §4.4 · p. 13, Figure 8 · p. 13

5 Conclusion, Limitations, & Future Works … A.3.1 Trajectory Reward

HExA is a training-free, in-context reinforcement-learning framework that learns reusable skills through active experimentation, while Interphyre evaluates this capability through tool-mediated physical reasoning. The paper reports strong gains and zero-shot transfer, but identifies evaluation scope, evolver dependence, metric generality, and scaling overhead as limitations motivating cross-domain and hybrid future work.

5 Conclusion, Limitations, & Future Works

  • Interphyre supplies tool-call APIs for inspecting scenes, testing interventions, and observing intermediate dynamics, enabling evaluation of learning through experimentation in 2D physics.The benchmark operationalizes the broader problem of using interaction to obtain instance-specific information beyond an LLM’s parametric knowledge.Source: §5 · p. 13, §A.1 · p. 23
  • HExA substantially outperforms base models and agentic baselines under matched interaction budgets, while reducing per-seed iteration cost and transferring evolved skills zero-shot to unseen levels.The same mechanism improves frontier and open-weight models; compared with GRPO, in-context skill evolution makes newly discovered strategies immediately usable without encoding them into weights over many rollouts.Source: §5 · pp. 13–14
  • Although HExA is intended to be environment-general, current evaluation is confined to 2D physics, while skill quality depends on the evolver and the binary efficiency-aware metric may not transfer directly to other domains.Future work includes scientific data analysis and interactive coding, cross-level and cross-domain transfer, scalable skill banks, and hybrid approaches that bootstrap exploration with HExA before parametric RL refinement.Source: §5 · p. 14

A.1 Overview

  • The framework addresses experimentation settings where agents lack oracle solutions, expert demonstrations, external reward models, or compact human-provided skills by autonomously evolving reusable natural-language knowledge.Its design must retain useful general principles while avoiding skill drift and keeping the library bounded for context-window use.Source: §A.1 · p. 23

A.3 The HExA Framework

  • HExA is a training-free two-agent framework in which an actor experiments, an evolver distills trajectory experience into a skill bank, and a retriever injects relevant skills into subsequent contexts.The framework improves policies through context augmentation rather than parameter updates and learns from scratch on each task family with a single frozen model.Source: §A.1 · p. 23, §A.3 · p. 24

A.2 Problem Formulation

  • HExA formalizes each task family as tool-mediated episodes with a binary success predicate, then maximizes solve rate without parameter updates using a retrieved bank of strategies and common-mistake records.The bank is built entirely from actor-evolver interaction, while scalar trajectory rewards combine outcome and efficiency to provide richer feedback than binary success alone.Source: §A.2 · pp. 23–24

A.3.1 Trajectory Reward

  • Trajectory rewards range from +1.00 for successes completed in at most three iterations to +0.25 for successes by the episode limit, reflecting both outcome and efficiency.The asymmetric failure penalties deliberately value extensive-but-unsuccessful exploration because it yields richer records for extracting partial skills and diagnosing mistakes.Source: §A.3.1 · pp. 24–25

A.3.2 Skill Distillation from Experimentation on Self-Proposed Hypotheses … A.3.5 Initialisation, Update Regimes, and Full Algorithm

HExA distills successful and failed experimental trajectories into reusable skills, mistakes, and partial skills, then retrieves bounded, reward-ranked knowledge for each new task. Its five initialization and update variants include static, online, and evolving banks, with Off2On + Evolution performing best across settings.

A.3.2 Skill Distillation from Experimentation on Self-Proposed Hypotheses

  • The evolver contrasts successful and failed trajectories to extract 4–6 reward-weighted strategy skills describing mechanisms and applicability conditions.Source: §A.3.2 · p. 25
  • From failures, it records mistakes with errors, root causes, and corrective strategies, while preserving partial correct reasoning unavailable from successful trajectories alone.Source: §A.3.2 · pp. 25–26

A.3.3 Skill Bank

  • The skill bank stores task-family-indexed skills and mistakes with reward labels, while hard caps of Mmax skills and Nmax mistakes constrain context growth and prioritize informative knowledge.Source: §A.3.3 · p. 26

A.3.4 Skill Retrieval and Injection

  • At each episode’s start, the retriever injects the top-M reward-ranked skills and top-N task-family mistakes as a fixed prompt prefix compatible with tool-augmented LLMs.Source: §A.3.4 · p. 26

A.3.5 Initialisation, Update Regimes, and Full Algorithm

  • Initialization ranges from warm-start distillation in Offline and Off2On regimes to an empty bank in Pure Online, which bootstraps skills through online interactions.Source: §A.3.5 · p. 26
  • After each round, Evolution merges, revises, or prunes prior skills, Iterative Replacement discards them, and Offline freezes the initialized bank, yielding five concrete variants.Source: §A.3.5 · p. 26
  • Off2On + Evolution works best across all settings and is the variant generally called HExA thereafter.Source: §A.3.5 · p. 26

A.4 Cross-Task Skill Transfer

HExA transfers reusable physical and procedural principles across task families by synthesizing target-specific skills from source skill banks without target-task trajectories, enabling zero-shot transfer and subsequent task-specific refinement.

Cross-task synthesis

  • HExA synthesizes a target-task skill bank from one or more source banks by selecting structurally relevant principles, re-grounding them in target mechanics, and rewarding cross-source corroboration.The method exploits recurring insights such as momentum conservation, lever mechanics, and collision geometry.Source: §A.4 · p. 27

Zero-shot transfer

  • Zero-shot transfer lets the actor attempt a target task using the synthesized bank through the RETRIEVER, without seeing target-task trajectories.The synthesized bank Kℓ∗ is injected into the actor’s context using the same mechanism as within-task skills.Source: §A.4 · p. 27

Cross-task synthesis

  • The synthesized bank can initialize subsequent within-task HExA evolution, combining transferred knowledge with task-specific refinement.Cross-task transfer and within-task skill evolution are complementary rather than alternatives.Source: §A.4 · p. 27

A.5 Design Principles … B.2.2 Two Body Problem — compute_relative_positions()

HExA is domain-agnostic, reward-function agnostic, and adapts entirely in context through a hierarchical skill bank without parameter updates. Its Interphyre interface combines shared and level-specific diagnostic and simulation tools, including geometry analysis for targeted placements.

A.5 Design Principles

  • HExA is domain-agnostic: a generic tool-call interface and trajectory-to-JSON evolver are architecture components, while tools and task prompts supply domain specificity.The framework is instantiated on physics puzzles but is intended for any sequential-interaction setting.Source: §A.5 · p. 27
  • HExA is reward-function agnostic: the evolver can guide skill acquisition without environment rewards, while user- or LLM-designed rewards can provide additional guidance.This broadens applicability to domains where environment rewards are unavailable.Source: §A.5 · p. 27
  • HExA learns hierarchical skills that abstract tool calls into strategies, build on earlier skills, and can be revised or merged as new evidence arrives.The resulting meta-learning concerns both strategies and how to learn strategies from interaction.Source: §A.5 · p. 27

B.2 Tools for Levels

  • The environment exposes shared tools on every level and level-specific tools only on corresponding puzzles, providing structured scene access and simulation control.The appendix describes the complete tool interface used by the REACT agent.Source: B Tool Descriptions · p. 29

B.1 Shared Tools

  • Shared tools report scene state, run full or partial physics simulations, retrieve recent collision logs, and submit a final placement for episode scoring.Simulation calls validate bounds, overlaps, and platform intersections before consuming an attempt.Source: §B.1 · p. 29

B.2.1 Down to Earth — compute_gap_analysis()

  • Gap analysis reports platform span, ball diameter, both gap widths, fit feasibility, and a recommendation for the larger viable side or a warning.This level-specific diagnostic supports choosing how the green ball can pass the separating platform.Source: §B.2.1 · p. 29

B.2.2 Two Body Problem — compute_relative_positions()

  • Relative-position analysis returns ball geometry, separations, contact distance, and the blue ball’s direction from the green ball to guide red-ball placement.The intended placement pushes the green ball into contact with the blue ball.Source: §B.2.2 · p. 29

B.2.3 Catapult — Four Analysis Tools … B.2.8 Tipping Point — compute_tipping_point_analysis()

The paper provides specialized geometry, contact, trajectory, kinematic, and timing analysis tools for physics puzzles. Together, these tools expose actionable scene structure and mechanism-specific quantities for planning interventions.

B.2.3 Catapult — Four Analysis Tools

  • Catapult exposes four complementary tools for surveying geometry, checking first contact, probing green-ball trajectories, and tracing full multi-body kinematics and contacts.The tools support strategy-neutral scene inventory, intended-arm verification, lightweight trajectory sampling, and detailed simulation diagnostics.Source: §B.2.3 · pp. 29–30

B.2.4 Falling Into Place — compute_intercept_setup()

  • Falling Into Place computes the green ball’s platform, the jar’s direction, the required platform crossing, gap center, and estimated fall time.Timing is reported in seconds and 60 Hz physics steps to support intercept planning.Source: §B.2.4 · p. 30

B.2.5 Basket Case — compute_basket_analysis()

  • Basket Case analyzes ball–basket geometry, including the basket opening, ground, center distance, and recommended LEFT/RIGHT deflection direction.It is designed to help the agent avoid the basket while selecting a push direction.Source: §B.2.5 · p. 30

B.2.6 Pass the Parcel — get_ramp_center()

  • Pass the Parcel infers ramp, platform, and basket geometry, including ramp angle and bounds, to guide red-ball placement and downstream interactions.The tool returns the ramp center together with top- and bottom-basket positions.Source: §B.2.6 · p. 30

B.2.7 Cliffhanger — compute_cliffhanger_analysis()

  • Cliffhanger identifies the nearer platform edge, falling-gap width, and required tip direction from detailed bar, platform, ceiling, and ground geometry.It also reports the bar’s endpoints, dimensions, and distances from its bottom point to both platform edges.Source: §B.2.7 · pp. 30–31

B.2.8 Tipping Point — compute_tipping_point_analysis()

  • Tipping Point computes wall-relative bar geometry, pivot-based tipping angle, and suggested LEFT/RIGHT direction for reaching the wall.The angle is approximated as arcsin(∆x/L) when ∆x < L, otherwise reported as N/A.Source: §B.2.8 · p. 31

C Baseline Implementation Details … C.4 GRPO: Weight-Update Baseline

The appendix specifies reproducible implementations for ReAct, Reflexion, DIRECT, and GRPO baselines on INTERPHYRE, including shared execution and decoding protocols. GRPO uniquely updates model weights, while the other methods operate through prompting and tool interaction.

C Baseline Implementation Details

  • The baselines use open-weight models with vLLM or Hugging Face across up to four 80GB A100 GPUs, while Claude Sonnet uses the Anthropic API.Together with level prompts and tool descriptions, these details are intended to reproduce all baseline numbers in Section 4.Source: C Baseline Implementation Details · p. 31

C.1 ReAct: Inner Loop

  • Every method runs the same INTERPHYRE ReAct loop, while HEXA adds skills and Reflexion adds reflection without changing the inner loop.Each turn emits Thought, Action, and Action Input; the harness executes the tool, appends an Observation, and continues from the full conversation.Source: §C.1 · p. 31
  • ReAct trials terminate on a successful finish or simulate_action placement, or after 25 iterations, with up to roughly 24 placement attempts and shared temperature 0.3 decoding.Success requires the target pair to remain in contact for at least 3 seconds, and each turn permits one greedily parsed tool call.Source: §C.1 · pp. 31–32

C.2 Reflexion: K=2 Trial Wrapper

  • Reflexion runs two trials per seed: after a failed first trial, a separate Claude Sonnet instance generates a reflection that conditions a fresh second ReAct trial.The reflection summarizes the failed trajectory, concrete failure causes, and one specific alternative strategy; successful first trials skip reflection and trial two.Source: §C.2 · p. 32

C.3 Direct: One-Shot, No-Tool Baseline

  • DIRECT removes iterative reasoning and simulator interaction, making one scene inspection and one committed placement with no read tools, probes, or retries.It uses a single Claude Sonnet 4.6 call with a no-CoT answer format, then evaluates the placement using the same simulator success predicate.Source: §C.3 · p. 33

C.4 GRPO: Weight-Update Baseline

  • GRPO fine-tunes Qwen-2.5-3B-Instruct on down_to_earth and two_body_problem using VERL-TOOL, serving solely as a sample-efficiency comparison and the only weight-updating method.Training uses four rollouts per gradient step, binary simulator rewards, 50 training seeds per epoch, and 10 epochs totaling 500 gradient steps.Source: §C.4 · p. 34
  • GRPO compares non-skilled prompts against prompts prefixed with HEXA’s fixed final skill bank, testing whether learned skills remain useful as a frozen prefix during gradient-based RL.The static-skilled variant increases maximum prompt length from 2048 to 3072 tokens to accommodate the skill-bank prefix.Source: §C.4 · p. 34
  • GRPO reports greedy validation success at each epoch boundary, with Table 7 and Figure 8 presenting the same data; matched compute is compared by unique seeds rather than rollouts or updates.At 50 unique seeds, GRPO has made 50 gradient updates and 200 rollouts, whereas HEXA has generated 51 trajectories.Source: §C.4 · p. 34

C.5 Seed Ranges per (method, level) Cell

Table 8 specifies the integer seeds for every (method, level) cell, with 50 seeds per cell and paired evaluation pools enabling reproducible within- and cross-level comparisons. HExA’s warm-start, online-update, and transfer protocols ensure attempted seeds are evaluated without using their own trajectories or target-level data.

Seed allocation

  • 50 seeds define each (method, level) cell, with Table 8 documenting the integer ranges passed to the simulator’s level generator.These seeds determine randomized object positions, masses, and dimensions.Source: §C.5 · p. 34, Table 8 · p. 35

Within-level comparisons

  • Within each level, REACT, Reflexion, and HExA use identical 50-seed evaluation pools, making reported solve-rate differences paired contrasts rather than independent-sample comparisons.HExA separately uses seeds 1–5 to build offline bank K0; those warm-start seeds are excluded from evaluation.Source: §C.5 · p. 35, Table 8 · p. 35

Online rounds

  • Each HExA online attempt uses a fresh batch of k=3 or k=5 seeds before the evolver updates the bank, so the attempted seed has not yet been distilled.The bank may incorporate earlier trajectories from the same level, but not trajectories from the current attempted batch.Source: §C.5 · p. 35

Cross-level transfer

  • For cross-level transfer, the synthesized target bank and target-level REACT baseline are evaluated on the same 50 seeds, while synthesis never observes target-level trajectories.Thus, Table 13 compares identical instances under a trajectory-level zero-shot transfer setting.Source: §C.5 · p. 35

D Additional Details of Experiments and Results … E.2 REACT system prompt (catapult seed 45)

HExA improves open-weight agents and is more sample-efficient than GRPO under matched interaction budgets, while GRPO eventually reaches stronger performance with sufficient training. A catapult case study isolates the skill bank’s contribution: REACT fails after 25 iterations, whereas HExA succeeds in six.

D Additional Details of Experiments and Results

  • 24.0% versus 8.0% on down_to_earth and 14.0% versus 6.0% on two_body_problem for Qwen-2.5-3B, while GPT-OSS-120B reaches 54.0% versus 0.0% on catapult with HExA versus REACT.HExA improves every tested open-weight model-level pair, indicating that its skill bank lifts weaker agents rather than exploiting Claude Sonnet’s prior knowledge.Source: D Additional Details of Experiments… · pp. 35–36, Table 10 · p. 37
  • 76% on catapult and 60% on pass_the_parcel are achieved by Off2On Evolving, which remains above other HExA variants and baselines throughout the cumulative seed sweep.The same configuration also converges to the lowest average turns per seed, indicating more efficient search rather than greater computation per instance.Source: Figure 11 · p. 36, Figure 12 · p. 37, Table 12 · p. 38
  • At the matched 50-seed budget, GRPO reaches 20% versus HExA’s 24% on down_to_earth and 6% versus 14% on two_body_problem, but later reaches 100% on down_to_earth and two_body_problem with sufficient training.GRPO’s exact-reward optimization eventually dominates, whereas HExA provides stronger early sample efficiency through context-injected skills; static skills show an epoch-1 dip before re-adaptation.Source: D Additional Details of Experiments… · pp. 36–37

E Working Example: ReAct vs. HEXA on Level Catapult of InterPhyre

  • On catapult seed 45, REACT fails after 25 iterations, whereas HExA succeeds in 6 iterations at final placement (x=0.3, y=0.9, r=1.5).Both agents use Claude Sonnet 4.6 and identical tools, format instructions, and kickoff messages; the prompts differ only by HExA’s injected LEARNED PHYSICS SKILLS block.Source: E Working Example: ReAct vs. HEXA on… · p. 38

E.1 Sparsified simulation frames (catapult seed 45)

  • HExA’s six-iteration placement puts the heavy red ball on the catapult arm, flings the green ball over the ceiling blocker, and settles it against the blue ball.REACT’s 25 iterations never produce a placement bringing the green ball into contact with the blue ball.Source: §E.1 · p. 38

E.2 REACT system prompt (catapult seed 45)

  • The REACT system prompt defines a 2D physics puzzle requiring one red-ball placement so the green ball contacts the blue ball for at least 3 seconds.It constrains the red ball to remain inside the box, avoid initial overlap, and use a radius between 0.1 and 2.0.Source: §E.2 · p. 39
  • REACT can inspect geometry and level state, simulate full or partial trajectories, trace the green ball, inspect contact logs, and predict first contact before finishing.The prompt requires Thought/Action/Observation cycles, one tool per step, and resetting the environment between simulations.Source: §E.2 · pp. 39–40

E.3 REACT trajectory (catapult seed 45, 25 iterations, FAILURE)

This section examines REACT’s iterative experimentation on a catapult-seed task, including contact prediction, simulation, and variations in ball size, drop location, impact target, and trajectory. Across these experiments, REACT identified some promising interactions but did not complete a successful intervention.

Contact planning

  • The initial drop at (1.5, 2.0) hit the ledge instead of the catapult arm, prompting a revised drop at (1.0, 1.0).The revised prediction validly contacted gray_platform at step 28 with approach_speed 2.006.Source: §E.3 · p. 41

Failure outcome

  • Success: False across the attempted full simulations, with the green ball failing to reach the blue ball.One trace also reported unknown object_names: catapult_arm.Source: §E.3 · p. 42

Catapult trials

  • A red_ball drop was placed at (1.000, 3.000); the green ball peaked at y=2.34 and moved right by 2.41 units but did not reach blue_ball.Source: §E.3 · pp. 42–43
  • Larger, higher drops repeatedly launched the green ball upward-left rather than toward the target, despite valid gray_platform contacts.The agent recorded this directional failure after the radius-1.0 trial and continued adjusting placement.Source: §E.3 · p. 43

Alternative strategy

  • The agent tried striking blue_ball directly from (3.5, 0.0), but the resulting simulation also failed.The trace recorded blue_ball–red_ball contacts without achieving task success.Source: §E.3 · pp. 44–45

REACT trajectory

  • The agent repeatedly revised its hypothesis after observing that the green ball launched leftward or missed the catapult arm.It tried a larger ball, a precisely aimed small ball, and basket displacement as alternative mechanisms for reaching the blue ball.Source: §E.3 · pp. 46–47
  • Simulation remained unsuccessful after testing large-ball catapult impacts, high-speed tip drops, and attempts to knock the basket toward the green ball.The r=2.0 trial and basket-impact trial both returned “Success: False.”Source: §E.3 · p. 47
  • +2.41 rightward displacement and v_max=10.22 for r=0.8 suggested a potentially useful launch trajectory toward the basket at (3.50, -3.03).REACT consequently tested a moderate-force r=0.4 launch and planned partial simulations to inspect the r=0.8 trajectory.Source: §E.3 · p. 48
  • Partial simulations were used to inspect the green ball’s position near the basket and its initial velocity direction after the r=0.8 catapult trigger.The agent requested states at step 120 and step 50 to diagnose the trajectory.Source: §E.3 · p. 48

E.4 Round-14 evolving skill bank used by HEXA · E.5 HEXA system prompt (catapult seed 45)

This section presents HEXA’s evolving catapult skill bank and system prompt, which together organize reliable launch configurations, geometry-aware fallback strategies, pre-simulation contact checks, and explicit review of failed simulations. It also records failure modes and cautions against ineffective parameter adjustments.

E.4 Round-14 evolving skill bank used by HEXA

  • At x=0.5, r=1.5 is the baseline because r=2.0 adds negligible range after arm rotation saturates, while x shifts substantially alter landing outcomes.The stable launch zone is x~=0.0 to 0.5; x<=-0.3 and x=0.7--1.5 are bifurcation zones, whereas x=0.1--0.5 is predictable and monotonically tunable.Source: §E.4 · pp. 49–50
  • The canonical first attempt is x=0.5, y=0.4, r=1.5, with y=0.9 reserved for higher arms and x=0.3 used to flatten ceiling-threatening arcs.At x=0.3, y=0.4 avoids gray_ball overlap while preserving the arc-flattening effect; the ceiling blocker’s position determines whether a ceiling escape is needed.Source: §E.4 · pp. 49–50
  • predict_first_contact is required before every simulate_with_trace call to validate placement, confirm red-ball contact with gray_platform, and catch gray_ball pivot overlaps.Invalid placements are adjusted before simulation, preventing wasted runs.Source: §E.4 · p. 49
  • The skill bank prioritizes a three-tier fallback: from (x=0.5, y=0.4, r=1.5), shift to x=0.3 after ceiling hits, increase y or adjust x for short range, and abandon radius tuning after repeated failures.The prescribed ceiling escape is x=0.3, y=0.4, r=1.5; persistent failures after 2+ radius variations trigger an x shift to 0.2--0.3.Source: §E.4 · p. 50
  • The negative skills warn against local micro-tuning: after repeated failures, agents should change x zones, avoid increasing y after ceiling hits, and fix green-ball trajectory rather than moving the blue ball with deflectors.These errors include radius increases at fixed x=0.5 despite rotation saturation and attempts to use deflectors near the basket to redirect the blue ball.Source: §E.4 · p. 51

E.5 HEXA system prompt (catapult seed 45)

  • The HEXA system prompt begins by defining the agent as an expert physics-reasoning solver for a 2D physics puzzle with available access.The supplied prompt fragment is introductory and does not specify further tools or procedures.Source: §E.5 · p. 52
  • The tool interface supports scene inspection, full and partial simulation, contact logging, trajectory tracing, first-contact prediction, and final submission through one-action Thought/Action cycles.Each simulation resets the environment, and the prompt requires parsing observations before continuing or using finish.Source: §E.5 · pp. 52–53
  • predict_first_contact verifies placement validity and intended gray_platform contact before expensive simulation, catching pivot overlaps and avoiding wasted runs.The tool reports the first contacted object, impact timing, approach speed, contact point, and surface normal within at most 90 physics steps.Source: §E.5 · pp. 53–54
  • The prompt prescribes a three-tier fallback sequence: shift to x=0.3 after ceiling hits, adjust y or x for short range, and stop radius tuning after repeated failures.The fallback sequence explicitly recommends x=0.3, y=0.4, r=1.5 after a ceiling hit and shifting x after two or more radius variations fail.Source: §E.5 · pp. 53–54
  • HEXA uses x≈0.5 and r=1.5 as the primary catapult placement and baseline radius, deviating to x=0.3 or reducing radius only when ceiling hits or overlap constraints require it.Source: §E.5 · pp. 53–54
  • The prompt treats ceiling-blocker position and launch x as key geometric conditions: x=0.3 supports ceiling escape, while x=0.0–0.5 remains the stable tuning zone.Blockers at x > -2.5 require ceiling escape, whereas far-left blockers can permit x=0.3 trajectories to reach the top wall and descend into the basket.Source: §E.5 · p. 54
  • The prompt warns that micro-tuning the catapult, increasing radius at x=0.5, raising y after ceiling hits, and searching bifurcation zones produce ineffective or chaotic behavior.It recommends shifting x instead of increasing radius or height, and resetting from x≤-0.3 or x=0.7–1.5 to x=0.2–0.4.Source: §E.5 · pp. 54–55
  • HEXA must solve the catapult puzzle within 12 iterations by making the green ball contact the blue ball.Source: §E.5 · p. 55
  • The agent must always call predict_first_contact first, then call trace_green_ball only when prediction confirms the intended contact.The prompt describes prediction as cheap and useful for checking whether the red ball hits the intended object.Source: §E.5 · p. 55
  • After each failed simulation, HEXA must review the learned physics skills, identify the applicable skill, and explain its use before the next placement.Source: §E.5 · p. 55

E.6 HEXA trajectory (catapult seed 45, 6 iterations, SUCCESS) · F More Background and Preliminaries · G Interphyre: Environment and Benchmark Details

The HExA trajectory solves the catapult task after using scene-dependent skills to revise a failed launch, while the appendix formalizes tool-augmented interaction, MDPs, reinforcement learning, and Interphyre’s benchmark specification.

E.6 HEXA trajectory (catapult seed 45, 6 iterations, SUCCESS)

  • The agent used scene geometry and learned placement rules to raise the drop height from y=0.4 to y=0.9 because the catapult arm was unusually high at y≈−0.87.It first verified that x=0.5, y=0.9, radius=1.5 was valid and contacted the gray platform.Source: §E.6 · pp. 55–57
  • 343 steps and reward 1.0 yielded SUCCESS after changing the launch to x=0.3, y=0.9, radius=1.5, causing the green ball to contact the blue ball.The initial x=0.5, y=0.9, radius=1.5 trial failed after a ceiling hit; the fallback x=0.3 flattened the arc and succeeded.Source: §E.6 · pp. 56–57

F More Background and Preliminaries

  • Tool-augmented LLM agents interleave thoughts, actions, and observations to gather information, execute APIs, and adapt strategies using real-time environmental feedback.This interaction pattern extends agents beyond static parametric knowledge when operating in complex environments.Source: F More Background and Preliminaries · p. 58
  • The appendix models agent–environment interaction as an MDP M = ⟨S, A, p, r, γ, p0⟩, generating trajectories from states, actions, rewards, and transitions.The state space, action space, transition dynamics, reward function, discount factor, and initial-state distribution define the tuple.Source: F More Background and Preliminaries · p. 58
  • Reinforcement learning alternates policy evaluation and control to estimate returns and optimize a policy through online or offline interaction data.The appendix distinguishes direct environment interaction from learning on static datasets of prior interactions.Source: F More Background and Preliminaries · p. 58
  • In-context reinforcement learning keeps pretrained weights frozen while conditioning actions on the current state and evolving context, enabling adaptation and in-context improvement.Performance generally improves with the length and quality of the context, including in test environments differing from pretraining environments.Source: F More Background and Preliminaries · p. 58

G Interphyre: Environment and Benchmark Details

  • The Interphyre appendix specifies the environment’s design, intervention API, benchmark details, and limitations.These materials provide the full specification for the benchmark environment used to study experimental interaction.Source: G Interphyre: Environment and… · p. 59

G.1 Introduction … G.4.2 Event-Driven Branch Points

Interphyre is an intervention-aware 2D physics environment designed for paired counterfactual analysis, mechanistic interpretability, and controlled RL generalization. Its Python-authored, parameterized curriculum combines deterministic snapshot/restore branching, LLM and RL interfaces, and event-driven experimentation.

G.1 Introduction

  • Interphyre addresses physics benchmarks’ lack of paired counterfactuals and independent physics control through snapshot/restore branching, tunable parameters, and a 25-level curriculum.Existing environments do not provide programmatic paired rollouts from a shared physical state or independent variation of physics while holding scene structure fixed.Source: §G.1 · pp. 59–60

G.2 Related Work

  • Existing benchmarks evaluate puzzle solving, dynamics discovery, or behavioral transfer, but none provides Interphyre’s activation-matched counterfactuals from a shared simulation state.PHYRE (Bakhtin et al., 2019), I-PHYRE (Li et al., 2024), Kinetix, and related causal environments lack this complete pairing and instrumentation combination.Source: §G.2 · pp. 60–61

G.3 Design; G.3.1 Levels as Python Code

  • Python-defined levels let researchers fork geometry, action slots, and success conditions, while the composable API supports editable, extensible task curricula without rebuilding the package.Researchers can register custom levels and alter scene topology directly through ordinary Python code.Source: §G.3 · p. 61, §G.3.1 · p. 61, §G.3.3 · p. 63

G.3.2 Physics Parameters as First-Class Inputs

  • Physics parameters are first-class configuration inputs, allowing the same level, seed, and action to be tested across regimes while keeping scene structure fixed.Gravity, friction, restitution, density, and solver step size can be varied independently, unlike Kinetix’s joint variation of physics and topology.Source: §G.3.2 · pp. 61–62

G.3.3 Curriculum and Oracle Verification

  • 250,000 pre-certified instances are guaranteed solvable through exhaustive oracle search over a discretized action grid, making failures attributable to agents up to grid resolution.Certification is decoupled from runtime and can be extended to custom levels or seed ranges.Source: §G.3.3 · pp. 62–63

G.3.4 Agent Interfaces and Observation Modes

  • A shared simulator backend exposes both Gymnasium-compatible RL interfaces and named tool calls for LLM agents, with symbolic, RGB, discrete-color, or combined observations.Because all observation modes run the same simulation, modality is a controlled experimental variable.Source: §G.3.4 · pp. 63–64

G.4 Intervention API; G.4.1 Snapshot/Restore and Counterfactual Branching

  • Snapshot/restore captures complete deterministic physics state and creates factual and perturbed branches with a bit-identical prefix, enabling causal interpretation of divergent outcomes.The snapshot includes contacts and contact-duration accumulators, while the protocol treats the perturbation as the treatment and the suffix as the outcome.Source: §G.4 · p. 64, §G.4.1 · p. 64

G.4.2 Event-Driven Branch Points

  • Event-driven triggers define branch points by physical conditions rather than fragile step indices, and composable triggers support ordered or alternative event sequences.This decouples experimental timing from seed-specific contact steps.Source: §G.4.2 · p. 64

G.4.3 Worked Example: Strategy-Dependent Causal Structure

The worked example shows that causal relevance depends on the strategy used: interventions identify object roles for individual trajectories but may cancel when pooled across strategies. The protocol snapshots a physical event, waits for trajectory divergence, and then branches under structural modifications.

G.4.3 Worked Example: Strategy-Dependent Causal Structure

  • Different strategies produce opposite intervention effects: removing the deflector breaks seed 8 but not seed 5, whereas shrinking the action ball breaks seed 5 but not seed 8.Seed 8 uses a deflector bounce, while seed 5 reaches the basket directly; both succeed in the factual branch.Source: §G.4.3 · p. 64
  • Causal relevance is strategy-dependent, so pooling strategies can average away opposite effects and recover neither causal relationship when strategies occur at similar rates.The same remove_object intervention is necessary for one strategy and irrelevant for the other.Source: §G.4.3 · p. 65
  • The protocol snapshots red-ball contact with the catapult arm, waits 30 steps for trajectories to diverge, and then applies three structural-modification branches.The branch point occurs once the green ball is airborne and the seeds’ trajectories have separated.Source: §G.4.3 · p. 65

G.4.4 Perturbation Primitives as Experimental Treatments … I Evolver Prompts for HEXA and Cross-Level Transfer

This section defines perturbation-based experimental analyses and the prompts, tools, and environments used to investigate 2D physics puzzles. It also describes HEXA’s evidence-driven skill extraction, evolution, grounding, and transfer across levels.

G.4.4 Perturbation Primitives as Experimental Treatments

  • Kinematic and structural perturbations define interpretable experimental treatments by varying motion variables or scene composition while holding intervention type constant and varying seeds.Kinematic treatments include impulse, continuous force, velocity or position overrides, and freezing; structural treatments add or remove objects.Source: §G.4.4 · p. 65

G.5 Conclusion

  • Strategy-conditioned probes predict counterfactual outcomes above chance across three levels, while snapshot/restore isolates perturbation effects from scene-statistics shifts.The finding is demonstrated with Qwen3-8B residual-stream activations and requires conditioning probe targets on each seed’s oracle strategy.Source: §G.5 · p. 65
  • The platform’s independent physics-parameter axis isolates physics transfer from structural variation, while its hand-authored curriculum supports curriculum learning and hypothesis-driven behavioral tests.The current probing evidence uses one model family on three levels, and the 25-level curriculum requires additional authoring effort to scale.Source: §G.5 · pp. 65–66

H ReAct System Prompts

  • ReAct receives factual scene descriptions, success conditions, placement constraints, complete tool lists, and standard format instructions without strategy or physics hints.The prompt requires alternating Thought/Action cycles, one tool call per step, careful observation parsing, and independent simulations that reset the environment.Source: H ReAct System Prompts · p. 67, §H.1 · p. 68

H.1 Down to Earth

  • Down to Earth provides state inspection, full and partial simulation, contact logging, gap analysis, and final submission tools for knocking the green ball from a platform.Placement must satisfy box-boundary and non-overlap constraints, and the goal requires the green ball to touch the purple ground for at least 3 seconds.Source: §H.1 · p. 67

H.2 Two Body Problem

  • Two Body Problem adds relative-position analysis that reports ball geometry and recommends a red-ball placement direction alongside simulation and contact tools.The agent follows the same stepwise Thought/Action protocol and can compare multiple reset simulations before finishing.Source: §H.2 · p. 69

H.3 Pass the Parcel

  • Pass the Parcel supplies a ramp-center analyzer plus simulation, partial simulation, contact logging, and submission tools for making the green ball contact the blue ball for at least 3 seconds.The level constrains one non-overlapping red ball within the box and includes baskets, a platform, and a ramp that guides the intervention.Source: §H.3 · p. 70
  • H.3 Pass the Parcel requires a Thought/Action cycle with one tool per step, careful observation parsing, reset simulations, and a final finish submission.Agents may simulate multiple actions to compare results, but each simulation starts from the reset environment.Source: §H.3 · p. 71

H.4 Catapult

  • H.4 Catapult asks agents to place one red ball so the green ball contacts the blue ball for at least 3 seconds, using geometry, simulation, contact, trajectory, and partial-dynamics tools.Available diagnostics include first-contact prediction, green-ball tracing, selected-object traces, and contact logs before final submission.Source: §H.4 · pp. 71–72

H.5 Falling Into Place

  • H.5 Falling Into Place asks agents to place one red ball so the green ball touches the downward-facing blue jar for at least 3 seconds.The prompt includes full and partial simulation, contact logging, and a level-specific intercept setup that reports the green ball’s platform, crossing edge, gap center, and estimated jar timing.Source: §H.5 · pp. 73–74

H.6 Basket Case

  • H.6 Basket Case asks agents to deflect the falling green ball away from the upward-facing basket so it touches the purple ground for at least 3 seconds.The red ball must remain inside the box, avoid initial overlap with the green ball or basket, and have radius 0.1 to 2.0.Source: §H.6 · p. 75
  • H.6 Basket Case provides full and partial simulation, contact logging, and geometry analysis tools for testing ball placements and deflecting the green ball away from the basket.The prompt enforces one Thought/Action cycle at a time, allows comparisons across reset simulations, and uses finish to submit the final placement.Source: §H.6 · pp. 75–76

H.7 Cliffhanger

  • H.7 Cliffhanger defines a goal in which one red ball must cause the green bar to contact the purple ground for at least 3 seconds, supported by simulation, contact-log, partial-run, and cliffhanger-geometry tools.The red ball must remain inside the box, avoid initial overlap, and have radius 0.1 <= radius <= 2.0.Source: §H.7 · pp. 76–77

H.8 Tipping Point

  • H.8 Tipping Point asks the agent to place one red ball so the green bar maintains contact with a purple wall, with analysis returning the required tip angle, wall relation, and suggested direction.The environment contains a green bar in a gray basket and a purple wall at either side of the box; valid placements must satisfy the box-boundary, non-overlap, and radius constraints.Source: §H.8 · p. 78

I Evolver Prompts for HEXA and Cross-Level Transfer

  • The appendix reproduces the runtime-filled evolver prompts used for every skill-distillation result in the paper, including level identifiers, trajectory blocks, and contextual placeholders.Double-braces represent literal JSON braces that appear as single braces at runtime.Source: I Evolver Prompts for HEXA and… · p. 80

I.1 Pass 1 Contrastive Skill Distillation

  • Pass 1 extracts actionable physics skills by contrasting successful and failed trajectories from the same level batch, weighting high-reward fast solves more heavily than low-reward outcomes.The prompt assigns successes rewards from +1.0 to +0.25 and failures from -0.5 to -0.75, then requests each skill’s title, principle, application trigger, and source seeds.Source: §I.1 · p. 80

I.2 Pass 2 Mistake & Partial-Skill Extraction

  • Pass 2 extracts 3 to 5 root-cause mistake patterns and 1 to 2 genuine partial skills from failed trajectories, including correct reasoning or discoveries embedded in otherwise unsuccessful attempts.Mistakes require an explanation of the error, its broken causal belief, and an actionable fix, while partial skills must state physics insights rather than merely restating failures.Source: §I.2 · p. 81

I.3 Skill-Bank Evolution

  • Skill-bank evolution merges existing and newly extracted insights while retaining validated skills, removing redundant or contradicted entries, and enforcing maximum skill and mistake counts.New-skill confidence reflects source-trajectory success, universality, clarity, and actionability; retained skills preserve their confidence, and low-confidence skills are removed unless directly observed in new trajectories.Source: §I.3 · p. 82

I.4 Catapult-Specific Variants (Claude)

  • Catapult-specific prompts contrast successful and failed trajectories, weighting skills from higher-reward, faster solutions more heavily and asking for distinct mechanisms grounded in observed behavior.The variant applies the same mistake and partial-skill extraction and skill-bank evolution procedures to the catapult scene, with explicit reward tiers for solution speed.Source: §I.4 · pp. 84, 86

I.5 Skill Transfer Prompts

  • Cross-level transfer synthesizes 6 to 10 target-level skills and 2 to 4 mistakes from source banks, requiring source citations, physics rationales, target grounding, and calibrated confidence.Skills must cite source entries, avoid invented coordinates and source-only entities, describe specific mechanisms, include transfer rationales, and receive confidence 0.7+ only when supported by at least two source banks and a clear target application.Source: §I.5 · p. 87

I.6 Cross-Level Synthesis on Qwen-7B

  • For Qwen 2.5 7B, the slim cross-level prompt removes audit fields, reduces the rules to three, and limits output to 4–6 skills and 1–3 mistakes.It tightens constraints on principles, applicability conditions, and examples to reduce platitudes and hallucinated entities such as inventing a lever where no pivot exists.Source: §I.6 · p. 88
  • The prompt requests 4–6 reusable skills and 1–3 mistakes for the target level.Skills should support reasoning from physics observations, mechanisms, or relationships rather than prescribe actions.Source: §I.6 · p. 89
  • Each skill must state a conditional or relational principle, a reasoning context, and an abstract or contrastive example, never an imperative or placement instruction.Skill fields must describe specific mechanisms rather than generic advice.Source: §I.6 · p. 89
  • Skills must use only entities in the target scene and exclude coordinates, placements, and source-only mechanisms absent from that scene.For example, scenes without pivots should not receive lever or torque-around-a-pivot skills.Source: §I.6 · p. 89
  • Mistakes must represent misconceptions and their corrected understandings, while the output must be a concise single JSON object containing the prescribed fields.The format includes skill metadata, mistake explanations, and synthesis reasoning about the transferred source primitive.Source: §I.6 · p. 89

Related papers

Loading 2606.29315v1…