PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses

Tailin Zhou

arXiv:2608.08466v1cs.AI

TL;DR

Existing approaches often leave the executable harness surrounding a frozen LLM fixed, motivating whether the harness itself can evolve endogenously. HSI hierarchically rewrites task harnesses and evolution strategies under frozen anchors, and BALROG experiments show gains on moderate tasks plus held-out generalization, subject to feedback and backbone limits.

  • Problem

    Existing self-improvement methods have not fully reached the harness layer endogenously, leaving open whether a frozen model can evolve its own harness and what limits improvement.

  • Method

    HSI uses one frozen LLM across task-harness, evolver, and meta-evolver scopes, with a frozen outer anchor and thinking disabled for task execution but enabled for rewriting.

  • Results

    HSI consistently improves initial harnesses on moderate-difficulty BALROG environments and generalizes to held-out tasks within BabaIsAI under unchanged backbone and task-time inference configuration.

  • Takeaways & Limitations

    Task-specific endogenous harness evolution is a viable way to extract additional capability from frozen models through environment-grounded adaptation.

  • Takeaways & Limitations

    Evolution is limited by uninformative feedback and the frozen backbone’s capability, with extremely sparse-reward environments providing insufficient signals for improvement.

Abstract

from arXiv · show

Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the \emph{harness}---is typically treated as a fixed artifact after deployment. This work studies an alternative where the harness is \emph{task-specific and continuously evolvable}: each task family maintains its own harness, which is hot-swapped across iterations through a fixed task-injection seam and rewritten using environment feedback. We introduce \textbf{Hierarchical Self-Improvement (HSI)}, a framework in which a single frozen LLM $M$ operates across three hierarchical scopes: a task harness $H$ that executes tasks, an evolver that rewrites $H$, and a meta-evolver that rewrites the evolver's strategy code under a frozen outer anchor. A thinking-on/off design isolates the contribution of harness evolution by disabling reasoning during task execution while enabling it during self-modification. HSI is bounded by two factors: a \emph{feedback-fidelity bound}, since evolution requires informative reward signals to guide selection, and a \emph{backbone capability bound}, since harness redesign cannot overcome limitations of the frozen model. On BALROG with DeepSeek-V4-Flash-Preview as the frozen backbone, HSI achieves consistent gains over the initial harness on moderate-difficulty tasks ($+39.3$ on BabyAI, $+33.0$ on Crafter, $+25.0$ on TextWorld, and $+15.0$ on MiniHack, all in raw \% Progress), while obtaining strong held-out generalization on BabaIsAI sub-suites ($0.98$ best-test on BreakStop and $1.00$ on GoTo from a $20\%$ unseen split). On tasks beyond the backbone's capability (NLE), harness evolution provides no improvement. These results demonstrate task-specific harness evolution as a viable axis for improving frozen LLM agents under clear empirical limits. Code is available at https://github.com/TailinZhou/hsi.

1 Introduction

HSI asks whether a frozen LLM can endogenously evolve its task harness and what limits that improvement. It answers with hierarchical harness evolution, controlled reasoning, and BALROG evidence of gains on moderate-difficulty environments.

  • Framework: HSI evolves a task harness around a frozen LLM rather than updating the model parameters.The framework separates task execution, harness rewriting, and evolution-strategy rewriting into hierarchical scopes.
  • Framework: HSI uses three scopes: a task harness executes tasks, an evolver rewrites the harness, and a meta-evolver rewrites the evolver strategy.A frozen outer anchor constrains meta-evolution and prevents unrestricted self-reference.
  • Experimental design: Thinking is disabled during task execution and enabled during self-modification to isolate harness evolution from inference-time reasoning.This design fixes the model’s per-step task-time capability ceiling while supporting self-modification.
  • Empirical evidence: On BALROG, HSI consistently improves matched initial harnesses across moderate-difficulty environments while keeping the frozen backbone and inference configuration unchanged.The evaluation compares full-set evolution for in-distribution improvement with split evolution for held-out generalization.
  • Limits: HSI identifies feedback availability and backbone capability as practical boundaries on harness evolution.The framework cannot overcome insufficient feedback or fundamental limitations of the underlying frozen model.

2 Related Work

Related work spans self-modifying agents, harness engineering, and evaluation of self-improvement. HSI differs by evolving a broader harness endogenously, using task-specific continuous evolution and hierarchical frozen constraints.

  • Self-improvement: Gödel-style work studies self-improvement through program modification, runtime editing, evolutionary search, and population-based exploration.These approaches include Gödel Agent, Darwin Gödel Machine, Huxley-Gödel Machine, and Group-Evolving Agents.
  • Harness engineering: Harness-engineering research optimizes prompts, tools, memory, and verification mechanisms through outer-loop search, automatic synthesis, or deployment-time adaptation.Examples include Meta-Harness, AutoHarness, TTHE, Live-SWE-Agent, Continual Harness, and Adaptive Auto-Harness.
  • Evaluation: Prior evaluation work shows that harness configuration can change performance under identical models and distinguishes producing harness updates from benefiting from them.These findings motivate treating harness design and self-improvement benefit as separate evaluation questions.
  • Theory: Theory characterizes statistical limits of self-improving agents, including preserved distribution-free PAC guarantees when reachable hypothesis families have bounded complexity.This provides a theoretical lens for capability boundaries under self-modification.
  • HSI comparison: HSI evolves the harness coordinating prompts, tools, memory, state, and cross-step interactions using the same frozen model that executes tasks.This broadens the endogenous editable surface beyond primarily evolving decision procedures or execution code.
  • HSI comparison: HSI uses task-specific continuous evolution rather than optimizing one universal harness, with generalization assessed through held-out task splits.The design responds to evidence that evolved harnesses may overfit and fail to beat simple test-time scaling baselines.
  • HSI comparison: HSI constrains self-modification through a fixed harness interface, editable higher-level evolution strategy, frozen outer anchor, and fixed evaluation signals and splits.These boundaries are intended to support recursive improvement without unrestricted self-reference.

3 Hierarchical Self-Improvement

HSI enables a single frozen LLM to evolve task-specific harnesses and the strategy that rewrites them through separated hierarchical scopes. Its bounded loop combines feedback-guided harness modification, persistent evolutionary information, and an immutable outer anchor.

  • Design principles: Each task family maintains a hot-swappable harness connected through a fixed task-injection seam and refined using environment feedback.
  • 3 Hierarchical Self-Improvement: HSI keeps model parameters fixed while improvements arise from modifications to the task harness and its evolution procedure.
  • Bounded self-modification: HSI leaves exploration scheduling to the evolvable strategy while enforcing editable boundaries, evaluation interfaces, and a frozen outer anchor.These constraints support endogenous evolution without unrestricted self-reference.
  • Hierarchical architecture: The task-harness scope executes task-facing components, while the evolver modifies the harness and the meta-evolver modifies the strategy Σ.The three scopes use the same frozen model but have different editable surfaces and execution contexts.
  • Self-governing evolution loop: Evolution proceeds through seed selection, main evolution, commit selection, and meta-evolution, followed by terminal best-version selection.The first three stages modify H, whereas meta-evolution modifies Σ.
  • Seed selection: Seed selection uses the evolution graph and accumulated feedback to generate a hypothesis containing an anchor version, motivation, expected direction, and falsification criterion.This converts mutation into a goal-directed search guided by explicit predictions.
  • Main evolution: Main evolution edits task-facing prompts, tools, memory, state management, hooks, and execution policies, while candidate evaluation supplies reward feedback for later edits.

4 Experiments

HSI evaluates task-specific harness evolution on BALROG under a frozen backbone, using controlled baselines, held-out splits, and trajectory analyses. It improves moderate-difficulty tasks and transfers on navigation suites, while gains diminish near the backbone’s capability boundary.

  • Experimental setup: BALROG spans six long-horizon interactive environments testing planning, memory, exploration, instruction following, and tool use.
  • Experimental setup: HSI compares evolved harnesses with matched initial-harness baselines while keeping the frozen backbone and task-time inference configuration unchanged.
  • In-distribution evolution: +39.3 BabyAI, +33.0 Crafter, +25.0 TextWorld, and +15.0 MiniHack in raw % Progress over the init harness.These gains come from the meta-evolution-on configuration under the same backbone and task-time reasoning budget.
  • In-distribution evolution: Removing meta-evolution reduces performance on every evaluated suite, including TextWorld from 65.0 to 46.0 and MiniHack from 15.8 to 5.8.The reported largest differences are +19.0 on TextWorld and +10.0 on MiniHack.
  • Capability boundary: Harness evolution produces the largest gains where the frozen backbone is already competent, but smaller or limited gains near capability and feedback boundaries.The authors describe harness evolution as reorganizing and amplifying existing capabilities rather than overcoming insufficient feedback or fundamental backbone limitations.
  • Evolution trajectory analysis: Across trajectories, early iterations discover missing representations, middle iterations add structured algorithmic components, and later iterations refine or prune designs.Crafter’s best version occurs at iteration 4 before a later regression, while meta-evolution converts local discoveries into reusable search heuristics.

5 Discussion and Conclusion

HSI shows that environment-grounded, task-specific harness evolution can improve a frozen model, while informative feedback and backbone capability constrain the gains. Its hierarchical scopes support recursive modification without unrestricted self-reference, and single-lineage evolution favors attribution over search efficiency.

  • Lessons: Execution-grounded reward feedback is necessary because static inspection can produce plausible harness changes that fail to improve task performance.HSI therefore evaluates candidates through interaction and uses reward feedback to guide successive modifications.
  • Lessons: Single-seed evolution improves attribution clarity but trades search efficiency for clearer measurement of endogenous improvement.Population-based exploration remains a complementary scaling option.
  • Limits: Harness evolution expands a fixed model’s effectiveness only within an empirical capability frontier defined by informative feedback and reachable backbone competence.Gains are larger on TextWorld, BabyAI, Crafter, and BabaIsAI-GoTo/BreakStop, while harder environments show smaller or negligible improvements.
  • Framework: HSI separates task-harness, evolver, and meta-evolver scopes with frozen outer anchors to enable recursive improvement without unrestricted self-reference.The task harness interacts with the environment, the evolver rewrites it, and the meta-evolver rewrites the evolution strategy.
  • Empirical findings: A fixed DeepSeek-V4-Flash backbone substantially improves through endogenous harness evolution alone across multiple BALROG environments.The comparisons keep the backbone and task-time inference configuration unchanged.
  • Empirical findings: Held-out BabaIsAI evaluation shows that evolved harnesses generalize to unseen tasks within the same task family rather than only individual trajectories.This supports task-family-specific reuse of evolved strategies.

A Implementation Details of the HSI Agent System

The implementation instantiates HSI as a shared execution system whose purpose is to realize the hierarchical architecture through controlled environments, tools, memory, and scope isolation.

  • Implementation details: The implementation details describe the execution environment, tool interfaces, memory organization, and scope isolation mechanisms underlying HSI.These components realize the framework’s hierarchical architecture.

A.1 Agent Execution Interface

All HSI scopes use the same frozen model and execution primitive, while distinct tools and context let the model perform task execution, harness evolution, or meta-evolution.

  • Agent execution interface: All HSI components use the same frozen LLM M and shared react() primitive, differing through tool sets and system context.The underlying model remains unchanged across scopes.
  • Agent execution interface: At each step, the model combines message history, available tools, and task-specific context to produce an action.Actions can modify files, request evaluation, record information, or terminate a stage.
  • Scope separation: The task-harness scope executes H against the benchmark, the evolver modifies harness files and evaluates development tasks, and the meta-evolver modifies evolution-strategy files.Tool availability defines these abstraction levels.
  • Scope separation: Scope separation lets one model operate at different abstraction levels without additional learned parameters or external optimizer models.The architecture changes access and context rather than the underlying model.

A.2 Evolution Tool Interface

The evolver receives atomic file and evolution tools, then chooses their ordering according to observed feedback and the current evolution objective.

  • Evolution tool interface: The evolver can inspect, modify, and evaluate candidate harnesses using a small set of atomic tools.These capabilities support iterative harness redesign.
  • File operations: File operations include read, write, edit, and bash for inspecting source, creating files, changing implementations, and debugging or verification.These tools provide direct access to harness code.
  • Evolution operations: Evolution-specific primitives maintain plans, compact context, evaluate harnesses, record lessons, and terminate evolution.The evaluate primitive returns environment feedback for candidate harnesses.
  • Evolution policy: HSI imposes no fixed operation ordering; the model selects tool use based on observed feedback and the current evolution objective.Tool sequencing is therefore part of the model’s evolution strategy.

A.3 Memory Organization

HSI organizes memory across temporary, persistent, and graph-based channels, while probes compress historical trajectories for bounded-context retrieval. Explicit scope boundaries constrain which layer may modify which component and preserve a fixed outer execution boundary.

  • Memory channels: HSI maintains iteration-local, persistent evolutionary, and evolution-graph memory channels with different persistence properties.Temporary notes roll back with discarded candidates; lessons persist across iterations; the graph stores committed versions, rewards, metadata, and relationships.
  • Memory channels: The evolution graph links harness versions by semantic relationships such as extensions, repairs, and alternative explorations for future seed selection.Each graph node contains a harness snapshot, achieved reward, and evolution-step metadata.
  • History retrieval: A probe retrieves compact summaries of stored trajectories instead of exposing all historical interaction traces to the LLM.Queries can target successful seed-selection behaviors, regression-prone patterns, and hypothesis structures preceding large improvements.
  • Scope boundaries: The task harness can edit H but not evolution strategy Σ, while the meta-evolver can edit Σ but not directly alter H during meta-evolution.Unauthorized modifications are rejected, and the meta-evolver’s execution logic remains loaded from an immutable initialization template.

A.6 Evaluation Interface

Evaluation occurs outside editable harness surfaces through an invariant task interface, enabling identical protocol-based comparisons across evolved versions. The evaluator supplies both scalar rewards for selection and optional textual feedback for subsequent evolution.

  • Evaluation interface: Tasks enter the harness through the fixed interface using_harness(agent, task).The interface remains invariant even as the harness’s internal implementation changes.
  • Evaluation interface: The invariant interface allows evolved harness versions to be compared under identical development, validation, and test protocols.Evaluation is performed outside the editable surfaces.
  • Evaluation signals: The evaluator returns a scalar reward for candidate comparison and optional textual feedback to guide later evolution.The two outputs provide quantitative selection information and qualitative guidance, respectively.

B.1 Per-Suite Experimental Configuration

BALROG suites share a controlled HSI configuration, while suite-specific protocol, split, episode, repetition, meta-evolution, and submission settings are varied in Table 3. The initial harness is not pre-evaluated, so iteration 1 begins cold.

  • Shared configuration: Shared settings use DeepSeek-V4-Flash, five outer iterations, up to 80 react() steps, LCB reward at z = 0.5, and maximum reasoning for the evolver.The task harness uses thinking disabled and temperature 0; the meta-evolver has up to 50 react() steps and evolvable seed-selection and commit pooling.
  • Selection stages: The terminal best-version selection stage runs once after every evolution and is fixed rather than evolvable.A short seed-validation probe of up to three evaluate() calls is enabled during seed selection.
  • Initialization: The initial harness is not pre-evaluated, so the first evolution iteration starts cold.This condition is part of the shared experimental configuration.
  • Suite-specific configuration: Table 3 varies setup protocol, dev and validation ratios, test and development episodes, test repeats, meta-evolver availability, and terminal submit-best steps across suites.Setup distinguishes full-set in-distribution evolution from sub-suite splits with held-out testing.

B.2 Detailed Survey of Harness Engineering and Agent Self-Evolution Methods

The survey covers harness engineering and agent self-evolution methods spanning recursive self-editing, population and group evolution, harness search, observability, and harness-model co-evolution. These approaches differ in their evolutionary unit, search mechanism, memory or observability design, and degree of self-modification.

  • Recursive self-improvement: Gödel Agent, SICA, and related systems use recursive or self-referential code modification to improve an agent through evaluation, editing, and verification loops.SICA promotes the best agent to meta-agent before proposing, implementing, and independently verifying changes.
  • Population and clade evolution: DGM and HGM broaden self-improvement with archive-based population exploration and clade-level productivity estimates for selecting expansions.HGM uses Thompson sampling over Beta distributions of clade success rates.
  • Group evolution: GEA changes the evolutionary unit from individuals to groups by sharing parents’ traces and using reflection to generate evolution directives.At each iteration, two parents are selected using Performance-Novelty scoring.
  • Editable meta-mechanisms and harnesses: HyperAgents makes the meta-mechanism editable, while Recursive Agent Harnesses evolve the full harness and report gains with fixed backbones.HyperAgents fuses task and meta agents into one editable program; RAH uses filesystem tools, code execution, and planning as the recursive unit.
  • Harness search and synthesis: Meta-Harness, AutoHarness, and related work formulate harness improvement as code-space search or synthesis guided by proposer models and sampling.AutoHarness explores verifier and policy harness templates through Thompson-sampling-guided tree search.
  • Design and observability: Other studies emphasize architectural dimensions, observability, modular harness evolution, and harness-model co-evolution as design axes.AHE separates component, experience, and decision observability, while HarnessX uses typed processors, an operational mirror, and cross-harness GRPO.

B.2.5 Evaluation, Attribution, and Theoretical Foundations

The section situates HSI within empirical and theoretical efforts to evaluate harness evolution, emphasizing trustworthy attribution, diagnostic coverage, and statistical limits on self-modification.

  • Empirical evaluation: Harness-evolution evaluations compare updating against benefit, revealing that performance gains depend on model capability and task activation.Reported findings include non-monotonic benefit across model tiers and differing skill-load and adherence rates.
  • Empirical evaluation: GSME proposes validity, activation, and significance gates to distinguish real harness improvements from infrastructure failures, inactive patches, and unsupported gains.Its archive is organized by defect location and cause rather than fixed tasks, and sealed-test gains are reported across cases.
  • Empirical evaluation: Harness-Bench treats the harness as the primary evaluation axis across sandboxed tasks, workflow categories, harnesses, and model backends.It reports a 23.8-point gap between the best and worst harness and identifies recurring failure modes linked to intervention points.
  • Empirical evaluation: SEAGym evaluates self-evolution with update-validation, transfer, replay, and cost views, showing that batch size and source diversity affect evolutionary behavior.Useful intermediate states may collapse and recover later, so final scores alone can miss important trajectories.
  • Theoretical foundations: Statistical theory bounds reliable self-improvement: distribution-free PAC guarantees are preserved if and only if the policy-reachable hypothesis family has uniformly bounded VC dimension.The framework also identifies a utility-learning tension, where performance-driven edits can undermine generalization conditions.
  • Comparative summary: Table 4 compares representative methods with HSI by proposer, editable code surface, evaluation domain, and distinguishing feature.These dimensions organize how methods differ in who proposes edits and what part of the agent is changed.
Loading 2608.08466v1…