PaperlayerView exact revision on arXiv

Source-linked AI summary

HELIX: Model-Harness Co-evolution for Recursive Self-Improvement

Tianyu Fan, Chao Huang

arXiv:2608.13951v1cs.AI

TL;DR

Interactive agents depend on runtime harnesses as well as models, but recursive improvement requires preserving intervention identity and verified evidence across harness changes. HELIX provides a source-traceable build–update–rebuild substrate and, in one code-repair round, finds stronger execution while producing structured sibling data for future model updates.

  • Problem

    Interactive-agent capability depends on a runtime harness, yet recursive improvement needs controlled harness evolution that preserves intervention identity and verified evidence.

  • Method

    HELIX decomposes harnesses into typed ports, atoms, recipes, product shells, and policies, preserving declared interventions and evidence across execution and model–harness co-evolution.

  • Results

    A 65-candidate portfolio exposes up to 58.0% more post-hoc portfolio coverage than Pi and yields 438 structured records from 200 curated sibling slots.

  • Takeaways & Limitations

    Harness evolution improves current-model execution while providing verified sibling data for model updating and subsequent harness rebuilding.

  • Takeaways & Limitations

    The paper materializes model-update data but does not train an updated model; larger-scale data and reliable recursive updating remain necessary for multi-round evaluation.

Abstract

from arXiv · show

Scaling agent capability has largely focused on improving the model, yet an interactive agent acts through a runtime harness that mediates context, tools, control flow, and stopping. The harness shapes both what a model can accomplish and the trajectories from which it learns. This coupling motivates model-harness co-evolution for recursive self-improvement: build harnesses for a fixed model, update the model from verified sibling trajectories, and rebuild the harnesses as model capabilities change. Realizing this loop requires a controlled way to evolve harnesses while preserving intervention identity and effect. We present HELIX, a source-traceable substrate for harness evolution. HELIX decomposes agent systems into typed ports, reusable atoms, recipes, product shells, and runtime policies. It makes interventions explicit and auditable while retaining trajectories, test outcomes, and provenance. Harness evolution thus serves two linked roles: improving fixed-model execution and producing matched successes, regressions, near misses, and alternative solutions as data for subsequent model improvement. We evaluate HELIX in one evolution round on code repair. A 65-candidate portfolio discovers a fixed harness that improves task coverage by 4.0% over Pi, while the full portfolio exposes up to 58.0% more verified coverage through complementary sibling behavior. Selected candidates are assessed with repeated runs and the SWE-bench evaluator. A 200-slot sibling slice yields 438 verified SFT, critic, filter, and preference records. These results show how harness, model, and data form a feedback system: harness evolution expands current capability and creates learning signal for the next model; model updates motivate the next round of harness evolution. HELIX provides an auditable interface for studying this recursive process. Code is available at https://github.com/HKUDS/HELIX.

1 Introduction

HELIX treats recursive improvement as co-evolution of a model and its runtime harness, using explicit, auditable interventions to improve current execution and generate verified data for later model updates.

  • The harness co-determines agent behavior by mediating observations, actions, recovery, permissions, and stopping decisions.
  • A harness-evolution round produces stronger fixed-model execution and matched successes, regressions, near misses, and alternative solutions for model updating.
  • HELIX organizes co-evolution as build, update, and rebuild: construct harnesses for a fixed model, learn from verified sibling trajectories, then evolve the harness for the updated model.
  • HELIX makes harness interventions explicit and auditable through typed ports, atoms, recipes, product shells, runtime policies, pre-execution checks, and evidence capture.
  • 65 candidates expose up to 58.0% more post-hoc portfolio coverage than Pi, while selected sibling artifacts yield 438 SFT, critic, filter, and preference records.
  • The system state couples the current model with its harness portfolio, while verified trajectories and evaluator evidence connect task execution to later updates and rebuilding.

3 HELIX System

HELIX separates current execution quality from the broader potential of a candidate portfolio, while preserving evidence for both outputs.

  • Oracle coverage measures the potential positive-outcome yield, or routing ceiling, of an observed candidate portfolio.It is deployable only when a router can select the successful candidate before seeing the answer.
  • Fixed coverage, successful attempt slots, and oracle coverage are reported as separate quantities.
  • HELIX preserves the evidence needed to build explicit harness interventions and support both fixed execution and portfolio-yield analyses.

3 HELIX: Explicit and Auditable Harness Evolution

HELIX operationalizes harness evolution by representing runtimes as explicit, source-traceable compositions whose identities and evidence persist through assembly and execution.

  • HELIX turns a monolithic runtime into source-traceable interventions that remain identifiable from declaration through compilation, execution, and evidence capture.
  • The harness representation covers eight dimensions spanning model-facing interfaces and environment-facing control policies.These include the shell, session and hooks, configuration, prompt, tools, turn loop, continuation, and stopping behavior.
  • Ports, atoms, packs, recipes, and product shells preserve intervention identity while supporting reusable and source-specific behavior.The compiler and lockfile retain the full expansion used by each rollout.
  • Each full product contract exposes 96 ports, bindings, and swap points, alongside a neutral Minimal contract for compact runtime and conformance checks.
  • The compiler resolves dependencies and bindings into a deterministic lockfile, while conformance, boundary, and source-purity checks reject invalid or ambiguous compositions before evaluation.
  • The system scale comprises 631 unique product atom IDs, 1,156 product-contract occurrences, and 1,332 occurrences including the neutral Minimal contract.
  • HELIX enumerates finite intervention spaces and uses deterministic smoke screening to select candidates for live evaluation.The live experiment evaluates a structured 65-candidate slice.
  • The runtime loop restores sessions, builds context, streams provider steps, executes checked tool calls, evaluates acceptance, manages continuation and stopping, and persists traces and workspace effects.

4 Design Benefits

HELIX’s runtime mediates each model action and records the resulting evidence, making interactions reproducible and intervention-specific.

  • Tool preflight, permission checks, execution batching, result hooks, truncation, and retries regulate each proposed tool interaction.
  • Acceptance evidence and provider finish reasons determine whether a trajectory continues or stops.
  • HELIX persists transcripts, session state, runtime traces, and workspace effects for evaluation.
  • Context compaction, tool normalization, permissions, continuation cadence, and assistant-part protocols can alter trajectories without changing model weights.Model actions also determine which harness transitions, tools, and stopping checks are reached next.
  • Each run is linked to its recipe, lockfile, model–task–attempt identity, runtime trace, workspace effects, verifier outcome, and policy evidence.
  • Persistent intervention identity supports fixed-runtime comparison and matched sibling-data construction.

4 Design Capabilities for Auditable Recursive Improvement

HELIX makes harness recipes the bounded unit of evolution, preserving interpretable interventions and turning matched outcomes into structured data for recursive improvement.

  • HELIX provides bounded modification of runtime behavior, attributable observation of its effects, and mechanisms for repeating the improvement cycle.
  • Recipes replace whole-runtime editing as the unit of evolution, while source-derived atoms preserve behavior behind stable ports.
  • Candidates select known behavior families at declared capability slots rather than anonymous code fragments or duplicated runtimes.
  • Reusing contracts while revising recipes after model updates makes build and rebuild repeatable component-level operations.
  • Expanded recipes, lockfiles, execution conditions, and evidence connect gains or regressions to inspectable component sets.
  • Matched sibling groups preserve intervention identity, workspace effects, and evaluation evidence across runs for the same model and task.
  • Sibling contrasts distinguish clean resolutions, regressions, near misses, no-action runs, and alternative solutions for model-update objectives.The portfolio elicits multiple behaviors on matched tasks and thereby acts as a curriculum operator for the current model.

5 Experimental Design

The evaluation tests a complete 65-candidate harness portfolio on trace-strict code repair, then deepens selected-member validation for reliability, transfer, and learning-data quality.

  • Data integrity: Persistent task and sibling identity keeps related trajectories within one train/dev split, while evidence-linked siblings connect execution outputs to repeatable co-evolution.The design preserves attribution across build, update, and rebuild stages.
  • Research questions: RQ1 tests whether evolution finds a stronger fixed runtime, RQ2 measures portfolio breadth and sibling diversity, and RQ3 assesses verified learning data.The questions separate fixed-harness execution, whole-portfolio behavior, and model-update records.
  • Portfolio and benchmarks: The main evaluation uses 100 LiveCodeBench-derived fixtures spanning 26 easy, 24 medium, and 50 hard tasks.Success requires an observed normalized bash call running test_solution.py with python or python3 and corresponding passing output.
  • Portfolio and benchmarks: The complete matrix evaluates Pi and 64 Pi-centered source-pure recipes once on each of 100 fixtures, producing 6,500 slots.The recipes vary session/hooks, config, and tools while retaining Pi’s prompt, turn loop, and acceptance behavior.
  • Follow-up validation: Selected members receive ten attempts per task in repeated LiveCodeBench follow-up, producing 3,000 slots and measuring coverage plus successful-slot reliability.The follow-up selects Pi and two OpenCode-family members for deeper mechanism analysis rather than simply choosing the one-attempt leaders.
  • Follow-up validation: A cross-benchmark follow-up samples 55 SWE-bench instances, gives selected members two attempts each, and uses the official evaluator across 550 slots.Runs allow up to 200 agent steps with local concurrency one; task resolution is labeled by the official SWE-bench evaluator.

6 Results: Execution Today and Learning Data Tomorrow

Across one evolution portfolio, HELIX finds stronger fixed harnesses, broader complementary coverage, and verified sibling distinctions for model-update data, while follow-ups expose protocol dependence and causal limits.

  • Execution today: 52/100 tasks is the best fixed-candidate result versus Pi’s 50/100, a 4.0% relative improvement, but only two of 64 non-baseline candidates exceed Pi.The winner uses Hermes session/hooks with Pi’s remaining components; the runner-up combines Hermes session/hooks and config with Nanobot tools and the Pi core.
  • Execution today: 75/100 tasks versus 72/100 for Pi is the repeated-run coverage of each selected OpenCode-family member, with 549 and 546 successful slots versus Pi’s 529.Coverage counts tasks succeeding at least once; successful slots measure reliability across attempts.
  • Execution today: 46/55 SWE-bench instances versus 44/55 for Pi is the strongest selected-member result, showing that the preferred fixed harness changes with evaluation protocol.The two one-attempt LiveCodeBench leaders match Pi at 44/55, while the selected OpenCode-family members resolve 46/55.
  • Execution breadth: 79/100 tasks is the complete portfolio’s post-hoc union coverage versus Pi’s 50/100, a 58.0% relative gain that is not a deployable fixed-harness score.The portfolio records 1,014 trace-strict successes across 6,500 slots and exposes complementary sibling behavior.
  • Execution breadth: Only two of 64 non-baseline candidates exceed Pi while 23 solve no task, making arbitrary composition unreliable and motivating selection mechanisms.The paper identifies typed gates, cheap screening, adaptive selection, and eventually a learned router as ways to exploit complementarity without evaluating every candidate.
  • Execution breadth: 49/55 instances is the selected-member SWE post-hoc union versus 46/55 for its best fixed member and 44/55 for Pi, supporting complementary coverage rather than online routing.The union is also treated as potential positive-data yield, not as a deployable score.
  • Learning data tomorrow: A target-passing patch that breaks regression tests is labeled a gap, whereas a sibling passing targets and all 123 PASS_TO_PASS tests is resolved.The Pytest case shows why target-only evaluation would assign the wrong label.

7 Discussion

HELIX turns sibling contrasts into structured model-update records, extending harness evolution beyond selecting a better runtime. A 200-slot selected-member slice yields 438 records spanning multiple training purposes.

  • Figure 4 maps failed, noisy, and alternative-success siblings into critic, filter, and preference categories beyond clean SFT.Verifier and patch-quality labels support the multi-purpose audit.
  • Sibling contrasts become model-update records rather than merely post-hoc interpretations.The records preserve outcome distinctions needed for different update objectives.
  • 438 structured records map 200 curated sibling slots into SFT, critic, filter, and preference data.Successful, failed, noisy, and alternative trajectories are retained as distinct training signals.
  • The evolution round produces both expanded current-model execution and verified sibling data for the next update.These outputs are connected through the build–update–rebuild process.

7 Discussion: From One Evolution Round to Recursive Self-Improvement

HELIX frames recursive improvement as repeated transitions of a model–harness state, with harnesses shaping trajectories and verified experience shaping later models. Portfolio evaluation separates execution quality from data-generation and routing opportunities.

  • From One Evolution Round to Recursive Self-Improvement: The recursive state is Z_t = (M_t, Φ_t), returning to the same improvement process after model and harness changes.This recursion is across build–update–rebuild rounds, not repeated calls within one execution.
  • From One Evolution Round to Recursive Self-Improvement: Verified trajectories shape the next model, while the updated capability profile changes which harness is preferred.The dependencies are ϕ_t → τ_t, τ_t → θ_t+1, and θ_t+1 → ϕ_t+1.
  • Portfolio coverage exposes execution headroom and data yield: The 52/100 fixed candidate and 79/100 portfolio union reveal separate execution and portfolio frontiers.Deeper validation can change which portfolio member is preferred.
  • Portfolio coverage exposes execution headroom and data yield: A fixed harness is optimized for accurate, reliable, safe, affordable execution, whereas a portfolio is optimized for novel positives, failures, alternatives, and contrasts.The best deployed harness need not be the best exploration policy.
  • Portfolio coverage exposes execution headroom and data yield: The gap between the best fixed candidate and an oracle union bounds potential routing improvement, but no selector is evaluated.Realizing that gap requires a pre-outcome signal such as task features, trace features, a critic, or staged policy.
  • Portfolio coverage exposes execution headroom and data yield: The 65-candidate matrix contains 23 zero-score candidates whose uses include pruning incompatible regions or generating compatibility negatives.Poor-quality runs should be routed, weighted, or excluded using retained harness identity, verifier rationale, outcome vectors, and failure classes.

8 Related Work

HELIX connects prior work on recursive self-improvement, runtime interfaces, automated agent-system evolution, and interaction data through a source-traceable build-to-update handoff. Its distinctive focus is preserving intervention identity and verified sibling evidence across heterogeneous harnesses.

  • Prior recursive self-improvement spans bounded refinement, model-policy changes, evaluator changes, and self-referential code rewriting.Darwin Gödel Machine uses empirical variation, benchmark selection, and an archive of agent-code variants.
  • Recent harness methods adapt prompts or jointly optimize harnesses and models, while HELIX targets recomposition across heterogeneous harness families.This is presented as a complementary systems layer.
  • ReAct, Toolformer, and SWE-agent establish that agent behavior depends on interfaces, actions, and tool use beyond static next-token prediction.HELIX extends the intervention surface toward the complete harness.
  • Automated workflow and agent-design systems evolve natural-language workflows, code-defined agents, or search-based workflow representations.These systems primarily report the quality of discovered workflows or agent programs.
  • HELIX connects these strands through a source-traceable handoff from declared interventions to execution evidence and matched sibling records for model updating.Non-winning candidates can contribute regression, near-miss, artifact-quality, or preference examples.

9 Limitations and Threats to Validity

The evaluation is limited in task scope, attribution, comparability, automated verification, and loop closure. HELIX materializes update data but does not yet train an updated model, and safe deployment requires isolation and renewed integration checks.

  • Scope of the evaluation: The evaluation uses 100 nonrandom AtCoder tasks with public tests, 55 selected-member SWE tasks, and coding agents only.It does not measure hidden-test or official LiveCodeBench performance, limiting generalization.
  • What causes an improvement: Each evolved harness changes several components simultaneously, so individual component causes cannot be identified.The paper recommends one-component-at-a-time experiments with other components fixed.
  • Coverage comparisons use different attempt counts and selected members, so matrix, LCB, and SWE rankings are not directly comparable.Oracle coverage is computed after observing outcomes, and deeper evaluation may introduce selection effects.
  • Limits of automated evaluation: Tests miss some requirements: passing patches may alter tests or exploit evaluators, while correct patches may fail because of timeouts or broken environments.Some cases still require human review.
  • The paper materializes model-update data but does not train an updated model.Scaling the sibling dataset, repeated rollouts, token budgets, and suitable training methods remain necessary to evaluate loop closure.
  • Unsafe commands, file changes, secret leakage, and permission conflicts require isolated workspaces, restricted access, trace redaction, and renewed safety testing.Upstream harness changes require integrations and safety checks to be tested again.

A.1 Repeated LCB evaluation

The paired verifier-grounded traces show that local target-test success and patch cleanliness can diverge from official resolution because compatibility and fallback semantics matter. Tables 6 and 7 isolate how these differences change evaluator labels.

  • Pytest case: Both Pytest patches pass the two target tests, but only the evolved sibling preserves the pre-existing serialization schema.The Pi patch replaces the longrepr schema, whereas the sibling extends it.
  • Pytest case: Official regression failures directly query a field removed by the Pi patch, creating a resolved-versus-regression supervision pair.The target tests pass despite the backward-compatibility regression.
  • Xarray case: Both Xarray patches are clean single-file edits that pass ad hoc checks, yet their official labels diverge on fallback semantics.The paired setup holds the session, prompt, turn, tools, and attempt index fixed while changing the configuration atom.
  • Xarray case: The Xarray negative preserves regression tests but uses the wrong fallback behavior, showing why patch cleanliness alone is insufficient as a training label.The resolved expression returns an empty mapping when fewer than two attributes are present.
Loading 2608.13951v1…