PaperlayerView exact revision on arXiv

Source-linked AI summary

RSIHub: A Composable Framework for Agent-Evolution Methods

Zimu Wang, Xiaobo Wang, Siyu Ye, Jiaqi Li, Silin Chen, Yuling Shi, Ruobing Wang, Jingjing Zhang, Chaofan Wang, Kai Wu, Mengnan Qi, Kai Cai, Xiaodong Gu, Zilong Zheng

arXiv:2610.32253v1cs.AI

TL;DR

Agent-evolution implementations often rebuild shared experimental machinery, making method implementation and comparison costly. RSIHub provides composable recipes and shared execution, evaluation, and provenance services; evaluations show gains and held-out transfer vary across settings.

  • Problem

    Agent-evolution implementations commonly bundle recurring execution, evaluation, recovery, and lineage infrastructure with method policy, so implementing and comparing methods can require rebuilding or reconciling end-to-end systems.

    Source: §1 · p. 2

  • Method

    RSIHub separates policy from shared framework services through recipes that compose operators and declare mutable resources, while framework-managed execution and evaluation retain candidate outcomes.

    Source: §1 · pp. 2–3

  • Results

    Gains and held-out transfer vary across settings, and recipe rankings vary across agents, benchmarks, and splits.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    RSIHub provides shared protocols, inspectable candidate histories, and reusable components for comparing evolution policies and developing new methods.

    Source: §5 · p. 10

    Evidence depends on trusted framework and host protections and applies to tested workloads and lifecycle boundaries; reported results characterize fixed recipes, not adaptive outer-agent revision.

    Source: §6 · p. 10

Abstract

from arXiv · show

Agent-evolution methods use execution feedback to improve prompts, skills, tools, and orchestration, but their implementations often rebuild the same experimental machinery. We present RSIHub, an open-source framework for building, comparing, and extending these methods on shared infrastructure. Recipes compose reusable operators and declare mutable resources; RSIHub supplies execution, evaluation, recovery, and lineage. Researchers can focus on evolution policy while evaluation stays outside the mutable surface and results remain tied to candidates and evaluation conditions. We instantiate AHE, Hyperagents, A-Evolve, and GEPA without modifying the framework core, keeping method-specific logic in recipes and operators. We evaluate them with MiniSWE and Codex on Terminal Bench 2 and $τ^3$ Banking under fixed model and evaluator configurations. The largest observed overall pass@1 gain is 13.5 percentage points, while method rankings vary across settings and optimization gains do not always transfer to held-out tasks. RSIHub provides a common protocol for comparing evolution policies, inspectable candidate histories, and reusable components for developing new methods.

1 Introduction

Agent-evolution methods repeatedly rebuild shared experimental infrastructure, complicating implementation and comparison. RSIHub separates method policy from execution and evaluation services, and the paper compares four recipes under fixed configurations.

1 Introduction

  • RSIHub addresses repeated infrastructure across agent-evolution methods by providing shared execution, evaluation, recovery, and lineage services.Implementations often bundle this infrastructure with method policy, so implementing or comparing methods can require rebuilding or reconciling end-to-end systems.Source: §1 · pp. 2–3
  • Its recipes bind reusable operators and declare mutable resources, while framework-managed services run and evaluate candidates and retain their outcomes.The framework keeps evaluation and its archive outside the declared mutable surface, associating results with the committed candidate and evaluation conditions.Source: §1 · pp. 2–3
  • The paper expresses AHE, Hyperagents, AEvolve, and GEPA as recipes over shared operators and provides a common lifecycle for execution, recovery, artifacts, evaluation records, and lineage.The released resources include runnable recipes, reusable operators, documentation, and an experiment viewer.Source: §1 · p. 3
  • The evaluation compares four recipes across two seed agents and two benchmarks under fixed model and evaluator configurations, with recipe rankings varying across agents, benchmarks, and splits.It also examines effectiveness, held-out transfer, execution fidelity, integrity, runtime, and implementation cost.Source: §1 · p. 3

2 Related Work

Related work spans agent improvement, benchmark and execution infrastructure, and provenance and measurement validity. RSIHub focuses on shared evolution lifecycle and evidence services for comparing selected policy adaptations.

2 Related Work

  • Prior approaches span feedback-based improvement, persistent artifact optimization, and self-modification; RSIHub adapts selected policies from AHE [14], GEPA, Hyperagents [41], and A-Evolve [15].Related methods also optimize prompts, skills, agent graphs, and multi-agent workflows [39].Source: §2 · p. 3
  • Harbor and the OpenHands Software Agent SDK provide execution and composable agent infrastructure, while RSIHub adds a versioned evolution lifecycle, recipe-level composition, and shared provenance.RSIHub delegates container execution, trajectory capture, and per-task verification to Harbor.Source: §2 · pp. 3–4
  • RSIHub records candidate lineage and binds outcomes to candidates, recipes, and evaluation conditions, supporting attribution under its framework trust assumptions.The related work distinguishes measured gains, retention, and held-out transfer, and notes that benchmark overfitting and specification gaming remain evaluator- and study-level concerns.Source: §2 · p. 4

3 The RSIHub Framework

RSIHub separates method policy from shared execution, evaluation, and record keeping. Its interchangeable recipe components, guarded candidate lifecycle, and provenance model provide common mechanics for studying evolution methods.

3.1 Design principles

  • RSIHub fixes lifecycle slots while leaving their implementations open: recipes name and configure operators for stages such as selection or mutation.A stage is a lifecycle slot; an operator is one implementation of that stage.Source: §3.1 · p. 4, §3 · p. 4
  • Recipes declare both the resources candidates may change and the mutation agent’s configuration and retained context, separating evolved resources from revision inputs.The same operator interfaces can support prompt-only or whole-harness studies.Source: §3.1 · p. 4
  • RSIHub reuses workspace construction, snapshots, evaluation, artifacts, terminal-state classification, recovery, and lineage across recipes, while keeping measurement outside method policy.Changing one operator while holding other recipe fields fixed isolates a declared policy component; parent eligibility remains a separate downstream choice.Source: §3.1 · p. 4

3.2 Composable candidate lifecycle

  • The framework snapshots and validates each recipe, runs operators in separate processes, and validates their structured outputs; experiments in this paper evaluate fixed recipes.An outer agent may revise bindings and settings between iterations, but the target, evaluator, runtime, data splits, and mutable surface remain fixed, and sealed tasks are excluded from adaptation feedback.Source: §3.2 · p. 5
  • The candidate lifecycle runs Select, Rollout, Analyze, Mutate, Guardrail, Evaluate, and Absorb, with optional Analyze, Validate, Novelty, and Reflect operators.Mutate proposes edits within the declared surface; framework guardrails check the tree diff, and protected evaluation certifies the committed candidate.Source: §3.2 · pp. 4–5
  • Guardrails, validation, and novelty checks can close proposals before evaluation; only benchmark-complete evaluations proceed to Gate, while failures and skipped operations are recorded through defined paths.The proposal remains uncommitted through Guardrail, and the committed tree is evaluated from a clean checkout; Gate runs only when trial outcomes satisfy the configured coverage policy.Source: §3.2 · pp. 5–6

3.3 Provenance

  • Certified evaluation events bind the committed candidate and parent to the recipe snapshot, evaluation conditions, outcomes, costs, artifacts, and lineage.This record distinguishes changes in method policy from changes in evaluation conditions.Source: §3.3 · p. 6
  • Researchers can inspect candidate snapshots, lineage, task outcomes, costs, and trajectories, then test policy components while reusing framework services.The four evaluated adaptations used 47–61 recipe lines and two operator bindings each without framework-source changes; method-specific operator code ranged from 216 to 1,087 lines.Source: §3.3 · p. 6

4 Evaluation

The evaluation compares evolution recipes across agents, benchmarks, and held-out splits, alongside controls for execution fidelity and integrity. Results show substantial but setting-dependent gains, with transfer and framework behavior assessed under tested conditions.

4 Evaluation; 4.2 Setup

  • The evaluation crosses agents, benchmarks, and held-out splits for effectiveness, uses generation-zero controls against direct execution for fidelity, and tests fault recovery and boundary enforcement for integrity.The experiments use fixed model and evaluator configurations.Source: §4 · p. 6, §4.2 · p. 7

4.1 Recipe instantiations; 4.2 Setup

  • The evaluation adapts A-Evolve, AHE, GEPA, and Hyperagents as recipes over shared operators, with exact source-method correspondence also depending on prompts, models, budgets, dependencies, and runtime adapters.The recipes are tested across MiniSWE and Codex using the same frozen evaluator.Source: §4.1 · p. 7, §4.2 · p. 7

4.3 Evolution effectiveness and transfer

  • On Terminal Bench 2, every method improves the MiniSWE baseline by 7.8–13.5 percentage points; Hyperagents and A-Evolve reach the highest overall score of 69.7.The table reports pass@1 before and after evolution across Terminal Bench 2 and τ 3 Banking.Source: §4.3 · p. 7, Table 1 · p. 8
  • A reported score increases from 24.7 to 34.0, while GEPA is close behind at 33.0; these shifts indicate that rankings depend on both task environment and starting agent.Source: §4.3 · p. 8
  • Split results reveal held-out tradeoffs: on Terminal Bench 2, only Hyperagents improves MiniSWE’s gate split, while several methods lose gate or sealed performance in other settings.On τ 3 Banking with Codex, AHE improves train, gate, and sealed scores by 1.0, 3.0, and 5.1 points; GEPA preserves train and improves gate and sealed by 4.1 points.Source: §4.3 · p. 8

4.4 Execution fidelity

  • Capability controls report Terminal Bench 2 changes of −2.6 points for both agents; direct executions themselves disagree on 12.0–18.9% of tasks.The controls characterize execution variability but do not establish execution-mode equivalence or statistical significance of single-run evolution gains.Source: §4.4 · p. 9, Table 2 · p. 9

4.5 Recovery and integrity

  • Across two benchmark campaigns, all 420 integrity trials reached the required outcome without admitting corrupted state, within the tested lifecycle boundaries and attack cases.Latency results are reported separately because the campaigns ran on different hosts.Source: §4.5 · pp. 9–10, Table 3 · p. 9

5 Conclusion

RSIHub offers reusable infrastructure for agent evolution and supports adaptive recipe revision while keeping evaluation records protected. Its results vary across settings, and held-out transfer is not consistent.

5 Conclusion

  • Gains and held-out transfer vary across settings, motivating shared protocols and inspectable evidence for comparing policies.RSIHub invites researchers to build on released recipes and contribute reusable operators.Source: §5 · p. 10
  • RSIHub provides open, reusable infrastructure for agent evolution.All four evaluated adaptations reuse framework services without framework source changes.Source: §5 · p. 10
  • An outer agent may revise operator bindings and settings between iterations while the evaluator, sealed split, and certified record remain protected.These interfaces provide a shared foundation for studying agent self-improvement.Source: §5 · p. 10

6 Limitations

RSIHub’s evidence guarantees depend on trusted services and valid evaluators, and the study’s findings have defined scope. Its capability-preservation measurements assess task-level agreement, not full behavioral equivalence.

6 Limitations

  • RSIHub’s evidence guarantees rely on trusted framework services, host and filesystem protections, Git and digest operations, configured external services, and valid evaluators.Runtime fingerprints do not replace host or service attestation, and hostile code is not contained; stronger deployments require independent attestation, isolation, and containment.Source: §6 · p. 10
  • The results apply to the workloads, hosts, and lifecycle boundaries tested in this study.Capability-preservation measurements assess task-level agreement rather than full behavioral equivalence.Source: §6 · p. 10
  • Reported results characterize fixed, recipe-driven runs rather than adaptive recipe revision by an outer agent.Source: §6 · p. 10

7 Responsible Use Statement

RSIHub can create safety and privacy risks and does not contain host-executed editable components. The paper recommends access restrictions, held-out safety evaluation, human review, and controls for resource use and records.

7 Responsible Use Statement

  • RSIHub can enable unsafe capability growth, reward gaming, or exposure of sensitive traces; higher pass@1 does not establish safe deployment.Banking-benchmark success does not establish suitability for financial decisions, and RSIHub supports auditing and rollback rather than containment.Source: §7 · p. 10
  • Deployers should restrict filesystem and network access, exclude production credentials, and require held-out safety evaluation and human review before promotion.Resource budgets, halt and rollback controls, and access controls, retention limits, and redaction for shared trajectories and archives are also recommended; rollback cannot undo external actions or disclosures.Source: §7 · p. 10

A Representative Recipe Configuration

  • The representative GEPA-inspired recipe configures Codex and identifies the target prompt and task-execution skill among its components.The listing notes that low-level execution settings are omitted for brevity.Source: A Representative Recipe Configuration · p. 15

B Extended Evaluation

Direct replicates differ in which execution mode more closely agrees with RSIHub across benchmarks, so neither is uniformly closer and the section motivates averaging the comparisons.

B.1 Pairwise Execution Agreement

  • Table 4 reports pairwise cross-mode task agreement over three repetitions, with mean D–R agreement defined as the arithmetic mean of the two pairwise agreements.All entries are percentages, and mean D–R agreement is the value reported in Table 2.Source: Table 4 · p. 16
  • On Terminal Bench 2, Direct-B–RSIHub agreement is 0.7 points higher for MiniSWE, while Direct-A–RSIHub agreement is 0.8 points higher for Codex.Source: §B.1 · p. 16
  • On τ 3 Banking, Direct-A is 2.4 points higher for MiniSWE, while Direct-B is 1.4 points higher for Codex.Source: §B.1 · p. 16
  • Neither direct replicate is uniformly closer to framework-mediated execution, motivating an average of the two comparisons.Source: §B.1 · p. 16

B.2 Runtime Overhead

Framework services account for a small share of complete generations in both measured workloads. The largest individual measured operation differs by campaign, and absolute latencies should not be compared across the different hosts.

B.2 Runtime Overhead

  • Terminal Bench 2 initialization used 50 microbenchmark repetitions, recovery used 30 replay trials, and regular-operation and generation-share estimates used 20 complete Gen-1 runs.Regular-operation medians and P95s were computed across per-generation totals; the generation share uses the union of framework-exclusive spans.Source: §B.2 · p. 16
  • 0.019% on τ 3 Banking and a median 0.02% on Terminal Bench 2: framework work remained small relative to complete generations.Model, mutation, and benchmark execution dominate runtime in both measured workloads.Source: §B.2 · p. 16, Table 5 · p. 16
  • 1.23 seconds was the median workspace-initialization time on τ 3 Banking; snapshot creation and state verification led regular operations on Terminal Bench 2 at 0.76 and 0.65 seconds.These absolute latencies should not be compared directly because campaigns ran on different hosts.Source: §B.2 · p. 16

B.3 Method-Integration Effort

The integrations reuse RSIHub’s shared interfaces and services while keeping method-specific logic in recipes and operators. The execution record and operator contracts expose candidate histories, evaluation conditions, and lifecycle behavior.

B.3 Method-Integration Effort

  • 47–61 lines of recipe configuration and two reused operator bindings per adaptation required no framework-source changes; method-specific operators ranged from 216 lines for Hyperagents to 1,087 for AHE.Each adaptation passed 14–47 method-specific integration tests.Source: §B.3 · p. 17

C Inspecting an Execution

  • The read-only viewer presents experiment status, generation lineage, and the champion’s cumulative target-tree changes; the illustrated accepted Generation 8 candidate is labeled champion.Its displayed scores are run-specific, while aggregate performance is reported separately.Source: C Inspecting an Execution · p. 17, Figure 2 · p. 17

D Executable Contracts and Recovery Semantics

  • Typed operator outputs are validated before use, while the framework enforces the mutable surface, commits candidates, runs protected evaluation, and protects framework record fields.Table 7 summarizes nine lifecycle operator interfaces in execution order.Source: D Executable Contracts and Recovery… · p. 18, Table 7 · p. 18

D.2 Evaluation identity and parent selection

Strict evaluation contracts bind scores to candidate and evaluation conditions, while parent selection applies a fixed-identity projection alongside integrity and lifecycle checks. Recovery preserves certified evaluations before Gate decisions are recorded.

D.2 Evaluation identity and parent selection

  • In strict-contract mode, the evaluation contract binds the exact candidate to evaluator, data, sampling, runtime, and model conditions, with contract_id as the complete payload’s canonical digest.Parent selection combines the listed fixed-identity projection with integrity and lifecycle checks; the full contract remains attached to each attempt for provenance.Source: §D.2 · p. 18, Table 8 · p. 18
  • Sealed evaluation is stored as auxiliary evidence and excluded from the default mutation-feedback bundle and candidate-or-genesis parent predicate.It runs as a separate, non-selectable anchor.Source: §D.2 · p. 19

D.3 Recovery checkpoints

  • A complete certified candidate evaluation is appended to the archive before Gate and remains pending until Gate records its decision, making the archive a recovery boundary.A pending child is unavailable to Select; Record annotations occur after Gate and do not determine parent eligibility.Source: §D.3 · p. 19, Table 9 · p. 19

Related papers

Loading 2610.32253v1…