PaperlayerView exact revision on arXiv

Source-linked AI summary

RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments

Sibo Zhu, Shicheng Fan, Xinyue Wang, Wenyi Wu, Kun Zhou, Biwei Huang

arXiv:2609.15364v1cs.AIcs.CLcs.CV

TL;DR

Digital agents must adapt to unfamiliar environments whose interfaces and failure modes are not fully represented in pretrained knowledge. RSIAgent addresses this with training-free multi-agent exploration that constructs reusable memory through broad and deep self-exploration. On OSWorld-v2 and Agent’s Last Exam, it substantially improves Kimi-K3 and GLM-5.3, enabling them to outperform frontier closed-source models without parameter updates.

  • Problem

    Digital agents need to adapt to new environments whose interfaces, tools, conventions, and failure modes may not be fully captured by pretrained knowledge, while conventional adaptation often requires costly additional training and human assistance.

  • Method

    RSIAgent coordinates curriculum, actor, and verifier agents in a training-free broad-then-deep exploration process that builds, verifies, consolidates, freezes, and reuses environment-specific memory.

  • Results

    RSIAgent substantially improves Kimi-K3 and GLM-5.3 on OSWorld-v2 and Agent’s Last Exam, enabling them to outperform frontier closed-source models without updating model parameters.

  • Takeaways & Limitations

    The results show that agent-level self-improvement can narrow and even reverse the capability gap between open- and closed-source foundation models without parameter updates.

  • Takeaways & Limitations

    RSIAgent incurs additional test-time exploration cost, depends on finite budgets and memory quality, and can propagate incorrect verifier judgments into later exploration and memory updates.

Abstract

from arXiv · show

Digital agents must often adapt to new environments whose interfaces, tools, and failure modes are not fully captured by pretrained models. We introduce \textbf{RSIAgent}, a training-free multi-agent framework for recursive self-improvement through autonomous memory construction. RSIAgent coordinates curriculum, actor, and verifier agents to continually explore the environment, validate outcomes, and retain environment-specific knowledge, including reusable causal relationships between actions, conditions, and consequences. It further adopts a \textbf{broad-then-deep} exploration strategy, combining parallel broad recursive self-exploration for discovering diverse environment structures with focused deep self-exploration for uncovering hard cases, hidden constraints, boundary conditions, and previously unknown causal dependencies. The resulting memory is frozen and can be directly reused for downstream tasks without updating model parameters. Experiments on OSWorld-v2 and Agent's Last Exam show that RSIAgent substantially improves strong open-source models, enabling Kimi-K3 and GLM-5.3 to outperform frontier closed-source models including GPT-6.

1 Aether AI, 2 University of California San Diego, 3 University of Illinois Chicago

The paper lists affiliations spanning Aether AI, the University of California San Diego, and the University of Illinois Chicago, alongside project resources for RSIAgent.

  • The authors are affiliated with Aether AI, the University of California San Diego, and the University of Illinois Chicago.
  • The paper provides a GitHub code repository and a project website for RSIAgent.
  • Figure 1 summarizes RSIAgent’s autonomous exploration and reports open-source models surpassing frontier closed-source models on two benchmarks.

1 Introduction

The introduction motivates training-free adaptation to unfamiliar digital environments and presents RSIAgent as a recursive, multi-agent approach that builds reusable memory through broad and deep exploration. Experiments report that it improves Kimi-K3 and GLM-5.3 enough to outperform frontier closed-source models on OSWorld-v2 and Agent’s Last Exam.

  • Digital agents need adaptation because new environments contain interfaces, tools, conventions, and failure modes not fully captured by pretrained knowledge.
  • RSIAgent coordinates curriculum, actor, and verifier roles to expand and refine environment-specific memory with newly acquired causal knowledge.
  • Broad Recursive Self-exploration gathers diverse experience, while Deep Recursive Self-exploration targets corner cases, hidden constraints, and boundary conditions before freezing memory for downstream reuse.
  • RSIAgent improves Kimi-K3 and GLM-5.3 on OSWorld-v2 and Agent’s Last Exam, enabling them to outperform frontier closed-source models.

2 Preliminary

The paper models digital agents as policies that act in environments using persistent memory and interaction histories. It defines recursive self-improvement as constructing reusable environment knowledge through autonomous exploration, reliable feedback, and memory consolidation without parameter updates.

  • At each step, the agent generates an executable action from the task, interaction history, and durable memory, then receives a new environment state and observation.
  • Interaction history resets between tasks, while durable memory persists across tasks and retains reusable knowledge.
  • Recursive self-improvement constructs persistent, reusable environment-specific memory through exploration without updating model parameters.
  • The central design problem is efficiently identifying informative experiences, grounding them in reliable environment feedback, and consolidating the resulting knowledge.
  • Figure 2 depicts broad parallel exploration, deep memory refinement, coordinated agent roles, and frozen memory reuse in a FreeCAD example.

3 Methodology

RSIAgent uses a coordinated curriculum–actor–verifier harness and a broad-then-deep exploration process. Broad exploration builds diverse coverage, deep exploration targets difficult gaps and boundary conditions, and the resulting memory is frozen for test-time reuse.

  • 3.1 Multi-Agent Harness Framework: The curriculum, actor, and verifier agents coordinate recursive exploration, execution, feedback grounding, and memory updates.
  • 3.1 Multi-Agent Harness Framework: The actor generates executable actions and maintains evolvable memory containing environment knowledge, procedures, scripts, and lessons from prior executions.
  • 3.2 Broad-Then-Deep Recursive Self-Exploration: BRS explores multiple directions in parallel to build broad environmental coverage, using curriculum-generated tasks and verifier-evaluated outcomes.
  • 3.2 Broad-Then-Deep Recursive Self-Exploration: DRS sequentially increases difficulty to expose knowledge gaps, hard cases, hidden constraints, and boundary conditions, consolidating each verified experience before the next task.
  • Test-time Memory Reuse: After exploration, memory is frozen and directly reused by the actor for downstream tasks while curriculum generation and memory updates are disabled.

4 Experiments

RSIAgent improves interactive-agent performance through autonomous exploration and memory reuse across OSWorld 2.0, Agents’ Last Exam, and GameCraft-Bench. Results also show that broad exploration followed by deep exploration is strongest, while targeted exploration, verification, and memory consolidation remain limiting factors.

  • Main Results: 78.98 OSWorld partial score and 84.82 ALE partial score exceed reported GPT-6 Astra scores by 6.38 and 2.56 percentage points, respectively, with RSIAgent also above Claude Opus 5.On OSWorld, partial score rises from 71.97 to 78.98 and binary accuracy from 37.80 to 42.68; on ALE, partial score rises from 83.75 to 84.82 and binary accuracy from 49.25 to 50.75.
  • Effect of RSI Rounds: By step 8, representative OSWorld tasks T044, T049, and T065 reach partial scores of 100%, 80%, and 100%, respectively.Broad exploration accumulates diverse procedures and environment knowledge, while deep exploration refines task-specific details.
  • Ablation Study: Full RSI achieves the highest partial score on all four ablation tasks, averaging 74.54% versus 65.52% for broad-only and 56.50% for deep-only exploration.Broad-only exploration improves over baseline on every task, whereas deep-only exploration falls below baseline on T085 and T089.
  • Evaluation in Game Environments: RSIAgent substantially improves game quality across all base generators and consistently benefits both weak and strong GameCraft-Bench games.Play2Code helps weaker generators but can degrade already high-quality games, whereas accumulated RSI experience provides reusable knowledge for diagnosing and editing different games.
  • Failure Modes Analysis: Failure analysis identifies insufficiently targeted exploration, incomplete verification, and unreliable memory consolidation as mechanisms limiting recursive self-improvement.These mechanisms can interact, allowing uncertain or incorrect operations to persist and influence later behavior.

5 Related Work

Prior work studies agent self-improvement through design, prompt, skill, and procedure changes, as well as exploration and verification-based memory refinement. RSIAgent instead targets recursive self-improvement in new digital environments without external task rewards, using autonomous broad-then-deep exploration and reusable memory.

  • Prior approaches explore changes to agent designs, prompts, skills, and improvement procedures, including self-modification and descendant-based selection.
  • Related systems also explore environments to collect reusable experience or use verification feedback to refine skills and memory.
  • RSIAgent studies recursive self-improvement in new digital environments without external task rewards, using environment feedback to construct reusable memory.
  • RSIAgent combines autonomous broad-then-deep exploration with memory that guides further exploration and downstream execution without parameter updates.

6 Conclusion

RSIAgent provides a training-free multi-agent framework that acquires, verifies, and consolidates environment-specific knowledge through broad and deep recursive self-exploration. Experiments show substantial improvements for strong open-source models and performance beyond frontier closed-source models without parameter updates.

  • RSIAgent coordinates curriculum, actor, and verifier agents to acquire, verify, and consolidate reusable environment-specific knowledge.Its broad stage builds diverse coverage, while its deep stage targets hard cases, hidden constraints, and boundary conditions.
  • Broad recursive self-exploration builds diverse environment coverage before deep recursive self-exploration refines hard cases and boundary conditions.
  • The framework extends recursive self-improvement beyond digital computer-use environments as a proposed direction for AI for Science and games.

7 Limitations

RSIAgent requires additional exploration and depends on finite budgets, stopping policies, learned memory, verifier reliability, and environment-specific design choices. Its autonomous software interaction also creates ethical risks, so experiments use controlled environments with permitted tools and data access.

  • Additional test-time exploration and practice can introduce substantial computation cost.
  • Performance depends on finite exploration budgets, stopping policies, and the quality of learned memory.
  • Incorrect judgments by the model-based verifier may propagate into later exploration and memory updates.
  • Different environments may require different tools, verification signals, and exploration strategies, while the experiments do not fully isolate every component’s contribution.
  • Autonomous software exploration and execution introduce risks including unintended actions, unauthorized access, and privacy leakage.The experiments are conducted in controlled environments with permitted tools and data access.

A Implementation and Agent Interfaces

The reference implementation separates curriculum, actor, verifier, and memory-update responsibilities through explicit interfaces and role boundaries. Broad exploration runs independent projects in parallel from shared snapshots, while deep exploration performs sequential, target-conditioned refinement before frozen-memory evaluation.

  • Role Interfaces and Information Boundaries: The actor executes projects, the verifier independently grounds PASS or FAIL judgments in environment evidence, and the curriculum agent selects further experiences.Role-specific contexts limit private reasoning and memory sharing across agents.
  • Memory Consolidation and Learning Handoffs: Memory consolidation distills procedures, constraints, and failure lessons, reconciles contradictions and unsupported conclusions, and promotes only completed actor learning to persistent memory.
  • Target-Conditioned Refinement: DRS begins with a target attempt using BRS memory, selects sequential practice for failures or uncertain successes, and retries the target after verified consolidation.
  • Frozen-Memory Evaluation: Final evaluation copies and integrity-checks frozen memory, disables curriculum decisions and host writeback, resets the environment, and scores the candidate externally.
  • Task Execution and Verification: The actor uses complete Python or Bash programs as actions, with runtime outputs, exit status, metadata, visual evidence requests, and read-only completion checks.
  • Broad Recursive Self-exploration: BRS derives diverse prerequisite, variant, contrast, and stress projects without attempting the unchanged target, then executes projects in parallel from a common memory snapshot.

C.1 Benchmark Descriptions and Evaluation

The evaluation covers OSWorld 2.0 and Agents’ Last Exam using partial and binary task-completion metrics, with RSI compared against a shared actor–verifier baseline and frontier systems. Reporting includes explicit task-selection, aggregation, coverage, and score conventions.

  • Benchmark Descriptions: OSWorld evaluates long-horizon desktop and web workflows, whereas ALE evaluates professional workflows combining software interaction, code execution, and verifiable deliverables.The reported OSWorld subset includes document editing, media production, and specialized engineering and scientific software workflows.
  • Evaluation Harness: The baseline and frozen-memory RSI conditions use the same code-as-policy actor–verifier harness, while RSI adds curriculum-guided exploration and reusable memory.The verifier returns PASS, FAIL, or UNVERIFIED, with unverified outcomes requesting additional evidence rather than being treated as success.
  • Exploration Configuration: BRS explores related experience before DRS attempts the target, and DRS continues until the curriculum agent judges that no further useful practice is needed.BRS checks its nominal eight-project budget only after complete waves, while DRS uses verified outcomes to guide memory updates and subsequent decisions.
  • Selection and Reporting: Reported aggregates retain improvements and regressions, replace baselines only for non-diagnostic RSI results, and keep baseline scores for tasks without completed RSI results.T082’s setup failure is counted as zero in both reported aggregates, and selected-run budgets and evaluation scopes are not fully matched.
  • Frontier Comparison: RSIAgent achieves the highest reported partial-credit scores among the systems compared, reaching 50.75 on ALE versus 52.24 for GPT-6 Astra on OSWorld partial credit.The cited comparison indicates that the partial-credit advantage does not extend to every reported metric.

D Autonomous Game Development Setup

The autonomous game-development evaluation tests whether RSIAgent can improve executable games through repeated play, diagnosis, and code modification. It compares full RSIAgent and a no-recursive-experience variant with Play2Code across sampled GameCraft-Bench tasks.

  • Evaluation Setup: The setup randomly samples 40 GameCraft-Bench tasks and evaluates complete playable games through executable interaction traces and rubric-based judging.GameCraft-Bench is used to assess autonomous game development in an interactive setting requiring repeated playtesting, diagnosis, and modification.
  • Compared Methods: The comparison includes Play2Code, RSIAgent without recursive experience evolution, and full RSIAgent.Play2Code is described as a continual game-improvement baseline based on repeated playtesting and code revision.
  • Case-Study Design: The case studies trace how exploratory observations become persistent records and reusable procedures before comparing execution without memory against execution with accumulated memory.Examples cover railway booking, presentation repair, video editing, FreeCAD, and REAPER audio production.
  • Memory-Based Procedure: The appendix organizes case studies around BRS, DRS, and test-time memory reuse, with fixed model weights throughout.BRS builds broad environment knowledge, DRS refines it through feedback-guided practice, and frozen memory is applied during task execution.

E.1 Memory Accumulates Through Exploration and Consolidation

Exploration builds a persistent memory corpus by recording outcomes, revising procedures, and consolidating verified lessons across related projects. The resulting memory grows through both accumulation and reorganization, including failure-derived checks and reusable domain procedures.

  • Memory Growth: T044, T049, and T065 finish BRS with 109,647, 89,086, and 159,065 bytes, then reach final frozen memories of 316,541, 141,312, and 281,085 bytes, respectively.Across the three trajectories, the archive contains 39 journal-linked snapshots whose file counts and byte totals were checked against the journals.
  • Memory Organization: Memory growth includes revision within stable files and occasional consolidation reductions, rather than simple concatenation of every interaction.T044 expands from about 110 to 317 kB while retaining twelve files, whereas T065’s third BRS consolidation shrinks from 13,243 to 11,510 bytes.
  • Cross-Branch Consolidation: Failed and successful branches jointly produce reusable rules, such as correcting a coordinate-system interpretation and re-enumerating current records before updating an index.The index preserves lessons about retrieving discoveries made by other branches, not only task-operation instructions.
  • Failure-Derived Memory: A failed Shotcut export creates dedicated failure and verification records, modifies nine existing files, and increases memory from 40,186 to 55,506 bytes.The retained lesson checks exported audio settings against a command-line render after the verifier identifies a default-setting discrepancy.
  • Reusable Editing Procedure: Repeated video checks validate a crop-and-letterbox procedure across 800 × 450 and 1920 × 1080 inputs, with measured top bands of 36 and 44 pixels and symmetric bars of 18 and 22 pixels.The procedure is grounded in visible frame results, saved video properties, and crop measurements repeated for each input.

E.2 A Branch in Detail: Learning to Observe Changing Seat Availability

The railway-booking branch learns to observe changing seat layouts, verify dynamic-page evidence, and distinguish payment confirmation from itinerary validity. DRS then extends these observations into stricter booking rules and retrieves them during later target execution.

  • Seat-Availability Semantics: A valid release requires two travelers to obtain same-row D and F seats simultaneously, with the first observed valid release preferred over later valid windows.The first release offers 17D and 17F; later releases include invalid or later alternatives, demonstrating why temporal and letter constraints must be checked together.
  • Dynamic Layout Observation: Content-driven column widths shift seat positions between releases, so a frozen pixel grid misclassifies states until columns are re-detected for each layout.FREE columns are about 72 pixels wide, occupied-only columns about 36 pixels wide, and releases #3/#4 were misclassified before state-specific detection.
  • Evidence Collection: A failed watcher and incomplete clipboard reads lead to an inline capture loop, completeness checks, and a verified record of 663 frames over approximately 335 seconds.The consolidated memory preserves timestamps, release sequences, monitoring checks, and the requirement to confirm that the process is producing evidence.
  • Deep Recursive Self-Exploration: DRS converts a payment-and-itinerary failure into a stricter transfer rule and retains failure, corrected interpretation, and successful construction records together.The presentation-repair examples similarly show DRS refining broad skills into a concrete, retrievable target solution.
  • Persistent-State Verification: The booking procedure distinguishes seat-selection checks from persistent order checks, reopening My Bookings when transient timeout messages conflict with a Paid state.Order details confirm route and payment, while seat-letter checks remain tied to seat-selection views.
  • Test-Time Memory Reuse: A later target execution retrieves accumulated playbooks and produces two Paid orders totaling $177.26 with arrival at Beijing South at 19:58, rather than copying an earlier itinerary.The final itinerary uses G98 and D756, illustrating procedural retrieval rather than literal reuse of prior route details.

E.5 Without Memory Versus Frozen Memory: Which Errors Change the Score?

Frozen memory changes task outcomes by supplying environment-specific procedures and checks that address scorer-visible errors across presentation repair, video editing, CAD reconstruction, and audio production. The retained cases show higher partial or full credit, while some gains remain incomplete or reflect historical rather than matched ablations.

  • Cross-task interpretation: Across the retained comparisons, memory reuses application-specific procedures, constraints, and failure lessons rather than guaranteeing uniformly faster execution or complete reconstruction.The comparisons are selected historical runs, not matched-seed memory ablations, and saved scores do not isolate individual practice contributions.
  • T044 video editing: T044’s memory run changes the score from 0.40 to 1.00 by using a native 76-pixel crop with Center disabled, satisfying all scorer components.The run preserves resolution and meets the size cap; memory also increases actor iterations from 26 to 42, so the gain reflects completion rather than shorter execution.
  • T103 CAD reconstruction: T103 improves from 0.2500 without memory to 0.6789 and 0.6897 with frozen memory, but support and mounting-hole geometry remain incorrect.The baseline already exports a valid solid, so the gain reflects higher partial reconstruction rather than merely successful file creation.
  • T085 audio editing: REAPER scores rise from 0.6800 to 0.9417 and 0.9413, averaging 0.9415, mainly through sentence-gap and processed-ending credit.The evaluator approximates sentence boundaries through acoustic activity, so these components measure recorded criteria rather than complete semantic or perceptual correctness.
Loading 2609.15364v1…