PaperlayerView exact revision on arXiv

Source-linked AI summary

InstAgent: A Recursively Self-Improving Agent Operating System for Heterogeneous Robot Collaboration

Haoyang Luo, Junhao Qin, Jingyan Cao, Yi Huang, Guangjie Yang, Xiaoyu Chen, Haotao Lu, Tianxiang Gui, Ye Shi, Jingya Wang

arXiv:2610.23098v1cs.RO

TL;DR

InstAgent addresses gaps in retaining execution-time collaboration corrections, maintaining task-relevant spatio-temporal context for handoffs, and explicitly representing inter-robot dependencies. It uses behavior trees, feedback-driven revision and reuse, and shared multimodal memory; on 20 real-world tasks, feedback-updated experience improved task success across three high-level models.

  • Problem

    Existing systems do not consistently retain corrected collaboration procedures for reuse, connect historical spatio-temporal observations to current handoff checks, or explicitly represent inter-robot dependencies and recovery logic.

    Source: §I · p. 2

  • Method

    InstAgent composes heterogeneous capabilities into behavior trees with dependencies, handoff conditions, and recovery branches, revises them from execution feedback, retains validated revisions, and retrieves shared observations for planning and handoff validation.

    Source: §I · p. 2, §III · p. 3

  • Results

    15–25 percentage points: feedback-updated experience improved task success across three high-level models versus BT execution without cross-task procedural experience on 20 heterogeneous multi-robot tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Real-robot evaluations across three models report higher task and first-attempt success with feedback-updated experience than with cold starts on the evaluated tasks.

    Source: §V · p. 8

    Future work will examine transfer to new task compositions and robot teams, and the reliability of experience updates over longer deployment periods.

    Source: §V · p. 8

Abstract

from arXiv · show

Heterogeneous robot teams enable complex, long-horizon tasks by combining complementary sensing, mobility, and manipulation capabilities across multiple embodiments. Yet existing systems struggle to retain execution-time corrections as reusable experience, maintain task-relevant context memory across space and time for reliable handoffs, and explicitly represent the inter-robot dependencies required for structured collaboration. In order to address these challenges, we present InstAgent, a recursively self-improving (RSI) Agent Operating System for heterogeneous robot collaboration. InstAgent organizes team behaviors as agentic behavior trees (BTs) that explicitly model capability assignment, inter-robot dependencies and handoff checks. The RSI mechanism updates these collaboration programs from execution feedback and preserves successful revisions for reuse in future tasks, while shared multimodal spatio-temporal memory provides context for planning and handoff validation. Experiments on 20 real-world multi-robot tasks with three high-level models show consistent gains in task success, first-attempt completion, and execution efficiency.

I. INTRODUCTION

Heterogeneous teams can tackle long-horizon tasks, but existing systems do not consistently retain collaboration corrections, connect historical context to handoff checks, or explicitly represent inter-robot dependencies. InstAgent combines behavior-tree collaboration programs, feedback-driven revision and reuse, and shared memory; evaluations report improved task outcomes across three high-level models.

Motivation

  • Three challenges are retaining execution corrections as reusable experience, linking persistent spatio-temporal context to physical handoff checks, and representing dependencies and recovery logic explicitly.The cited prior approaches address reflection, memory retrieval, or skill invocation, but the passage says corrected collaboration procedures are not always retained for inheritance and further improvement.Source: §I · p. 2

Approach

  • InstAgent represents collaboration as executable behavior trees that encode inter-robot dependencies, handoff conditions, and recovery branches.Execution feedback revises failed structures, validated revisions are retained for reuse, and shared multimodal spatio-temporal memory supplies task-relevant observations for planning and handoff validation.Source: §I · p. 2

Results

  • 15–25 percentage points: feedback-updated experience improves task success across three high-level models compared with BT execution without cross-task procedural experience.The evaluation covers 20 heterogeneous multi-robot tasks; the passage also reports increased first-attempt success and fewer recoveries.Source: §I · p. 2

Evaluation

  • The paper evaluates InstAgent on real-world heterogeneous multi-robot tasks using three high-level models.The introduction also identifies an agentic operating system and feedback-updated experience as contributions.Source: §I · p. 2

II. RELATED WORK

Prior embodied-agent systems connect language reasoning to execution and support coordination, feedback, and memory, while heterogeneous collaboration methods use negotiation, allocation, or behavior trees. The related work identifies persistent reuse of validated handoff and recovery logic across tasks as an unresolved need.

A. Physical Agents and Embodied Execution

  • Physical-agent systems integrate language reasoning with robot programs, interfaces, and environmental feedback, while AgentOS frameworks add spatial memory, capability scheduling, or runtime management.InstAgent combines shared observations with collaboration behavior trees to revise inter-robot dependencies and handoff conditions from feedback, retaining effective revisions for later tasks.Source: A. Physical Agents and Embodied… · p. 2

B. Heterogeneous Multi-Robot Collaboration

  • Multi-robot systems use dialogue negotiation, communication architectures, hierarchical allocation, and behavior trees to coordinate actions and recover from failures.The passage notes that communication and context requirements increase with team size and task horizon; existing methods leave persistence and inheritance of corrected handoff conditions and recovery dependencies underspecified.Source: B. Heterogeneous Multi-Robot… · pp. 2–3

C. RSI and Executable Experience

Prior RSI approaches retain feedback as reflections, executable skills, or memory-policy updates, but do not directly preserve collaboration programs with heterogeneous-team dependencies. InstAgent combines experience-guided revisions with reusable behavior-tree subtrees and retrieves shared observations by semantic content, location, and time.

C. RSI and Executable Experience

  • Earlier reflection, skill-reuse, and physical AgentOS systems primarily target single-agent skills or experience and memory policies, rather than heterogeneous multi-robot collaboration programs.InstAgent addresses role bindings, handoff conditions, and recovery dependencies as reusable collaboration structures.Source: C. RSI and Executable Experience · p. 3
  • Experience-Context RSI guides revision with experience documents, failure feedback, and shared observations, while Structured-Program RSI stores validated collaboration behavior-tree revisions for inheritance and further improvement.The passage distinguishes flexible context-based guidance from program structures that retain conditions, execution orders, and recovery branches.Source: C. RSI and Executable Experience · p. 3

III. INSTAGENT: A SELF-IMPROVING AGENT OS FOR HETEROGENEOUS COLLABORATION

  • InstAgent’s Collab-RSI revises collaboration trees from execution feedback, saves verified successful trees with applicability documents, and lets later tasks inherit and adapt them.SMAR-T retrieves shared robot observations according to semantic content, location, and time for planning and failure diagnosis.Source: §III · p. 3

A. Skill BTs: From Atomic Skills to Executable Trees

InstAgent represents executable skills as minimal behavior trees, then composes reusable programs and atomic skills into collaboration trees that assign capabilities and encode dependencies, handoffs, and recovery.

A. Skill BTs: From Atomic Skills to Executable Trees

  • A Skill BT is a minimal behavior tree containing one Action node that invokes a concrete skill; reusable capability procedures can also be composed from atomic instances.Shared modules provide navigation and perception, while robot modules expose robot-specific motion, speech interaction, and manipulation capabilities.Source: A. Skill BTs: From Atomic Skills to… · p. 3
  • During planning, the Agent selects Skill BTs for task requirements and supplies the arguments needed to compose the complete task tree.The passage identifies these skill trees as the basic subtrees for later composition and cross-robot dependency specification.Source: A. Skill BTs: From Atomic Skills to… · p. 4

B. HeteroComposer: Collaborative BT Construction

  • HeteroComposer retrieves applicable experience documents for a task goal and context, then obtains their associated behavior-tree programs.The documents describe goals and usage scope, and each document identifies its associated subtree.Source: B. HeteroComposer: Collaborative BT… · p. 4
  • The Agent builds a complete collaboration tree by reusing existing program structures and filling uncovered steps with atomic skills, assigning robots or modules to actions.Sequences, conditions, and recovery branches encode cross-robot coordination so actions can proceed when prerequisite requirements are met.Source: B. HeteroComposer: Collaborative BT… · p. 4

C. Collab-RSI: Experience-Guided BT Evolution for Heterogeneous Collaboration

Collab-RSI revises inherited collaboration procedures using execution feedback, then validates and retains successful changes for later tasks. Retrieved experience preserves inter-robot dependencies, handoff checks, and recovery logic while allowing adaptation to new tasks.

C. Collab-RSI: Experience-Guided BT Evolution for Heterogeneous Collaboration

  • Collab-RSI can revise reused subtrees or change cross-robot action assignments, handoff conditions, and recovery paths when execution exposes coordination problems.The Agent identifies affected steps using the current program, task goal, observations, and related experience.Source: C. Collab-RSI: Experience-Guided BT… · p. 4
  • A revised candidate is re-executed after structural and skill-interface checks to verify handoff conditions and task outcome.Validated programs can be stored as linked BT structures and experience documents; failed execution records remain diagnostic context.Source: C. Collab-RSI: Experience-Guided BT… · p. 4
  • Figure 2 depicts the progression from reusable Skill BTs, through task-specific composition with handoff conditions, to retained validated programs with linked context.The RSI Engine retains programs for inheritance and further revision.Source: Figure 2 · p. 4
  • Stored BTs preserve execution dependencies, handoff checks, and recovery procedures, while linked documents record applicability conditions, failure context, and revision rationale.A validated composition of existing subtrees can itself become a retrievable experience entry.Source: C. Collab-RSI: Experience-Guided BT… · p. 4
  • For later tasks, the Agent retrieves applicable documents and program structures, then inherits revised subtrees or validated composite programs while adapting them to the current task.The cycle of inheritance, execution, revision, validation, and storage allows further reuse and revision.Source: C. Collab-RSI: Experience-Guided BT… · pp. 4–5

D. SMAR-T: Shared Multimodal Spatio-Temporal Memory with Agent-Driven Retrieval

SMAR-T pools robot observations in shared environmental memory, linking visual content with captions, pose, time, and source identity. The Agent retrieves and checks candidates against semantic, spatial, and temporal constraints, continuing retrieval when evidence is insufficient.

D. SMAR-T: Shared Multimodal Spatio-Temporal Memory with Agent-Driven Retrieval

  • Each shared-memory observation links an image and caption with visual and caption embeddings, the observing robot’s pose, timestamp, and source robot.The pose is stored in the shared map; pose, time, and source metadata link observations across robots and task stages.Source: D. SMAR-T: Shared Multimodal… · p. 5
  • The Agent scopes retrieval by semantic, spatial, and temporal constraints, selects read-only tools, and evaluates returned images, captions, and metadata against the original request.Ranking alone does not establish that constraints are satisfied; unresolved constraints or missing fields prompt further retrieval.Source: D. SMAR-T: Shared Multimodal… · p. 5
  • Insufficient evidence leads to further retrieval, and after at most ten real-robot tool calls the Agent may report insufficient evidence or request a fresh observation.Sufficient evidence can support a grounded response and, when requested, BT construction or revision.Source: D. SMAR-T: Shared Multimodal… · p. 5
  • Position retrieval ranks by distance from the query location using pose position, while time retrieval ranks observations by temporal distance from the query time.Observation and query times use the same scene origin; orientation is retained for navigation rather than position ranking.Source: D. SMAR-T: Shared Multimodal… · p. 5
  • For semantic queries, visual similarity recalls at most ten frames before caption-based reranking returns at most five candidates for Agent evaluation.The reranking score uses min–max-normalized similarities and λ = 0.15.Source: D. SMAR-T: Shared Multimodal… · p. 5

IV. EXPERIMENTS

The experiments evaluate real-robot BT execution, revision, and procedural-experience reuse, alongside a service case and a separate assessment of environmental-context retrieval.

IV. EXPERIMENTS

  • The evaluation compares real-robot BT execution, revision, and procedural-experience reuse.The study also includes a service case and assesses task-relevant environmental context through RAVEN-QA retrieval.Source: §IV · p. 5
  • A service case is included as part of the real-robot evaluation.The passage does not specify its outcome.Source: §IV · p. 5
  • RAVEN-QA retrieval separately assesses task-relevant environmental context.Source: §IV · p. 5

A. Experimental Setup

The setup tests heterogeneous robots on recurring collaboration structures under controlled perturbations and recovery budgets. It compares configurations that vary cross-task experience reuse and retention of validated revisions, and separately evaluates environmental-memory retrieval.

A. Experimental Setup

  • G1, Go2, and ALOHA provide interaction and guidance, navigation and transport, and manipulation and loading for 20 heterogeneous tasks involving capability allocation and inter-robot handoffs.The tasks vary objects, locations, and goals to test subtree reuse and adaptation within a fixed robot team.Source: A. Experimental Setup · pp. 5–6
  • Each of three models runs each task twice under four configurations, yielding 480 measured runs with paired task repetitions across configurations.Task instances, failure settings, and perturbation types, counts, and trigger stages are held constant in paired comparisons.Source: A. Experimental Setup · p. 6
  • Perturbations include arrival error, navigation and loading failures, and user-requested destination changes; success requires verified physical and interaction postconditions.Runs allow at most five executed failure-triggered recoveries or retries, with up to three BT revision candidates per failure episode.Source: A. Experimental Setup · p. 6
  • The four configurations range from textual-plan skill invocation to BT execution with no cross-task retrieval, fixed subtree reuse, or RSI-updated retained experience.E0–E1 tests experience reuse, while E1–E2 tests retention of validated revisions; libraries are frozen during evaluation and evaluation tasks are excluded from library construction.Source: A. Experimental Setup · p. 6
  • RAVEN-QA retrieval is evaluated under its official protocol across four observation sources and queries covering object retrieval, reasoning, information recall, and spatial understanding.Full InstAgent adds candidate comparison and, when needed, position or time retrieval and caption verification against source images; the no-Agent variant returns Top-1 directly.Source: A. Experimental Setup · pp. 6–7
  • Table II covers real-robot performance across high-level models, execution configurations, and procedural-experience stages.Source: Table II · p. 7

B. Model Comparison and BT-Based RSI Evaluation

Across three models, feedback-updated experience improves task success, first-attempt success, and recovery counts on the 20 evaluated tasks; the E0–E2 comparison combines experience reuse and retained revisions.

B. Model Comparison and BT-Based RSI Evaluation

  • SR rises by 15, 22.5, and 25 points, FSR by 12.5, 30, and 20 points, and recovery count falls by 0.70, 1.30, and 1.20 across the three models from E0 to E2.Paired task-success and recovery-count tests yield p < 0.05 for all three models.Source: B. Model Comparison and BT-Based RSI… · p. 7
  • E0–E2 measures the combined benefit of experience reuse and retained revisions, while E1–E2 assesses the additional benefit associated with retaining validated revisions.E2 SR gains over E1 are 10, 17.5, and 15 percentage points for GPT-5.6 Sol, DeepSeek-V4-Pro, and Qwen3.8-27B, respectively.Source: B. Model Comparison and BT-Based RSI… · p. 7
  • In a qualitative cola-delivery case, the Agent added basket-content checks before return navigation and delivery, with failed checks triggering reloading.The validated subtree retains both checks and recovery logic for automatic reuse, while the linked document records the diagnosis and revision rationale; atomic Skills remain unchanged.Source: B. Model Comparison and BT-Based RSI… · p. 7

C. RAVEN-QA Memory Evaluation

RAVEN-QA evaluates retrieval of task-relevant environmental observations. The method combines visual retrieval, caption reranking, and Agent verification, with higher overall SR than RAVEN and ReMEmbR.

C. RAVEN-QA Memory Evaluation

  • Visual similarity forms a high-recall candidate set, captions refine its order, and the Agent verifies selected captions against original images.RAVEN indexes original observations without language descriptions as a second ranking signal, while ReMEmbR retrieves caption-based memories that may omit image details.Source: C. RAVEN-QA Memory Evaluation · p. 7
  • 93.14% overall SR exceeds RAVEN by 8.57 points and ReMEmbR by 28.65 points.The near tie on IRS-hard (82.93% versus 82.90%) marks a boundary: reranking cannot recover targets absent, occluded, or missed by the visual candidate set.Source: C. RAVEN-QA Memory Evaluation · p. 8
  • Returning Top-1 without the Agent reduces overall SR from 93.14% to 69.31%, measuring the integrated Agent-driven workflow rather than any individual component.The Agent performs candidate comparison, position and time tool calls, evidence interpretation, and caption verification.Source: C. RAVEN-QA Memory Evaluation · p. 8

V. CONCLUSION

InstAgent links capability binding, shared context, and physical execution through BT-based recursive self-improvement. Real-robot evaluation reports higher task and first-attempt success with feedback-updated experience than with cold starts on the evaluated tasks.

  • Across three models, feedback-updated experience yields higher task and first-attempt success than cold starts on the evaluated tasks.Source: §V · p. 8
  • Linked documents and executable subtrees retain feedback-driven collaboration-program changes, allowing inherited procedures to be composed and revised again.Source: §V · p. 8
  • The service case illustrates how a retained BT condition changes the handoff between loading and transport.Source: §V · p. 8
  • Future work will examine transfer to new task compositions and robot teams, and experience-update reliability over longer deployments.Source: §V · p. 8
Loading 2610.23098v1…