PaperlayerView exact revision on arXiv

Source-linked AI summary

EMERGE-Policy: A Robot Mind Emerges Beyond a Single Policy

Zhirui Fang, Qingchi Yu, Ziyang Chen, Longfei Li, Haoran Ma, Keru Zhou, Xinrun Xu, Samith Va, Yuxuan Hu, Peixuan Song, Qiang Du, Bin Qian, Yongkang Deng, Xin Li, Yezhen Wang, Zhe Li, Hao Luo, Shuyan Li, Ziwei Wang, Weijian Deng, Xiu Li

arXiv:2608.29896v1cs.RO

TL;DR

Robot manipulation systems need coordinated perception, reasoning, action, verification, and memory beyond a single policy. EMERGE-Policy provides this coordination through a graph-structured framework, and reports improved benchmark performance plus real-robot deployment without additional fine-tuning. The paper also identifies reproducibility, cost, and adaptation boundaries for interpreting these system-level results.

  • Problem

    Robot intelligence involves coordinated perception, prediction, control, memory, and feedback, but existing systems commonly pass detailed evidence directly to high-level planners.

  • Method

    EMERGE-Policy coordinates a Main Agent, isolated role-specific Sub Agents, external memory, and Operational, Imagination, and Evaluation Skills through structured interfaces, verification, recovery, and context control.

  • Results

    Across reported evaluations, EMERGE-Policy improves over its π0.5 and Cosmos Policy execution backends, with particularly large gains under several perturbation and implicit-instruction settings, and is deployed on a physical robot.

  • Takeaways & Limitations

    The reported outcomes support treating robot behavior as system-level coordination among heterogeneous components rather than as another low-level control policy.

  • Takeaways & Limitations

    Interpreting the gains requires matched ablations and stronger reporting of repeated trials, intervention rules, execution cost, and failure categories.

Abstract

from arXiv · show

A robot's effective ``mind'' need not reside in a single policy. It can emerge when specialized components perceive, reason, predict, act, verify, and remember within a shared orchestration process. EMERGE-Policy turns this perspective into a graph-structured agentic framework that coordinates both capability invocation and information exchange. A Main Agent retains task-level state within an active context window, while role-specific Sub Agents process perception, execution monitoring, verification, and memory consolidation in isolated contexts and return structured, task-relevant evidence. Role-specific contexts control information load by exposing only decision-relevant evidence to the Main Agent, while the functional Skill interface composes heterogeneous backends as Operational, Imagination, and Evaluation Skills. Criterion-grounded verification, textual failure diagnosis, and Branch Stack recovery provide localized correction, with token-aware external memory preserving task-relevant state. Together, their closed-loop interaction realizes the system-level policy captured by the name EMERGE-Policy. Without additional fine-tuning, we achieved outstanding performance on several public benchmark that have had a wide-reaching impact, and conducted a series of real robot experiments. These system-level results suggest that through the division of different functional sub-tasks among multiple agents and their concurrent collaboration, as well as the technical paradigm where the model is regarded as a skill and called within the framework, EMERGE-Policy can extend the robust robot policies beyond isolated runs.

1 Introduction

EMERGE-Policy treats robot intelligence as coordinated system behavior rather than the output of a single policy. It combines specialized agents, functional skills, structured information exchange, verification, recovery, and memory, with reported benchmark and real-robot gains.

  • Motivation and framework: The framework addresses manipulation by coordinating perception, prediction, task reasoning, physical control, memory, and environmental feedback.This coordination supports grounding instructions, executing behavior, verifying physical outcomes, and responding to deviations.
  • Information and skill interfaces: EMERGE-Policy separates evidence processing from planning by returning structured, decision-relevant results to the planner.This limits the amount of dense perceptual, monitoring, verification, and memory-processing information exposed to high-level reasoning.
  • Information and skill interfaces: Its shared subgoal specification connects Operational, Imagination, and Evaluation Skills to intended changes and explicit completion criteria.The functional interface allows heterogeneous policy and world-model backends to participate in the same decision loop.
  • Motivation and framework: EMERGE-Policy frames robot behavior as emerging from coordinated Main Agent, Sub Agents, Skills, and external memory rather than a single policy.Different agents independently handle sub-tasks or sub-functions within the overall problem.
  • Evaluation: The authors report gains over π0.5 and Cosmos Policy on LIBERO and LIBERO-plus, a 3.59% Robodojo success-rate increase, and physical-robot long-horizon experiments.Reported improvements are 2.0% and 0.7% on LIBERO, and 2.6% and 11.7% on LIBERO-plus, respectively.

2 EMERGE-Policy: Graph-Structured Agentic Policy Orchestration

EMERGE-Policy organizes robot manipulation as closed-loop coordination among a Main Agent, role-specific Sub Agents, and a unified Skill library. The Main Agent maintains task context, delegates evidence processing and capabilities, and uses returned outcomes to advance or recover the plan.

  • Workflow: The workflow maps language instructions and observations to verified execution through coordinated Main Agent, Sub Agent, and Skill interactions.Structured evidence and Skill outcomes update task state after each interaction.
  • Main Agent: The Main Agent maintains active task context, decomposes instructions into subgoals with completion criteria, and selects the next Sub Agent or Skill.This provides the task-level control loop for the system.
  • Sub Agents: Sub Agents process task-specific evidence in isolated contexts and return concise structured results to the Main Agent.The design keeps role-specific processing separate from task-level reasoning.
  • Skill library: Operational Skills execute actions, Imagination Skills predict outcomes, and Evaluation Skills assess feasibility, progress, or completion.The shared functional interface permits heterogeneous capabilities to be organized within one decision loop.

2.1 Main Agent: Planning, Recovery, and Memory

The Main Agent plans through explicit subgoals, maintains compact task state, verifies outcomes, and handles failures through diagnosis and localized recovery. Token-aware memory consolidation preserves relevant state while avoiding cross-episode transfer.

  • Planning and orchestration: The Main Agent chooses among delegation, Skill invocation, state updates, recovery, plan revision, and termination under an LLM-based orchestration policy.Its active context contains task state, memory, recent results, and compact Skill signatures.
  • Hierarchical planning: Each subgoal specifies a target entity, intended state transition, and completion criterion for executable planning and verification.The ordered subgoal sequence and execution status are maintained in PLAN.md.
  • Verification and recovery: After execution, verification compares the observed outcome with the completion criterion; failures trigger textual diagnosis and subsequent recovery decisions.The Main Agent may retry the operation, create a localized recovery subgoal, or revise the high-level plan.
  • Memory and state: Task state is divided among active context, PLAN.md, HISTORY.md, MEMORY.md, and a separate Branch Stack for transient recovery subgoals.The files distinguish working memory, plans, chronological summaries, consolidated beliefs, and recovery state.
  • Memory and state: When the context reaches token capacity, a Memory Integration Agent archives summaries, revises consolidated facts, and reconstructs the next active context.The full chronological archive is not automatically reinserted into the active window, and model weights remain unchanged.
  • Memory and state: All task-state structures are reinitialized before each evaluation episode, so reported evaluation uses no cross-episode online adaptation.Episode order does not change the memory available to the agent.

2.2 Sub Agents: Role-Specific Evidence Processing

Sub Agents isolate evidence-intensive processing from Main Agent planning and return structured, task-relevant outputs. Specialized roles cover perception, verification, monitoring, and failure diagnosis while asynchronous execution preserves the planning cycle.

  • Context isolation: Each Sub Agent receives only information required for its assigned request and returns structured evidence rather than raw observations or complete processing traces.This keeps dense evidence processing outside the Main Agent’s active context.
  • Asynchronous execution: Sub Agents run as non-blocking background workers with isolated prompts containing the active subgoal, spatial bounds, and permitted tools.Independent requests can be processed asynchronously while the Main Agent retains task-state control.
  • Evidence processing: Role-specific processing maps raw multimodal observation history and an active subgoal to structured evidence conditioned on that subgoal.The conditioning prevents processing the complete scene without task context.
  • Specialized roles: Perception Sub Agents extract task-relevant objects, states, and spatial relationships from visual and spatial observations.Their outputs contain scene evidence rather than the complete perceptual processing trace.
  • Specialized roles: Verification Sub Agents assess preconditions and compare observed outcomes with completion criteria, returning judgments with supporting evidence.They operate before or after execution depending on the verification request.
  • Specialized roles: Monitor Sub Agents detect deviations during physical interaction and return anomaly reports whose signals depend on available sensors and execution backends.Failure reports can include observed evidence, failure type, state changes, and an inferred cause.
  • Failure handling: The Main Agent uses structured failure reports to retry, create localized recovery subgoals, or revise the high-level plan.This closes the loop between execution, observation, verification, and recovery.

2.3 Unified Skill and Tool Orchestration

EMERGE-Policy unifies heterogeneous robot capabilities through functional Skills, tool APIs, and structured invocation. Its lifecycle coordinates planning, perceptual grounding, physical dispatch, and outcome verification with localized recovery.

  • Unified Skill Interface: A Skill exposes a callable capability with defined inputs and outputs, while Sub Agents invoke Skills and return structured evidence to the Main Agent.The shared interface accommodates learned policies, world models, perception models, analytical controllers, and system tools.
  • Functional Skill Categories: Operational Skills execute actions, Imagination Skills predict candidate outcomes, and Evaluation Skills assess feasibility, progress, or completion.These categories are defined by function rather than model architecture, allowing heterogeneous backends to participate in one protocol.
  • Extensibility: New capability implementations can be registered through the same Skill schema without changing the orchestration protocol.The framework can incorporate additional implementations, such as a value estimator as an Evaluation Skill.
  • Standardized Schema: Each callable interface uses ⟨Imeta, Θparam, Φpre, Ψpost⟩, with full parameters and conditions retrieved only after selection.Lazy loading keeps only compact capability metadata in the active context until invocation.
  • Tool APIs: Tool APIs expose execution, workspace, perceptual, and hardware capabilities to the Main Agent or authorized Sub Agents.Perceptual interfaces provide scene evidence for target selection and spatial constraints before execution.
  • Execution Lifecycle: The execution lifecycle proceeds through planning, perceptual grounding, physical dispatch, and outcome verification.Failed verification returns a structured failure report that triggers Branch Stack recovery or high-level plan revision.

3 Experiments

The experiments test whether EMERGE-Policy provides system-level benefits beyond execution backends and identify which design choices support those benefits. The evaluation covers benchmark performance, Sub-Agent workflow, imagination, memory, physical deployment, and planning efficiency.

  • System-Level Benchmark Performance: RQ1 compares EMERGE-Policy with execution backends and representative agentic systems on standard and perturbed manipulation benchmarks.This evaluates system-level benchmark performance beyond the underlying execution backend.
  • Sub-Agent Workflow: RQ2 tests isolated Perception, Verification, Monitor, and Memory Sub Agents against a single-agent harness under matched tools, observations, and interaction budgets.The matched design targets the benefit attributable to the Sub-Agent workflow.
  • Imagination and Evaluation: RQ3 examines how Imagination and Evaluation Skills affect candidate-action selection, failure avoidance, and final task success.The question focuses on the action-selection pathway rather than execution alone.
  • Memory Architecture: RQ4 studies long-horizon visual memory and environment-state tracking as mechanisms for preventing error accumulation and information loss.The research question concerns memory architecture during complex manipulation tasks.
  • Physical Deployment: RQ5 measures success rate, execution cost, and failure profile on a multistep real-robot task.This assesses physical deployment behavior in addition to benchmark performance.
  • Task-Specific Priors: RQ6 evaluates whether task-specific planning priors influence trajectory efficiency and overall operational overhead.This isolates the contribution of planning priors to execution efficiency.

3.1 Experimental Setup

The evaluation spans tabletop and bimanual manipulation benchmarks and uses a common high-level planner with benchmark-specific instructions. The setup also adds agent-only camera views and increases interaction limits for multistage orchestration.

  • Benchmarks: The evaluation uses LIBERO, LIBERO-Plus, LIBERO-Pro, and RoboDojo for tabletop and bimanual manipulation.LIBERO, LIBERO-Plus, and LIBERO-Pro cover tabletop manipulation, while RoboDojo covers bimanual manipulation.
  • Common Planner: Across experiments, EMERGE-Policy uses Codex prowered by codex-5.6-sol as the high-level planner and fixed basic action primitives.Benchmark-specific AGENTS.md files adapt instructions to environment interfaces and task structures.
  • World-Model Pathway: Cosmos Policy serves as both an Operational Skill for executable action chunks and an Imagination Skill for future-frame prediction.An Evaluation Skill scores predicted outcomes, and greedy selection dispatches the highest-scoring candidate.
  • Observation Setup: Five additional fixed camera views improve agent-level scene perception and localization without changing WAM or VLA visual inputs.The added views are used only by perception and planning modules.
  • Interaction Budget: Per-episode step limits are moderately increased because multistage perception, planning, and execution typically require longer interaction trajectories.The adjustment prevents valid executions from being terminated by insufficient interaction budgets.

3.2 Results on Benchmarks

EMERGE-Policy reports strong performance across standard, perturbed, and implicit-instruction benchmarks relative to execution backends and selected baselines. However, ceiling effects, unmatched evaluation conditions, and non-isolating comparisons limit causal interpretation of the gains.

  • Standard LIBERO: 99.2% average success rate is achieved by the world-model variant on standard LIBERO, versus 98.5% for Cosmos Policy.The largest suite-level difference is on LIBERO-10: 98.6% versus 97.6%.
  • Standard LIBERO: 98.8% success is obtained by the π0.5 variant, compared with 96.8% for the underlying policy.These results compare the complete variant with its execution backend.
  • Interpretation Limits: Near-ceiling benchmark performance requires interpreting differences alongside trial counts and uncertainty estimates rather than as evidence of a broad breakthrough.Published-baseline comparisons are controlled only when observation, camera, and interaction budgets match.
  • LIBERO-Plus: 93.9% success on LIBERO-Plus exceeds 82.2% for Cosmos Policy and 93.2% for the reported RoboHarness result.The passage describes the complete system as robust under the evaluated perturbations.
  • LIBERO-Pro: 95.7% versus 20.0% on Spat-S and 59.1% versus 8.0% on L10-S are reported for EMERGE-Policy versus π0.5.These implicit-instruction results are consistent with high-level re-grounding and closed-loop orchestration, but the comparison does not isolate asynchronous cognitive decomposition.

3.3 Real-Robot Deployment

The real-robot deployment evaluates EMERGE-Policy on long-horizon, multi-layer cup stacking and under four interference settings. The system maintains high completion rates, with camera changes producing the largest observed decline.

  • The multi-layer cup-stacking task tests concurrent-agent collaboration, skill initiation and termination, actuator robustness, positioning, grasping, placement, and progress memory.
  • 100 real-robot experiments evaluated five task types, while 30 experiments covered each of four interference settings.
  • Camera perspective and quantity changes reduced success from 94% to 85%, the largest decline among the evaluated interference settings.
  • The closed-loop workflow coordinates Main Agent planning, Sub Agent perception and verification, and Operational Skill execution during three-tier pyramid construction.
  • Execution interruption still yielded an 86% success rate, cup-size changes of 5%-15% yielded 92%, and scene-color changes preserved 93%-94%.

3.4 Experiment Case Analysis

The case analysis examines externalized memory, imagination-evaluation action selection, and Main Agent coordination with execution skills. These mechanisms support bounded-context state tracking, perturbation-robust selection, and more efficient prior-guided execution.

  • Memory and Recovery: The three-tier memory architecture combines persistent task and environment files, structured Sub Agent evidence, and asynchronous visual verification for bounded-context state maintenance and correction.
  • Imagination and Evaluation Skills: The Evaluation Skill scores predicted candidate outcomes against subgoal criteria, while closed-loop execution performs only an initial action or short prefix before replanning.
  • Imagination and Evaluation Skills: The imagination-evaluation loop provides internal what-if assessment before physical execution, especially when execution is stochastic or environments are perturbed.
  • Imagination and Evaluation Skills: 93.9% versus 82.2% was achieved by the world-model variant over the raw Cosmos Policy backend on LIBERO-Plus perturbations, while open-loop selection averaged 85.1%.
  • Execution Skills and Main-Agent Planning: Prior-guided planning removes redundant move_to_pose cycles and reduces average planning steps per subgoal by 24.7% and wall-clock time per episode by 16.3%.
  • Execution Skills and Main-Agent Planning: With prior knowledge, success remained comparable at 96.7% versus 95.2% without prior while redundant moves decreased.

4 Related Work

Related work spans VLA and world-action models, code and agent harnesses, hierarchical closed-loop agents, and supervision or recovery systems. EMERGE-Policy distinguishes itself by explicitly organizing the information plane alongside capability invocation and control transfer.

  • Vision Language Action Models and World Action Models: VLA models map visual observations and language instructions to actions, while flow-based policies improve expressive action generation across tasks and embodiments.
  • Vision Language Action Models and World Action Models: World-action research connects prediction to action selection or rollout assessment, while EMERGE-Policy assigns Operational, Imagination, and Evaluation roles by function.
  • Agent and Harness Engineering: Code-as-Policies, ProgPrompt, RoboCodeX, and CaP-X connect language-model reasoning with executable robot programs, perceptual grounding, and interaction design.
  • Hierarchical and Closed-Loop Robot Agents: Hierarchical and closed-loop systems demonstrate the value of separating high-level decisions from low-level skills and incorporating environment feedback, but neither property alone distinguishes EMERGE-Policy.
  • Recent Robot Harnesses: Recent robot harnesses primarily organize capability execution and control transfer, whereas EMERGE-Policy additionally assigns perception, monitoring, verification, and memory processing to explicit Sub Agents.
  • Supervision, Outcome Assessment, and Recovery: Supervision and recovery systems assess progress, verify outcomes, and select retry strategies, forming a complementary line of work to EMERGE-Policy's orchestration.

5 Conclusion and Future Work

EMERGE-Policy organizes robot behavior as a coordinated system of perception, prediction, execution, verification, recovery, and memory rather than as a low-level policy alone. Reported LIBERO evaluations show improvements over π0.5 and Cosmos Policy backends, while future work targets reproducibility, trade-offs, and transfer.

  • Conclusion: EMERGE-Policy coordinates perception, prediction, execution, verification, recovery, and memory through a graph-structured agentic policy framework.A Main Agent maintains compact task state, while role-based Skills and isolated Sub Agents organize heterogeneous components into system-level behavior.
  • Conclusion: Operational, Imagination, and Evaluation Skills provide a shared interface for action production, future prediction, and outcome assessment.Verified failures are routed through localized Branch Stack recovery before global replanning.
  • Results: Across reported LIBERO evaluations, the complete system improves over its π0.5 and Cosmos Policy execution backends, especially in several perturbation and implicit-instruction settings.The reported gains are system-level, and attributing them to individual components requires matched ablations.
  • Future Work: Future work should make evaluation protocols reproducible, quantify accuracy–latency–token trade-offs, and test transfer across planners, low-level policies, tasks, and embodiments.The paper also identifies separating within-episode adaptation from persistent cross-episode learning as important for experience reuse without benchmark contamination.
Loading 2608.29896v1…