PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniBrain-0: A Unified Agentic Model for Embodied Intelligence

Xiaomi Embodied Intelligence Team, Yuechen Luo, Chong He, Fang Li, Shaoqing Xu, Jingkai Wang, Menglun Su, Jiaxin Liu, Shengyin Jiang, Zhiyi Lai, Long Chen, Longlong Wang, Liang Hu, Haochen Liu, Yuncheng Jiang, Hongwei Xie, Jieming Cui, Hanbing Li, Junjie Huang, Zhixiang Duan, Hexin Zhang, Weixiang Liang, Ying Gong, Yuhan Wang, Yong Pan, Pengna Li, Kangyi Wu, Ken Deng, Kun Ma, Fuxi Wen, Zhi-xin Yang, Hangjun Ye

arXiv:2610.36013v1cs.CLcs.CV

TL;DR

Compact embodied agents need to learn task-level planning and recovery across manipulation and navigation, while powerful agents incur reasoning overhead and interaction data for feedback-driven correction remain limited. OmniBrain-0 combines a shared decision model, progressive expert distillation, and rubric-guided interaction-data refinement. The paper reports strong task performance with substantially lower token consumption and shorter task execution time in simulated manipulation and navigation benchmarks.

  • Problem

    Compact agents have limited interaction supervision for learning task decomposition, skill coordination, progress tracking, and adaptation to feedback across domains, while repeated powerful-model reasoning adds overhead and latency.

    Source: §1 · p. 4

  • Method

    OmniBrain-0 trains a shared manipulation-and-navigation decision model through embodied initialization, domain-specialized interaction distillation, MOPD consolidation, and rubric-guided refinement of reminders and verified trajectories.

    Source: §1 · p. 4

  • Results

    Simulated manipulation and navigation benchmarks show strong task performance with substantially lower token consumption and shorter task execution time.

    Source: §7 · p. 23

  • Takeaways & Limitations

    OmniBrain-0 consolidates planning, skill orchestration, and recovery into a compact shared decision model while retaining specialized low-level skills.

    Source: §7 · p. 23

    OmniBrain-0 remains dependent on low-level skill coverage and reliability, has gaps on unseen task compositions and spatial localization, and has limited real-world task evaluation.

    Source: §7 · p. 23

Abstract

from arXiv · show

General-purpose embodied intelligence requires robots to coordinate diverse skills over long horizons, track task progress, and recover from failures. Specialist policies alone do not ensure task-level reliability, while powerful multimodal agents rely on costly reasoning and external experience that compact models struggle to reproduce. We introduce OmniBrain-0, a lightweight unified embodied agent that coordinates manipulation and navigation through a shared high-level decision model, integrating task understanding, planning, progress tracking, and feedback-driven recovery. Our progressive training framework combines embodied pretraining, domain-specialized agent distillation, and multi-teacher on-policy distillation (MOPD). It distills powerful teachers' interaction experience into manipulation and navigation experts, then consolidates their decision capabilities into OmniBrain-0. Unification therefore occurs within the learned decision model rather than only at the tool interface. We further introduce Rubric-Guided Recursive Self-Improvement (RSI), a data flywheel in which a Meta-Agent turns execution evidence into refined rubrics, step-level reminders, and verified trajectories for domain-expert updates before their reconsolidation into OmniBrain-0 through MOPD. We evaluate OmniBrain-0 on LIBERO-Pro, LIBERO-Plus, RoboCasa365, RoboTwin 2.0, and R2R-CE Val-Unseen, as well as multiple real-world manipulation tasks. OmniBrain-0 combines strong cross-domain task performance with substantially lower inference and execution overhead than its closed-source teacher, demonstrating efficient transfer and consolidation of embodied capabilities into a compact unified agent.

1 Introduction

OmniBrain-0 targets reliable long-horizon coordination across manipulation and navigation, where specialist skill reliability alone does not ensure task success. It combines shared high-level decision-making with progressive expert distillation and rubric-guided data refinement.

1 Introduction

  • Effective compact-agent supervision must preserve task progression, skill coordination, execution feedback, and corrective decisions during recovery.Fixed demonstrations may not cover states encountered during autonomous execution, and powerful-model reasoning adds deployment overhead.Source: §1 · p. 4
  • OmniBrain-0 coordinates manipulation and navigation through a shared high-level planner for task understanding, long-horizon planning, skill orchestration, state tracking, and feedback-driven recovery.The planner conditions decisions on observations, instructions, and interaction history while retaining low-level specialization.Source: §1 · pp. 4–5
  • Progressive training initializes a compact model, distills interaction trajectories into manipulation and navigation experts, then consolidates their decisions through MOPD.The resulting unification is within learned high-level decisions as well as the skill-invocation interface.Source: §1 · p. 4
  • Rubric-Guided Recursive Self-Improvement uses execution evidence to refine reminders, verify trajectories, update domain experts, and reconsolidate their capabilities through MOPD.The updated model's subsequent interactions provide evidence for the next refinement and collection round.Source: §1 · p. 4
  • Evaluation covers LIBERO-Pro, LIBERO-Plus, RoboCasa365, RoboTwin 2.0, and R2R-CE Val-Unseen, alongside spatial-understanding, spatial-referring, and real-world manipulation assessments.The supplied passage identifies evaluation scope but does not report benchmark outcomes.Source: §1 · p. 5
  • The contributions combine a unified agent, structured interaction-data construction, and progressive specialization with multi-expert consolidation.The data pipeline preserves observation–decision–feedback dependencies and recovery while using challenge-driven collection, guidance refinement, and environment verification.Source: §1 · p. 5

2 Related Work

Prior work supplies specialist policies, tool-using agents, distillation methods, and recursive improvement strategies. OmniBrain-0 builds on these directions by learning a shared compact decision layer that coordinates domain-specific skills and feedback across manipulation and navigation.

2.1 Specialist policies for manipulation and navigation

  • Manipulation and navigation policies offer reusable domain skills, but their distinct interfaces and task structures motivate a shared decision layer for coordination.The passage reports degradation of π0.5 and LingBot-VLA 2.0 on long-horizon tasks under object and initial-position shifts [50].Source: §2.1 · p. 5

2.2 Tool-using embodied agents

  • Tool-using agents combine task decomposition with environmental feedback and recovery, while OmniBrain-0 internalizes planning, progress tracking, and recovery in a compact model learned from interaction experience.Prior harnesses include SayCan [2], Code as Policies, Inner Monologue [16], Harness VLA [61], RoboHarness [15], and Pigey [14].Source: §2.2 · p. 5

2.3 Agent distillation and expert consolidation

  • Trajectory tuning and hierarchical distillation transfer agent capabilities, while on-policy methods supervise student-generated sequences and MOPD consolidates domain-specialized teachers [1, 31, 35].OmniBrain-0 applies consolidation to manipulation and navigation expertise, including planning and recovery.Source: §2.3 · p. 5

2.4 Recursive self-improvement of embodied agents

  • OmniBrain-0 links rubric-guided reminder refinement and verified trajectory curation with domain-expert SFT and MOPD for a decision model shared across manipulation and navigation.Reminders are validated without parameter updates; subsequent training updates the agent, whose new interactions guide the next round.Source: §2.4 · p. 6

3 Unified Embodied Agent Architecture

OmniBrain-0 uses a shared high-level agent to invoke heterogeneous manipulation, navigation, perception, and motion tools through a unified feedback interface. Its decisions use current observations and interaction history, with optional experience-library priors, and adapt after each tool execution.

3.1 Problem Formulation and Overview

  • At each decision step, OmniBrain-0 selects tools using the task instruction, current observations, and interaction history, then updates decisions from environmental feedback.The objective is to fulfill the instruction within an execution budget under environment-defined completion criteria, while preserving distinct low-level action spaces and durations.Source: §3.1 · pp. 6–7

3 Unified Embodied Agent Architecture

  • The architecture separates task-level decision-making from specialized execution modules, connecting them through a unified invocation-and-feedback loop.The shared agent handles instruction understanding, long-horizon decomposition, orchestration, and recovery; execution modules provide perception, manipulation, navigation, and motion control.Source: §3.2 · p. 7

3.2 Skill Interfaces and Context Organization

  • Parameterized semantic tools expose a common invocation-and-feedback interface while retaining their native conditioning and low-level implementations.The tool set includes scene understanding and localization, manipulation/contact, navigation, and explicit pose or actuator adjustment.Source: §3.2 · p. 7

3.3 Closed-Loop Planning and Failure Recovery

  • The agent selects one tool per turn and makes its next decision after execution feedback, adapting localization, invocation parameters, or tool choice when execution deviates from expectations.Interaction history supports progress tracking, while an optional experience library supplies priors for task organization, tool composition, and recovery.Source: §3.3 · p. 7, §3.2 · p. 7

4 Embodied Capability Learning and Consolidation

OmniBrain-0 is trained through a three-stage transfer process: curated interaction data supports shared embodied initialization and domain specialization, then MOPD consolidates manipulation and navigation expertise into one student. The pipeline retains canonical successes and successful recoveries while filtering unreliable trajectories.

4.1 Interaction Data Construction (Data Pipeline)

  • The data pipeline normalizes interaction records, filters trajectory quality, reconstructs grounded reasoning, and validates consistency, retaining canonical successes and successful recoveries.Incomplete, unsuccessful, noisy, evidence-deficient, or inconsistent trajectories are discarded; recovery trajectories capture failure diagnosis, correction, and continuation.Source: §4.1 · pp. 8–9, Figure 4 · p. 8

4.2 Embodied Pretraining

  • Embodied question-answering and long-horizon interaction data establish OmniBrain-Base, a shared initialization for the domain teachers and unified student.Question answering supplies scene and task knowledge, while interactions guide multi-turn decisions using feedback.Source: §4.2 · p. 9

4.3 Domain-Specialized Agent Distillation

  • Manipulation and navigation teachers are initialized from OmniBrain-Base and trained on expert interaction trajectories to learn domain-specific goal understanding, skill orchestration, and recovery decisions.Supervision targets decision text and tool invocations; observations and tool feedback condition the decisions.Source: §4.3 · pp. 9–10

4.4 Multi-Expert Consolidation with MOPD

  • MOPD consolidates frozen manipulation and navigation experts into a unified student using masked reverse-KL supervision on student-generated prefixes, updating only the student.The resulting model uses one set of high-level agent parameters while retaining specialized low-level execution tools.Source: §4.4 · p. 10

5 Rubric-Guided Recursive Self-Improvement

Rubric-Guided Recursive Self-Improvement (RSI) turns execution evidence into guidance and verified training data, then uses expert training and MOPD to update OmniBrain-0 for another interaction round.

5 Rubric-Guided Recursive Self-Improvement

  • RSI links execution evidence, step-level guidance, verified trajectories, expert training, and MOPD to iteratively improve OmniBrain-0.The updated unified model is redeployed to generate evidence for the next round.Source: §5 · p. 11, §5.2 · p. 13

5.1 Rubric-Guided Reminder Policy

  • Successful and failed trajectories are verified and analyzed to add, revise, or reinforce rubrics for prerequisites, skill ordering, progress checks, and recovery.Trajectory patches connect rubric candidates to observed execution evidence.Source: §5.1 · p. 11, Figure 6 · p. 11
  • Selected rubric criteria become concise reminders that condition tool choices with task input, and are inserted only when the current context signals an issue.Reminders can specify relevant prerequisites, action order, or execution evidence.Source: §5.1 · pp. 11–12
  • Candidate reminders are evaluated with OmniBrain-0 parameters fixed; verifier feedback identifies effective reminders to retain and trajectories for later rubric updates.This stage refines guidance, while parameter updates occur through expert training and MOPD.Source: Figure 6 · p. 11, §5.1 · p. 12

5.2 RSI Data Flywheel

  • Challenging cases guide further collection, with trajectory selection considering task relevance, diversity, learning value, and difficulty.Rubric-guided reminders support interaction on the identified challenges.Source: §5.2 · pp. 12–13, Figure 7 · p. 12
  • Verified trajectories are curated into manipulation and navigation pools, while failed and inefficient executions remain available for diagnosis.Expert training retains environment-verified canonical successes and successful recoveries.Source: Figure 7 · p. 12, §5.2 · p. 13
  • The augmented domain experts provide domain-specific supervision for MOPD, which transfers their refined decision capabilities into OmniBrain-0 for redeployment.New interactions then supply evidence for the next rubric and reminder-policy update.Source: §5.2 · pp. 12–13

6 Experiments and Analysis

The experiments evaluate manipulation, navigation, spatial tasks, and physical manipulation under matched task settings, tools, and execution budgets. Across simulated benchmarks, OmniBrain-0 reports strong task performance with lower completion times and token use than the closed-source teacher.

6.2 Overall Performance

  • 65.5% overall success on LIBERO-Pro and 83.8% on LIBERO-Plus show strong performance across task and perturbation evaluations.On LIBERO-Pro, RSI reaches 69.3%; on LIBERO-Plus, it reaches 84.5%.Source: §6.2.1 · p. 14, Table 1 · p. 14, §6.2.2 · p. 14
  • 39.6% overall success on RoboCasa365 exceeds RLDX-1, while 88.2% on RoboTwin 2.0 exceeds LingBot-VLA and few-shot GLM-5.3-Flash.RSI raises RoboCasa365 success to 43.2% and narrows RoboTwin 2.0’s gap to GPT-5.6-Sol to 1.8 points.Source: §6.2.3 · p. 14, Table 3 · p. 15, §6.2.4 · p. 16

6.2.5 R2R-CE

  • 65.0% SR and 54.8% SPL on R2R-CE improve on the few-shot Qwen3.5-9B backbone, with 3.78 m NE.These comparisons use the 100-episode rand100 subset of Val-Unseen.Source: §6.2.5 · p. 16, Table 5 · p. 17

6.2.6 Spatial understanding and referring

  • 81.6% overall accuracy on EmbSpatial-Bench improves 4.4 points over the backbone, while RefSpatial-Bench transfer improves Placement and Unseen but lowers Location accuracy.The Placement and Unseen gains are 7.0 and 5.2 points, respectively.Source: §6.2.6 · p. 16, Table 6 · p. 17, Table 7 · p. 17

6.2.7 Real-world manipulation

  • 73.3% macro-average success across three real-world manipulation tasks exceeds π0.5’s 58.3%, but remains below GPT-5.6-Sol’s 86.7%.OmniBrain-0 reaches 75% on block stacking, 60% on apple placement, and 85% on drawer opening.Source: Figure 8 · p. 18, §6.2.7 · p. 18

6.2.8 Inference Token Efficiency

  • OmniBrain-0 uses 2.6×–12.7× fewer tokens than GPT-5.6-Sol across the five benchmarks, with task completion speedups ranging from 1.1× to 8.6×.End-to-end completion time also includes tool calls and low-level execution.Source: §6.2.8 · p. 18, Table 8 · p. 18

6.3 Ablation Studies

The ablations examine reminder guidance, data processing, and the Experience Library; qualitative examples trace task execution and reminder timing across simulated and physical settings.

6.3 Ablation Studies

  • The reminder-policy ablation compares fixed-checkpoint execution with and without state-dependent reminders under the same task splits, seeds, interfaces, and budgets.The supplied passages describe the comparison and the Figure 9 success-rate panels, but do not report the outcome values.Source: §6.3 · p. 18, Figure 9 · p. 18
  • Removing failure-recovery trajectories drops LIBERO-Pro success by 9%, while using expert reasoning as free-form text also causes a substantial performance drop.The authors attribute the latter to overfitting to expert linguistic style and weaker learning of visual grounding and decision logic.Source: §6.3 · p. 19, Figure 10 · p. 19
  • The Experience Library improves GPT-5.6-Sol success by 27.7, 6.8, and 26.4 points on LIBERO-Pro, RoboTwin 2.0, and RoboCasa365, respectively.The library also increases canonical and recovery trajectories available for distillation because only successful trajectories are retained for training.Source: §6.3 · p. 19, Figure 11 · p. 19

6.4.1 Task Execution Process

  • Qualitative examples span manipulation, navigation, and physical execution, illustrating object grounding, skill composition, progress tracking, and feedback-driven recovery.Examples include stove operation, block placement, household tasks, landmark-grounded routes, and real-world block stacking and apple placement.Source: §6.4 · pp. 19–20, §6.4.1 · pp. 20, 22

6.4.2 Reminder-guided execution

  • Reminder examples show guidance triggered selectively during execution, including before grasping and moving an arm, and during route selection and termination decisions.The illustrated sequences align observations, reminder triggers, and tool invocations.Source: §6.4.2 · pp. 22–23, Figure 17 · p. 23, Figure 18 · p. 23

7 Conclusion and Limitations

OmniBrain-0 consolidates manipulation and navigation decisions in a shared high-level model through progressive training and RSI. The authors identify low-level skill dependence, gaps in unseen compositions and localization, and limited real-world task coverage.

7 Conclusion and Limitations

  • OmniBrain-0 coordinates manipulation and navigation through a shared high-level decision model, with training that transfers and consolidates planning, orchestration, and recovery.The framework combines embodied pretraining, domain-specialized distillation, and multi-teacher on-policy distillation; RSI connects execution evidence, reminders, verified trajectories, and expert consolidation.Source: §7 · p. 23
  • Simulated benchmarks show strong task performance with lower token consumption and shorter completion times than the closed-source teacher, while spatial and real-world evaluations assess grounding and physical applicability.The authors conclude that compact models can learn transferable embodied decision-making capabilities while retaining specialized low-level skills.Source: §7 · p. 23
  • OmniBrain-0 remains dependent on low-level skill coverage and reliability; gaps persist on unseen task compositions and spatial localization, and real-world evaluation covers limited tasks.Future work aims to broaden task and embodiment coverage and strengthen grounding and recovery under distribution shifts.Source: §7 · p. 23

Appendix

The appendix provides supplementary detail on the agent, evaluation, training data, and task execution.

Appendix

  • Section A describes manipulation and navigation tools and the Experience Library.It organizes the system components and library.Source: Appendix · p. 30
  • Section B specifies benchmark settings, evaluation episodes, and metrics for simulated tasks, spatial tasks, and real-world manipulation.Source: Appendix · p. 30
  • Section C details VQA and agentic data, RSI trajectories, expert collection, model sizes, hardware, and training hyperparameters; Section D provides execution traces and comparisons with InternVLA-N1.Source: Appendix · p. 30

A Details of the Agentic System Design

The system uses standardized tools to connect the agent with heterogeneous skills and the environment, while an Experience Library supplies reusable interaction knowledge. Its four categories cover general principles, task patterns, failure recovery, and skill boundaries.

System design

  • The shared tool interface supports interaction with heterogeneous model-based and scripted skills, while the Experience Library organizes reusable expert knowledge and interaction experiences.The library is built from reviewed cross-episode experiences and provides reusable priors for later execution.Source: A Details of the Agentic System Design · pp. 30–31

Experience Library

  • General principles collect reusable perception, coordinate, gripper, controller, navigation, and safety guidance applicable across tasks.Examples include image-first decisions, object-to-end-effector offsets, landmark awareness, and short, capped motions.Source: A Details of the Agentic System Design · p. 31
  • Task execution patterns encode reusable recipes for manipulation and navigation, emphasizing task phases, skill composition, and verification gates rather than fixed scene coordinates.They cover behaviors such as pick–transport–place, articulated fixtures, multi-object ordering, and navigation.Source: A Details of the Agentic System Design · p. 31
  • Failure cases pair observable symptoms with recovery actions and execution budgets for manipulation and navigation failures.Examples include grasp failures, collisions, inaccurate localization, navigation deviations, and ambiguous landmark grounding.Source: A Details of the Agentic System Design · p. 31
  • Capability-boundary knowledge guides when to invoke learned VLA/VLN policies, switch between them, or use scripted tools instead.It captures conditions where learned skills are more reliable and where explicit geometric or rule-based actions are preferable.Source: A Details of the Agentic System Design · p. 31

B Benchmarks and Evaluation Protocol

The evaluation spans manipulation, navigation, spatial understanding, and real-world tasks under benchmark-specific episode and scoring protocols. The benchmarks include perturbations, held-out tasks, or continuous navigation settings to assess performance across varied conditions.

Evaluation protocol

  • Across closed-loop benchmarks, the planner receives instructions, observations, robot state or proprioception, and tool feedback; completion requires environment-confirmed success.Navigation uses NE, SR, OSR, and SPL on the same 100 episodes; failures and infrastructure issues are recorded separately.Source: B Benchmarks and Evaluation Protocol · pp. 31–33

LIBERO-Pro

  • LIBERO-Pro evaluates eight perturbation conditions across 800 episodes, with task or instruction redirection and spatial object-configuration swapping.Each condition contains ten tasks evaluated with seeds s1–s10.Source: B Benchmarks and Evaluation Protocol · p. 32

LIBERO-Plus

  • LIBERO-Plus samples 840 episodes across four suites and seven perturbation types, covering visual, linguistic, and physical changes.The perturbations include camera, robot, language, lighting, background, noise, and layout.Source: B Benchmarks and Evaluation Protocol · p. 32

RoboTwin 2.0

  • RoboTwin 2.0 evaluates 50 bimanual tasks in Clean and Randomized settings, using five seeds per task and 500 episodes overall.Success rates are reported separately for Clean and Randomized environments and overall.Source: B Benchmarks and Evaluation Protocol · p. 32

RoboCasa365

  • RoboCasa365 Target-50 evaluates 18 Atomic-Seen, 16 Composite-Seen, and 16 Composite-Unseen tasks across 340 episodes, with Overall averaging the 50 task-level success rates equally.This task-macro Overall differs from the episode-micro average used by the general rule.Source: B Benchmarks and Evaluation Protocol · p. 32

R2R-CE

  • R2R-CE evaluates 100 Val-Unseen episodes in continuous Habitat space, with ground-truth shortest paths and goal positions used only for scoring.The planner receives instructions, camera observations, and execution feedback, and uses SmartWay for navigation.Source: B Benchmarks and Evaluation Protocol · p. 32

Spatial benchmarks

  • Spatial tests report accuracy on 3,640 EmbSpatial-Bench questions and correct-point percentages on RefSpatial-Bench Location, Placement, and Unseen categories.RefSpatial Overall weights the three categories by their sizes; Unseen overlaps the other task sets.Source: B Benchmarks and Evaluation Protocol · p. 33

Real-world manipulation

  • The real-world suite tests block stacking, apple placement, and drawer opening over 20 trials per task, reporting per-task and macro-average success rates.A trial counts as successful when the task is completed within its execution budget.Source: B Benchmarks and Evaluation Protocol · p. 33

C Datasets and Experimental Details

The training data combines embodied pretraining with verified interaction trajectories from expert collection and the RSI flywheel. Experimental configurations specify the collection models, low-level policies, and progressive training stages.

Training data

  • Embodied pretraining combines selected spatial grounding, reasoning, affordance, and planning data from four sources with agentic interaction data, yielding approximately 820K samples.The source datasets include EmbSpatial-SFT, RefSpatial, ShareRobot, and RoboInter-VQA.Source: C Datasets and Experimental Details · p. 33
  • The agentic corpus preserves temporally aligned instructions, observations, robot states, decisions, tool calls, and feedback, retaining verified successful trajectories from initial collection and RSI.Usable expert demonstrations comprise 2,349 from LIBERO/LIBERO-Pro, 513 from RoboTwin 2.0, 880 from RoboCasa365, and 1,246 from R2R-CE.Source: C Datasets and Experimental Details · p. 33
  • The corpus composition figure reports 820,453 VQA samples, 3,846 initial agentic demonstrations, and 1,142 additional RSI demonstrations.Percentages indicate each source’s share within its corresponding corpus.Source: C Datasets and Experimental Details · p. 33, Figure C.1 · p. 34

Expert collection

  • GPT-5.6-Sol and GLM-5.3-Flash collect closed-loop trajectories using an experience library; GPT-6.1-Sol serves as RSI Meta-Agent and GPT-5.6-Sol as reminder-policy judge.Manipulation skills include π0.5, LingBot-VLA, and RLDX-1; navigation uses SmartWay.Source: C Datasets and Experimental Details · p. 34

Model configuration

  • The manipulation policies contain approximately 3.617B, 4.197B, and 6.913B parameters for π0.5, LingBot-VLA, and RLDX-1, respectively; SmartWay’s waypoint predictor has 54M.Manipulation counts cover complete deployed policy weights, while SmartWay’s count covers only its waypoint prediction component.Source: C Datasets and Experimental Details · p. 34

Training configuration

  • Training progresses through embodied pretraining, domain-specialized distillation, and MOPD, with reverse-KL distillation and equal VLA/VLN rollout weights in MOPD.The reported schedule uses 2, 3, and 2 epochs for the respective stages.Source: C Datasets and Experimental Details · p. 34

D More Qualitative Examples

The qualitative traces show the agent connecting task stage and interaction state to evidence for its next action, then verifying outcomes. Examples span grounded manipulation, multi-object progress tracking, and navigation with route adjustment.

Execution traces

  • Manipulation traces use Phase, State, Evidence, and Decision fields to connect task stage and interaction state to the next action, followed by tool use and verified termination.Across the examples, the traces show object and destination grounding, specialist-skill invocation, and intermediate state tracking.Source: D More Qualitative Examples · pp. 34–35, Figure D.1 · p. 36, Figure D.2 · p. 37

LIBERO-Pro

  • In the LIBERO-Pro example, the agent localizes the dressing and basket, grasps and transports the bottle, then uses placement evidence and the environment success signal to terminate.The grasp is established with a VLA skill, followed by explicit arm motion above the basket.Source: D More Qualitative Examples · p. 35, Figure D.1 · p. 36

RoboTwin 2.0

  • In the RoboTwin 2.0 example, the agent tracks each gripper separately and continues after releasing one loaf, terminating only after both loaves are placed.It retains the right-hand grasp while completing the second placement.Source: D More Qualitative Examples · p. 35, Figure D.2 · p. 37

RoboCasa365

  • In the RoboCasa365 example, the agent distinguishes the completed croissant placement from the remaining jam-jar sub-goal and checks both objects together at the end.The sequence spans mobile-base adjustment, manipulation, and successive placements.Source: D More Qualitative Examples · p. 35, Figure D.3 · p. 38

R2R-CE

Related papers

Loading 2610.36013v1…