PaperlayerView exact revision on arXiv

Source-linked AI summary

Cracking Mobile Manipulation: From Failure Diagnosis to Agentic Robot System

Zixuan Wang, Jia-Ning Li, Yinzhou Tang, Yanzhao Yu, Biao Wu, Meng Fang, Xin Li, Yiwu Zhong

arXiv:2610.19600v1cs.RO

TL;DR

Long-horizon mobile manipulation remains difficult despite strong local subtask performance, with failures spanning the execution process. The paper diagnoses these bottlenecks and adds targeted interventions around a fixed subtask VLA; success improves across simulated benchmarks and real-robot tasks.

  • Problem

    Strong local manipulation performance does not readily translate to long-horizon mobile manipulation, where reliable execution requires carrying skills through sequences of decisions and physical states.

    Source: §1 · pp. 1–2

  • Method

    The authors diagnose failures in planning, navigation and base positioning, and local manipulation, then add demonstration guidance, visual memory, accessibility-aware base adjustment, and end-effector adjustment around a fixed subtask VLA.

    Source: §1 · p. 2

  • Results

    Interventions improve success across RoboCasa365, BEHAVIOR-1K, and real-robot tasks, including held-out RoboCasa365 success of 49% versus 18% for the baseline.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results suggest that mobile-manipulation reliability can be improved by diagnosing and targeting the current bottleneck rather than uniformly scaling the underlying policy.

    Source: §1 · p. 2, §6 · p. 10

    The accessibility check considers target distance but does not validate joint limits, inverse kinematics, or full-body collisions.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Recent vision-language-action (VLA) models have shown strong performance on local robot manipulation, yet their success does not readily translate to long-horizon mobile manipulation. In our experiments, a pretrained subtask VLA achieves 80% success rate on atomic manipulation tasks but only 19% when these skills are composed into mobile manipulation. We argue that this gap stems from failures across the execution process rather than from local manipulation capability alone. To uncover these bottlenecks, we systematically diagnose failures in long-horizon mobile manipulation and identify three dominant regimes: task planning, navigation and base positioning, and local manipulation. Based on this diagnosis, we develop failure-guided agentic intervention that augments a fixed subtask VLA with demonstration-based planning guidance, adaptive visual memory, accessibility-aware base adjustment, and local end-effector adjustment. On RoboCasa365, these interventions progressively improve task success from 19% to 47%, with the finalized system reaching 49% on held-out episode seeds, compared with 18% for the baseline. The same design improves performance from 14% to 37% on BEHAVIOR-1K and from 5/20 to 13/20 on real-robot tasks. These results suggest that reliable mobile manipulation can be advanced by diagnosing and intervening on the current bottleneck, rather than uniformly scaling the underlying policy. Our code will be released at https://github.com/PKU-VaLuE-Lab/Cracking-Mobile-Manipulation.git.

1 Introduction

Strong atomic manipulation does not ensure reliable long-horizon mobile manipulation. The paper diagnoses failures across execution stages and adds targeted interventions around a fixed subtask VLA.

1 Introduction

  • A pretrained subtask VLA achieves 80% success on atomic tasks but only 19% when skills are composed into long-horizon mobile manipulation.Task progress, base configuration, and viewpoint change as the robot moves between workspaces.Source: §1 · p. 1
  • Progressively adding failure-guided interventions raises RoboCasa365 success from 19% to 47%, with held-out success reaching 49% versus 18% for the baseline.The same design raises success from 14% to 37% on BEHAVIOR-1K and from 5/20 to 13/20 on real-robot tasks.Source: §1 · pp. 2–3
  • Failure diagnosis identifies task planning, navigation and base positioning, and local manipulation as the dominant regimes.The failures can occur before manipulation, at the navigation-to-manipulation handoff, or during the final interaction.Source: §1 · pp. 2–3
  • The framework keeps the subtask VLA fixed and targets diagnosed bottlenecks with demonstration-based planning guidance, adaptive visual memory, base adjustment, and end-effector adjustment.The interventions address task procedures and progress, accessibility of the current base pose, and residual spatial misalignment.Source: §1 · pp. 2–3

2 Related Work

Prior mobile-manipulation work spans modular systems, learned policies, failure diagnosis, and agentic tools. This paper focuses on mapping failures across an agentic system and using that diagnosis to guide interventions.

2.1 Mobile Manipulation Systems and Policies

  • Mobile-manipulation systems range from modular perception, navigation, and manipulation pipelines to learned policies coupling mobility and manipulation.Recent work also targets mobile manipulation with fixed-base VLA transfer, whole-body action prediction, spatially enriched observations, and action-conditioned world models.Source: §2.1 · p. 3

2.2 Robot Failure Diagnosis and Recovery

  • Failure benchmarks, diagnosis methods, and integrated recovery systems address capability evaluation, failure causes, or corrective action.This paper instead analyzes where failures arise in an agentic mobile-manipulation system, guides interventions from that diagnosis, and tracks shifts in remaining failures.Source: §2.2 · p. 3

2.3 Agentic Robots, Demonstrations, and Memory

  • Video demonstrations can specify procedures, while memory and reusable skills help agents retain progress and reuse experience.The framework uses a video-based skill library and visual memory as targeted interventions on top of a fixed subtask VLA.Source: §2.3 · p. 4

3 Failure Diagnosis of Agentic Mobile Manipulation

The diagnosis studies long-horizon mobile-manipulation failures in RoboCasa365 and attributes failed episodes to unresolved errors. It identifies three sequential bottlenecks: task progress, manipulation-ready base pose, and precise local interaction.

3.1 Diagnostic Setting

  • The diagnostic setting covers 20 RoboCasa365 composite tasks with 2–15 subtasks requiring movement between workspaces and object or articulated-object interactions.The baseline uses a high-level planner and a shared subtask VLA trained on atomic-task demonstrations and subtask segments.Source: §3.1 · p. 4
  • The subtask VLA succeeds on 80% of atomic tasks, while the agentic system succeeds on only 19% of mobile-manipulation tasks.The gap motivates locating the system's failure points.Source: §3.1 · p. 4

3.2 Failure Attribution

  • Attribution excludes errors recovered within three replanning attempts and labels the earliest unresolved error that prevents task completion.Offline diagnosis uses rollout videos, planner history, robot states, and simulator object states; simulator states are unavailable to the online agent.Source: §3.2 · p. 4

3.3 Failure Analysis

  • Among 81 failed episodes, primary failures are attributed to task planning (21.0%), navigation and base positioning (25.9%), and local manipulation (53.1%).The evaluation covers 20 mobile-manipulation tasks, each with five rollouts.Source: §3.2 · p. 4, Figure 2 · p. 5
  • Planning failures involve wrong targets or task order, often when early actions change the scene and invalidate later decisions.Reliable planning requires tracking completed and remaining subtasks and how past actions changed the scene.Source: §3.3 · p. 5
  • Reaching the correct workspace can still leave the robot at a base pose that makes the next manipulation difficult or infeasible.The base pose shapes both the arm's reachable workspace and its viewing geometry.Source: §3.3 · p. 5
  • Even after planning and navigation succeed, localization errors, grasp misalignment, or inaccurate end-effector positioning can derail contact-rich interactions.The findings form sequential weak links, motivating interventions at the stage where each bottleneck arises.Source: §3.3 · pp. 5–6

4 Failure-Guided Agentic Intervention

The framework keeps the subtask VLA fixed and adds tools that target diagnosed planning, memory, and geometric bottlenecks. Its components supply task guidance, retrieve prior observations, and adjust base or end-effector pose.

4 Failure-Guided Agentic Intervention

  • The system adds failure-guided interventions around a fixed subtask VLA, using a skill library, visual memory, and geometric skills.Geometric skills check accessibility and adjust the base or end-effector pose.Source: §4 · p. 6, Figure 3 · p. 6

4.1 Skill Library from Video Demonstrations

  • A skill library retrieves a demonstration and summarizes it as a skill card with the goal, subtask order, keyframes, and completion criteria.The planner conditions its next-step decision on both the skill card and current observation and action history.Source: §4.1 · p. 6

4.2 Adaptive Visual Memory

  • Adaptive visual memory stores current-episode observations and retrieves relevant past frames when the planner requests information.It returns the 16 most relevant frames with a short trajectory summary and cannot access future frames or simulator states.Source: §4.2 · p. 6

4.3 Geometric Skills

  • Geometric skills check target accessibility, adjust the base when needed, and apply a local end-effector correction for residual misalignment.Accessibility is based on target distance; marginal stances trigger a small local base correction, while out-of-reach stances trigger selection and navigation to a new stance.Source: §4.3 · p. 7
  • End-effector adjustment estimates a target position from masked depth points and applies a bounded correction toward an offset pose along an estimated surface normal.Translation is limited to 0.3 m per call, and the geometric skill changes only the local pose while leaving the subtask VLA fixed.Source: §4.3 · p. 7

5 Experiments and Results

The fixed subtask VLA is paired with targeted interventions that raise success on development and held-out RoboCasa365 episodes, with gains also reported on BEHAVIOR-1K and real-robot tasks. Progressive integration shifts the dominant remaining failures toward later-stage manipulation, while ablations examine the intervention design.

5.2 Progressive Integration

  • Progressive integration raises RoboCasa365 development success from 19.0% to 47.0%, with base adjustment contributing the largest incremental gain, from 29.0% to 40.0%.Configurations add intervention groups successively while keeping the underlying subtask VLA fixed.Source: §5.2 · p. 8, Figure 4 · p. 8
  • As planning and navigation failures decrease, manipulation becomes the dominant remaining failure type, revealing a moving failure frontier.The progressive failure counts use the same 100 matched episodes; targeted fine-grained errors also fall, including placement assignment errors (9→0), poor approaches or base positioning (11→1), and grasp misalignment or empty grasps (10→3).Source: §5.2 · p. 8, Figure 5 · p. 8
  • Remaining failures include incorrect placement, object drops, and collision-induced scene disruption, which current geometric corrections do not fully address.These errors occur later in execution than the failure modes targeted by the interventions.Source: §5.2 · p. 9

5.3 Held-Out Evaluation

  • On held-out RoboCasa365 episode seeds, the full system improves success from 18.0% to 49.0% over the agentic baseline.The comparison uses unseen scenes and object initializations after system design, prompts, and tool parameters were fixed.Source: Table 1 · p. 9, §5.3 · p. 9
  • On five BEHAVIOR-1K tasks, success rises from 14.0% to 37.0%, and real-robot success rises from 5 of 20 to 13 of 20 trials.The BEHAVIOR-1K evaluation uses low-level policies trained on its training data while retaining the agentic intervention design.Source: §5.3 · p. 9

5.4 Ablation Study

  • With in-domain demonstrations, removing visual keyframes reduces success to 43.0%, while replacing them with HIW-500 demonstrations reduces success from 47.0% to 40.0%.The results suggest domain alignment and visual context contribute to demonstration-based guidance.Source: §5.4 · pp. 9–10, Table 2 · p. 10
  • Planner-triggered visual-memory retrieval outperforms always-provided history across tested frame budgets, while more frames do not consistently improve performance.The adaptive design retrieves frames selected for relevance to the planner's query.Source: §5.4 · p. 10
  • Removing the accessibility check causes the largest base-adjustment degradation, while RGB-D geometric grounding yields 47.0% success versus 43.0% for planner-specified end-effector targets.Removing stance replanning has a larger effect than removing local base-position correction.Source: §5.4 · p. 10

6 Conclusion

The paper diagnoses long-horizon mobile-manipulation failures across planning, navigation and base positioning, and local manipulation, then targets those bottlenecks while keeping the subtask VLA fixed. Results across simulation benchmarks and real-robot tasks support failure diagnosis and targeted intervention as an alternative to uniformly strengthening the underlying policy.

6 Conclusion

  • The failure-guided interventions improve long-horizon task success across RoboCasa365, BEHAVIOR-1K, and real-robot experiments while leaving the underlying subtask VLA fixed.The interventions include demonstration-based planning guidance, adaptive visual memory, accessibility-aware base adjustment, and local end-effector adjustment.Source: §6 · p. 10
  • The analysis identifies task planning, navigation and base positioning, and local manipulation as three dominant failure regimes.These regimes arise as subtasks are composed in long-horizon mobile manipulation.Source: §6 · p. 10
  • The authors argue that diagnosing changing bottlenecks could provide a systematic path toward more reliable mobile manipulation in long-horizon, multi-stage environments.They frame failure diagnosis as a tool for designing agentic embodied systems.Source: §6 · p. 11

AI Use Statement

The paper reports that generative AI assisted with literature collection, manuscript drafting, language polishing, and parts of code implementation and debugging. It evaluates mobile-manipulation tasks and uses a hierarchical agentic baseline in which a planner directs a shared subtask VLA.

AI Use Statement

  • Generative AI assisted with literature collection, partial manuscript drafting and language polishing, and implementation and debugging of parts of the experimental code.The authors state that the research ideas, methodology, experimental design, analysis, and conclusions were developed and verified by them, and that all reported results came from their experiments.Source: AI Use Statement · p. 11

A Mobile Manipulation Evaluation Tasks

  • The evaluation covers 20 RoboCasa365 mobile-manipulation tasks with 2–15 subtasks each, totaling 91 subtasks and averaging 4.55 subtasks per task.Tasks include navigation between workspaces, pickup and placement, transport, articulated-object interaction, and manipulation around appliances and receptacles.Source: A Mobile Manipulation Evaluation Tasks · p. 18

B.1 Agentic Baseline

  • The agentic baseline uses a high-level planner to select subtask instructions or tools, while a shared subtask VLA executes navigation and manipulation.The planner observes the scene after each 350-control-step VLA call and decides whether to continue, retry, or proceed.Source: §B.1 · p. 18

B.2 Subtask VLA Training

This material details the fixed subtask VLA, the failure-attribution procedure, and the implementation and evaluation of failure-guided interventions. On held-out episodes, the full system outperforms the agentic baseline, while analysis identifies manipulation as the dominant remaining failure category.

B.2 Subtask VLA Training

  • The subtask VLA jointly trains on atomic-task demonstrations and segmented composite-task demonstrations, uses the same checkpoint for navigation and manipulation, and achieves 80% success on RoboCasa365 atomic-task evaluation.Its training split contains 16,181 composite demonstrations and 84,401 valid subtask segments; no evaluation trajectory is used for policy training.Source: §B.2 · pp. 18–19, Table 3 · p. 18

C.1 Attribution Procedure and Annotation Consistency

  • Three reviewers agreed on 26 of 30 primary failure labels (86.7%; Fleiss’ κ = 0.86) and unanimously agreed on 24 of 30 fine-grained labels (80.0%; κ = 0.79).Blinded majority fine-grained labels matched the original attribution in 27 of 30 cases (90.0%); disagreements mainly involved closely related execution failures.Source: §C.1 · p. 20, Table 5 · p. 20, Table 6 · p. 20

D Skill Implementation Details

  • Demonstration-derived skill cards provide an ordered plan and keyframes, while adaptive recall retrieves query-relevant past frames and geometric skills use RGB-D grounding to support base and end-effector correction.The planner is instructed to treat demonstrations as guidance rather than copy coordinates, and to use current observations to verify state and recover from grasp or placement errors.Source: §D.1 · p. 21, §D.2 · p. 21, §D.3 · p. 21

E.1 Progressive Integration

E.2 Leave-One-Out and Held-Out Evaluation

  • After the design was fixed, the full system improved from 18.0% to 49.0% over the agentic baseline on held-out episodes, a paired difference of 31 percentage points (p < 0.001).The held-out comparison uses paired episodes.Source: Table 9 · p. 23, §E.2 · p. 23

E.3 Failure Category of Recovered Episodes

  • Recovered episodes largely matched the intended mapping: skill library and adaptive visual memory with planning failures, base adjustment with navigation/base-positioning failures, and end-effector adjustment with manipulation failures.Because interventions can affect later stages, the analysis is episode-level rather than a one-to-one causal decomposition.Source: §E.3 · p. 23, Table 10 · p. 23

F Paired Failure Transitions

  • The full system recovered 30 of 81 baseline failures (37.0%) and retained 17 of 19 baseline successes (89.5%); recovery was 9/17 for planning, 12/21 for navigation/base positioning, and 9/43 for manipulation.Six planning failures and six navigation/base-positioning failures became manipulation failures, illustrating a survivor effect; these conditional episode-label rates do not estimate stage-specific success or attribute recovery to one tool.Source: Table 11 · p. 23, F Paired Failure Transitions · p. 24

G Fine-Grained Failure Attribution of the Full System

  • Among 53 failed full-system episodes, 47 were attributed to manipulation, five to navigation and base positioning, and one to task planning.The manipulation failures mainly involved incorrect placement, object drops during transport, and collision-induced scene disruption.Source: G Fine-Grained Failure Attribution of… · p. 24, Figure 6 · p. 24

H Per-Task and Horizon Results

The full system improves over the baseline in both task-horizon groups, while observed success is lower for longer tasks. A compounded-success reference is also reported alongside the observed agentic baseline.

H Per-Task and Horizon Results

I Cross-Benchmark Evaluation on BEHAVIOR-1K

The framework is evaluated on five BEHAVIOR-1K challenge tasks, with low-level policies retrained on that benchmark’s training data. The full system outperforms Harness VLA across all five tasks and reaches a higher overall success rate.

I Cross-Benchmark Evaluation on BEHAVIOR-1K

J Real-Robot Evaluation

On two real-robot tasks, the full system outperforms the agentic baseline, following the same overall trend as simulation. The evaluation uses a fine-tuned π0.5 subtask VLA and a high-level planner, with added system interventions in the full system.

J Real-Robot Evaluation

  • Storage Box Transport requires pickup, transport while navigating, and placement; Table Service comprises multiple manipulation stages.Source: J Real-Robot Evaluation · p. 26
  • The real-robot tasks use a π0.5 checkpoint fine-tuned on demonstrations, converted into subtask-level training examples for the agentic methods.The study collected 500 Storage Box Transport and 300 Table Service demonstrations; Kimi K3 plans from the current observation and execution history.Source: J Real-Robot Evaluation · p. 26
  • The full system succeeds in 13/20 real-robot trials versus 5/20 for the agentic baseline, with improvement observed on both tasks.The two tasks are Storage Box Transport and Table Service; Table 15 reports 10 trials per task.Source: J Real-Robot Evaluation · p. 27, Table 15 · p. 27
  • A representative successful Table Service rollout transfers the dish, adds topping, and places the wine glass beside it.Source: J Real-Robot Evaluation · p. 27, Figure 7 · p. 27

K Leave-One-Out and Design Ablations

Leave-one-out results identify base adjustment as the intervention whose removal produces the largest observed decrease. Fine-grained results support adaptive memory and accessibility checks, while memory access timing and frame selection each contribute to the adaptive-memory design.

K Leave-One-Out and Design Ablations

K.1 Fine-Grained Ablations

  • Adaptive memory has higher observed success than uniform-history memory at all three frame budgets, while removing the accessibility check produces the clearest within-base-adjustment drop.The planner-specified versus geometrically estimated end-effector target difference is smaller, and its confidence interval includes zero.Source: §K.1 · p. 28, Table 17 · p. 28

K.2 Separating Memory Access and Frame Selection

  • Relevance-based frame selection improves observed success under both access settings, and planner-triggered access improves over providing memory every turn.Combining planner-triggered access with relevance-based retrieval gives the highest observed success; configurations use a 16-frame budget and matched episodes.Source: §K.2 · p. 28, Table 18 · p. 28

L Qualitative Execution Examples

  • Four representative rollouts illustrate standard multi-stage execution, navigation-manipulation coordination, base adjustment, and end-effector correction after failed placement.Source: L Qualitative Execution Examples · p. 29

Task Instruction

  • The coordinated task example composes cabinet opening, bread pickup and placement, basket transport, navigation, and final placement using the same subtask VLA.Source: Figure 10 · p. 30
  • After an initial placement fails, the planner invokes local end-effector adjustment and retries with the same subtask VLA.Source: Figure 11 · p. 30

Related papers

Loading 2610.19600v1…