MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos

I. INTRODUCTION

• Household robots need to interpret open-vocabulary requests across diverse objects and environments, motivating learning from experience beyond their own interactions.Passive wearable-camera recordings capture everyday human interactions without requiring robot teleoperation or prescribed demonstrations.Source: §I · p. 1
• Passive videos lack explicit object coordinates, labeled places, and traversable connections, so spatial context must be reconstructed across time and viewpoints.Human and robot cameras, bodies, and available motions also differ, requiring actions to be grounded in robot geometry and kinematics.Source: §I · p. 1
• Human activity does not explicitly separate navigation from manipulation, and reaching an object’s vicinity does not ensure a manipulation-ready robot pose.The recordings do not label which movements bring a person closer versus enable the subsequent interaction.Source: §I · pp. 1–2
• MoCaP extracts reusable spatial memory and human pre-contact approach cues, then uses current observations and robot-grounded primitives to coordinate and revise navigation and manipulation.The agent is not limited to a predefined division between navigation and manipulation.Source: §I · p. 2
• 80% success on each of two mobile manipulation tasks; neither baseline succeeds, while spatial memory raises navigation success from 70% to 100%.The human approach prior found a reachable base pose in 5/5 trials, versus 0/5 for naive proximity-based docking.Source: §I · p. 2

A. Human Video as Scene and Interaction Experience

• Scene representations and recorded-video memories support search beyond the current view, but mobile manipulation also requires an interaction-ready approach configuration.Prior methods include spatial-semantic maps [4]–[6], relational video memories, and tour-conditioned navigation [8], [9].Source: A. Human Video as Scene and… · p. 2

B. Language-Guided Mobile Manipulation

• Language-guided systems select skills or compose executable programs, and execution feedback can support plan revision; MoCaP adds video-derived guidance for search and pre-contact positioning.CaP-X [3] centers on tabletop evaluation; MoCaP grounds base motion in the real robot and verifies arrival before manipulation.Source: B. Language-Guided Mobile Manipulation · p. 3

C. Base Placement for Mobile Manipulation

• Reachability maps are nearly symmetric around a target and miss contextual constraints such as free table sides; MoCaP uses a recorded human standing position to order base-pose search.Learning from successful rollouts [23] or reinforcement learning with reachability priors requires robot trials in each new environment; prior human torso-pose demonstrations are task-specific.Source: C. Base Placement for Mobile… · p. 3

A. Overview

• MoCaP takes a language instruction, passive human RGB recording, and live robot observations, with calibration, kinematics, and local costmaps available; learned components remain frozen.The recording supplies no metric camera poses or target coordinates.Source: A. Overview · p. 3
• The advisor uses the memory, instruction, and initial robot view to provide navigation subgoals and target cues, while a code agent revises short programs from fresh observations and execution feedback.The agent invokes robot-grounded primitives, and the human preparation prior guides docking.Source: A. Overview · p. 3, B. Offline Spatial and Interactive… · pp. 3–4

B. Offline Spatial and Interactive Memory Construction

• An offline builder organizes video into scene text and timestamped reference frames, retaining places, connections, landmarks, object relations, and uncertain inferences.Up to 32 frames support the descriptions; a separate event set records objects, pre-reach views, and contact frames before downstream instructions are supplied.Source: B. Offline Spatial and Interactive… · pp. 3–4

C. Code-as-Policy and Robot-Grounded Primitives

• Given the instruction, startup guidance, current visual and state summaries, and conversation history, the code agent generates Python programs that invoke robot-grounded primitives.Execution feedback and fresh observations inform the next code-generation step; failed calls interrupt the program so the agent can revise its plan.Source: C. Code-as-Policy and Robot-Grounded… · p. 4
• Navigation primitives dock to visible targets using a ground-plane standoff goal, while bounded motions support search and recovery when docking is unavailable.Nav2 handles navigation, and fresh perception verifies arrival.Source: C. Code-as-Policy and Robot-Grounded… · p. 4
• For grasping, the system converts proposed grasps into tool-center-point goals and rejects approaches whose open-gripper swept volumes intersect observed non-target geometry.GraspGen-X proposes grasps for the segmented target.Source: C. Code-as-Policy and Robot-Grounded… · p. 4

D. Approaching Object under Guidance from Human Prior

• Live depth aligns the human camera with the robot camera, and VGGT estimates camera intrinsics and relative poses from the robot image and recorded event views.The retrieved event is the first in H sharing a non-article word with the target description.Source: D. Approaching Object under Guidance… · p. 4
• The aligned camera position and optical axis are projected onto the ground plane to estimate the human approach position and heading.The valid depth pixels determine the metric scale.Source: D. Approaching Object under Guidance… · p. 4
• When the human is already close to the object, their heading gives a more stable approach estimate than relative position.The approach bearing is derived from the estimated human stance and target position.Source: D. Approaching Object under Guidance… · p. 4

E. Manipulation Preparation and Grasp Execution

• The preparation primitive searches nearby base configurations and filters candidate grasp poses with gripper-collision and IK checks before whole-arm trajectory planning.The target must be visible within 0.8 m; the local grid includes zero adjustment so an already suitable pose can proceed directly to grasping.Source: E. Manipulation Preparation and Grasp… · p. 4
• Candidate TCP goals are transformed from the current camera frame through the candidate camera frame into the selected arm-base frame.The transformation uses the fixed camera-to-arm-base calibration and the TCP-to-grasp-model transform.Source: E. Manipulation Preparation and Grasp… · p. 4
• Among feasible candidates, MoCaP favors small base adjustments and configurations with multiple robustly feasible grasps, storing the selected base pose with its grasp set.Candidates are shortlisted using workspace constraints and a lightweight geometric score.Source: E. Manipulation Preparation and Grasp… · pp. 4–5
• After base motion, MoCaP rechecks grasp feasibility with fresh observations and passes retained alternatives to cuRobo for whole-arm planning.Planning or execution failure returns control to the agent.Source: E. Manipulation Preparation and Grasp… · p. 5
• Spatial memory guides the search and the human approach prior guides initial placement, while live observations and robot-specific feasibility checks shape executable motions.The code agent combines these sources with execution feedback.Source: E. Manipulation Preparation and Grasp… · p. 5

IV. EXPERIMENTS

• The experiments ask whether MoCaP completes tasks that existing modular systems fail, whether scene memory reduces object-search effort, and whether the human approach prior improves reachable base placement.Source: §IV · p. 5

A. Experimental Setup

• The YOR platform combines an omnidirectional base, vertical lift, two gripper-equipped arms, and a stereo RGB-D camera.A Jetson Orin NX hosts the agent runtime and perception/planning services; Raspberry Pi services control the base and arms.Source: A. Experimental Setup · p. 5, Figure 3 · p. 5
• The system uses only egocentric RGB video recorded with Aria Gen 2 glasses, constructs spatial memory offline, and reuses it across tasks.The tasks are navigating to a blue trash bin and grasping a cardboard box or a can.Source: A. Experimental Setup · p. 5
• Trials use matched tasks, object placements, and robot starts with scene resets; outcomes are judged from physical execution and reported as successes/trials and success rate.Navigation is also evaluated by episodes completed within 180 and 300 seconds.Source: A. Experimental Setup · p. 5

B. Complete Mobile Manipulation

• CaP-X uses a geometric standing-pose heuristic, while CaP-X+ retains its interface but adds collision-aware base navigation.CaP-X+ checks goals against the online costmap and uses Nav2; on the can, both baselines search only by rotating in place and never localize the target.Source: B. Complete Mobile Manipulation · pp. 5–6
• MoCaP succeeds in 16/20 box trials and 8/10 can trials (80% each), while neither baseline succeeds.The box begins in view; the can starts outside the robot’s initial view and must first be located.Source: B. Complete Mobile Manipulation · p. 5, Table I · p. 6
• On the box, collision-aware navigation raises CaP-X+ grasp attempts from 2/20 for CaP-X to 8/20, but neither baseline succeeds.Several CaP-X+ attempts began too far from the box; other failures involved furniture collisions or a target outside the arm’s workspace.Source: B. Complete Mobile Manipulation · pp. 5–6

C. Spatial Memory and Object Navigation

• MoCaP succeeds in 10/10 kitchen-start navigation episodes, compared with 7/10 for no-prior, 2/10 for ApexNav, and 0/10 for CoW.Within 180 seconds, the counts are 6, 4, 1, and 0, respectively; within 300 seconds, they are 9, 6, 2, and 0.Source: C. Spatial Memory and Object Navigation · p. 6, Table II · p. 6
• CoW stops at a white bin with a blue notice, while ApexNav either keeps exploring after detecting the target or selects a nearby shredder.CoW’s localizer gives the distractor high confidence; ApexNav’s outcomes exceed the remaining time budget or identify the wrong object.Source: C. Spatial Memory and Object Navigation · p. 7

D. Effect of Human Approach Prior in Base Placement

• Readiness means that at least one proposed grasp passes gripper-collision checks and nominal IK from a base pose.Each trial uses docking followed by local preparation, with five trials per start and condition; neither condition can recover from a failed approach or retry from another direction.Source: D. Effect of Human Approach Prior in… · p. 7
• With the prior, readiness is reached in 38/45 trials, compared with 18/45 without it; at start A, results are 14/15 versus 0/15.At C, where the recorded human interaction occurs, the prior shows no advantage; the pooled comparison is not significant at this sample size.Source: D. Effect of Human Approach Prior in… · p. 7, Table III · p. 7, Figure 5 · p. 7
• The effect is concentrated at A, where the arrival direction without the prior most disagrees with the recorded human stance on the C side.The study compares matched starts with and without the prior, changing only the approach direction supplied to docking.Source: D. Effect of Human Approach Prior in… · p. 7, Figure 5 · p. 7

• MoCaP combines spatial memory, human approach cues, live perception, and robot-specific planning to let a real mobile manipulator complete tasks a tabletop coding agent could not.Source: §V · p. 7
• Evaluation covers one environment, one video recording, and three target objects with five to ten trials per condition, establishing feasibility rather than generality.Source: §V · p. 7
• Performance is constrained by potentially stale startup summaries, geometry-dependent local search, partial-depth collision checks, and grasp-and-lift-only manipulation.Source: §V · p. 7
• The authors propose testing priors across homes and recordings and guiding sustained contact, precise placement, or coordinated two-arm skills with human video.Source: §V · p. 7