Source-linked AI summary
Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V
Peiyuan Zhi, Zhiyuan Zhang, Yu Zhao, Muzhi Han, Zeyu Zhang, Zhitian Li, Ziyuan Jiao, Baoxiong Jia, Siyuan Huang
TL;DR
Open-vocabulary mobile manipulation requires robots to reason and replan in open environments. COME-robot combines GPT-4V with multi-level perception and closed-loop recovery, improving goal completion over baselines in mobile and tabletop settings.
Problem
Open-vocabulary mobile manipulation requires robots to understand natural-language instructions and act across complex scenes, while existing approaches face limits in open-ended exploration and long-horizon recovery.
Source: §I · p. 1Method
COME-robot combines GPT-4V with multi-level open-vocabulary perception and situated reasoning, plus closed-loop feedback that verifies feasibility and execution and traces failure causes.
Source: §I · p. 2, §III · p. 2Results
COME-robot improves goal completion in both mobile and tabletop settings, achieving 65% success versus CaP* at 30% in mobile manipulation.
Source: B. Experimental Results and Analyses · p. 5Takeaways & Limitations
Completing open-ended tasks requires both LLM commonsense knowledge and interactive exploration that updates scene information.
Source: D. Discussions · p. 6
Abstract
from arXiv · showhide
Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. In this work, we present COME-robot, the first closed-loop robotic system utilizing the GPT-4V vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios.COME-robot incorporates two key innovative modules: (i) a multi-level open-vocabulary perception and situated reasoning module that enables effective exploration of the 3D environment and target object identification using commonsense knowledge and situated information, and (ii) an iterative closed-loop feedback and restoration mechanism that verifies task feasibility, monitors execution success, and traces failure causes across different modules for robust failure recovery. Through comprehensive experiments involving 8 challenging real-world mobile and tabletop manipulation tasks, COME-robot demonstrates a significant improvement in task success rate (~35%) compared to state-of-the-art methods. We further conduct comprehensive analyses to elucidate how COME-robot's design facilitates failure recovery, free-form instruction following, and long-horizon task planning.
I. INTRODUCTION
Open-vocabulary mobile manipulation requires robots to understand complex 3D scenes, align them with language, and recover from failures during long-horizon execution. COME-robot combines GPT-4V reasoning with multi-level perception and closed-loop recovery for real-world tasks.
I. INTRODUCTION
- Robots must align natural-language instructions with complex 3D environments and diagnose and recover from errors to complete long-horizon mobile manipulation tasks.These challenges require structuring and grounding scene understanding for reasoning and reliable execution.Source: §I · p. 1
- Existing closed-loop systems use predefined symbols, while many VLM approaches remain largely confined to tabletop settings and leave exploration under partial observability unaddressed.These limits reduce their effectiveness for open-ended, long-horizon mobile manipulation.Source: §I · p. 1
- COME-robot combines multi-level open-vocabulary perception and situated reasoning with closed-loop feedback and restoration to explore scenes, identify targets, and trace failure causes across modules.The system considers dependencies such as navigation and local manipulation when diagnosing failures.Source: §I · p. 2
- In real-world experiments involving eight challenging OVMM tasks, COME-robot significantly outperformed existing LLM-based robotic systems in all tasks.The authors also analyze system failure modes and recovery.Source: §I · p. 2
II. RELATED WORK
Prior work combines learned planners and robotic skills, but long-horizon mobile manipulation remains difficult because sequential errors and execution failures require recovery across modules.
II. RELATED WORK
- End-to-end manipulation approaches [3], [2], [14] are typically limited to short sequences in relatively simple tabletop settings, while planners generate plans for robot primitive skills [31], [22].Source: §II · p. 2
- On the HomeRobot OVMM benchmark, OK-Robot [16] showed that naively combining a VLM-based high-level planner with a low-level executor can produce sequential errors.Source: §II · p. 2
- Closed-loop systems use feedback for task verification and replanning, but often retry execution failures without tracing interdependencies between modules [22],, [9],, [27],,, [11], [25], [18], [10].COME-robot instead focuses on tracing failure causes across modules for recovery.Source: §II · p. 2
III. METHOD
COME-robot organizes open-vocabulary perception and situated reasoning into staged scene exploration and target identification, then uses GPT-4V to select task-relevant objects from observed context.
A. Open-Vocabulary Perception and Situated Reasoning
- The environment explorer builds a global map of large furniture, then maps small objects locally and captures close-up images to extract fine-grained target attributes.Global exploration uses frontier-based scanning; local perception focuses on instruction-relevant regions.Source: A. Open-Vocabulary Perception and… · pp. 2–3
III. METHOD
- COME-robot’s planner pairs open-vocabulary perception and reasoning with closed-loop feedback and restoration to ground open-ended instructions in the environment.Figure 2 distinguishes planned actions in blue, identified failures in red, and post-observation or verification analysis in green.Source: §III · p. 2, Figure 2 · p. 3
A. Open-Vocabulary Perception and Situated Reasoning
- GPT-4V reasons about target location and object function and availability to identify a goal object suited to the task.For example, it selects an empty cup for pouring water.Source: A. Open-Vocabulary Perception and… · p. 3
B. Closed Loop Feedback and Restoration
COME-robot verifies whether tasks can proceed and whether actions succeeded, then uses a hierarchical recovery mechanism to restore progress after infeasibility or execution and perception failures.
B. Closed Loop Feedback and Restoration
- The status verifier checks task feasibility and execution success, using perception, VLM dialogue, wrist-camera images, and action-function feedback to diagnose failure causes.An infeasible current table can trigger rollback to search for another table with an empty cup.Source: B. Closed Loop Feedback and Restoration · p. 3
- The plan restorer uses hierarchical recovery: it retries a failed manipulation with a different grasp pose, then can update the local object map and reclassify a moved target.Object-level recovery escalates when an object has shifted or retries are insufficient.Source: B. Closed Loop Feedback and Restoration · p. 3
- When the plan is infeasible or access from the current position is inadequate, global-level recovery navigates to another target position for improved access or alternative furniture.Source: B. Closed Loop Feedback and Restoration · p. 4
C. Implementation Details
COME-robot uses GPT-4V to generate executable robot plans from a prompt and provides it with perception and execution APIs for closed-loop operation.
C. Implementation Details
- The system prompt and user query are fed to GPT-4V, which generates Python code for the robot to execute.The prompt covers the task, feedback handling, robot setup and skills, response guidelines, and useful tips.Source: C. Implementation Details · p. 4
- Perception APIs build global and local object maps, while report observation captures a wrist-camera image for GPT-4V.The perception API set includes explore global, explore local, and report observation.Source: C. Implementation Details · p. 4
- Execution APIs provide navigation to mapped objects, grasping from observed point clouds, and placement on receptacles or at specified locations.The real-robot setup uses a four-wheeled differential-drive base with RGB-D, LiDAR, and IMU sensors, plus a 7-DoF Kinova Gen3 arm and wrist RGB-D camera.Source: C. Implementation Details · p. 4
IV. EXPERIMENT
The experiments compare COME-robot with baselines and analyze its execution trials to assess the system and its design.
IV. EXPERIMENT
- The experiment section describes task settings and baseline implementation details.The authors use comparisons with baselines and analyses of COME-robot execution trials to demonstrate system effectiveness and examine design significance.Source: §IV · p. 4
- The evaluation includes thorough comparisons with baseline methods.The passage does not specify the baseline methods or comparison outcomes.Source: §IV · p. 4
- Execution trials of COME-robot are analyzed systematically to assess the significance of its design.Source: §IV · p. 4
A. Experimental Setup
The setup evaluates mobile and tabletop manipulation with baseline comparisons, varied tasks, repeated trials, and measures of task and action success and recovery.
A. Experimental Setup
- CaP* is the baseline: the authors augment Code as Policies with exploration and navigation APIs because original CaP assumes a fully observable tabletop environment.Source: A. Experimental Setup · p. 4
- Four mobile tasks test basic transfer, object search, sequential visual reasoning, and efficient target-table selection in a furnished real-world room.The tasks are MOVE TOY, TRANSFER ALL TOYS, MOVE CUP AND TOY, and GATHER CUPS.Source: A. Experimental Setup · p. 4
- Four tabletop tasks, aligned with the CaP* setup, evaluate basic execution, spatial concepts, commonsense target identification, and free-form instruction following.The supplied task descriptions include placing fruit on a plate, placing fruit among cups, selecting a clean unused cup, and tidying a cluttered table.Source: A. Experimental Setup · pp. 4–5
- The GATHER CUPS example shows global mapping, local exploration, situated selection of a target table, and verification of grasp and placement.In the illustrated query, the robot moves one cup to the table already holding two cups.Source: Figure 4 · p. 5, A. Experimental Setup · p. 5
- Table I covers mobile-manipulation quantitative results, while Table III reports COME-robot failure and recovery statistics.Table III defines DF as failures leading to direct task failure and R-RR as the recovery rate of replanning steps.Source: Table I · p. 5, Table III · p. 5
- The authors run five trials per mobile task and ten per tabletop task, varying object types, arrangements, and quantities between trials.Source: A. Experimental Setup · p. 5
- Table II covers tabletop-manipulation quantitative results.Source: Table II · p. 5
- Success rate measures goal completion, step-wise success rate measures successful execution API calls, and recovery rate measures successful replanned executions.Perception calls such as report observation are excluded from the step-wise success-rate count.Source: A. Experimental Setup · p. 5
B. Experimental Results and Analyses
COME-robot improves goal completion over CaP* in both mobile and tabletop settings. The paper attributes its higher overall success to feedback-based error identification and correction, and illustrates two recovery cases.
B. Experimental Results and Analyses
- 65% (13/20) mobile success for COME-robot exceeds CaP* at 30% (6/20) by 35%; tabletop success is 75% versus the baseline’s 47.5% (19/40), a 27.5% difference.The passage also reports stepwise comparisons of 101/122 versus 76/125 and 123/140 versus 98/138, respectively.Source: B. Experimental Results and Analyses · p. 5
- The illustrated recovery examples adjust grasp position after a failed grasp and use visual feedback after a false-positive detection.Source: Figure 5 · p. 6
- COME-robot’s closed-loop feedback identifies and corrects execution errors, contributing to higher overall success on challenging real-world tasks.Source: B. Experimental Results and Analyses · p. 6
C. Failure Analysis
COME-robot uses closed-loop feedback to diagnose and recover from perception and execution failures. Recovery is not complete in every case: some false detections persist, and grasp failures remain tied to constrained robot positions.
Perceptual failures
- 100% of missed detections were recovered by rebuilding the local object scene graph and re-detecting the missing object.Wrong detections are checked with GPT-4V, but six false detections remained after verification and three led to task failure.Source: C. Failure Analysis · p. 6
- Six false detections remained after verification, and three of them led to task failure.The verification step can mitigate wrong detections but remains error-prone.Source: C. Failure Analysis · p. 6
Execution failures
- Plan verification triggers exceptions for incorrect plans or API calls, giving GPT-4V explicit feedback to correct them.Examples include trying to place an object before grasping it or passing an object name to navigation.Source: C. Failure Analysis · p. 6
- Unsuccessful grasps are primarily linked to impractical navigation positions that constrain manipulation space.The paper illustrates recovery from a grasp failure in Fig. 5.Source: C. Failure Analysis · p. 6
D. Discussions
The discussion argues that open-ended mobile manipulation needs both commonsense reasoning and interactive scene updates. It also links closed-loop replanning's value to longer mobile task sequences, where execution failures become more likely.
Open-ended instructions
- Choosing among detected cups requires knowing their scene status; instruction-based commonsense alone makes the PREPARE CUP task challenging.The robot must compare the cups to identify the one most suitable for the task.Source: D. Discussions · p. 6
Closed-loop replanning
- Mobile tasks require more execution steps than tabletop tasks, and longer sequences increase the likelihood of execution failure.Closed-loop replanning lets COME-robot detect failures and attempt recovery, reducing the likelihood of task failure.Source: D. Discussions · p. 6
V. CONCLUSION
The conclusion presents COME-robot as a closed-loop framework combining GPT-4V with robotic primitives for open-vocabulary mobile manipulation. Real-world experiments show abilities in interpreting open-ended instructions, reasoning over multimodal feedback, and recovering from perception and execution failures.
Conclusion
- COME-robot combines GPT-4V with robust robotic primitives in a closed-loop framework for open-vocabulary mobile manipulation.Source: §V · p. 6
- Real-world experiments show the system can interpret open-ended instructions, reason over multimodal feedback, and recover from perception and execution failures.Source: §V · p. 6
- The authors hope this work will inspire further research integrating foundation models with robotics to enhance intelligence and autonomy.Source: §V · p. 6