PaperlayerView exact revision on arXiv

Source-linked AI summary

BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulation

Rutav Shah, Albert Yu, Yifeng Zhu, Yuke Zhu, Roberto Martín-Martín

arXiv:2410.06237v1cs.ROcs.AI

TL;DR

Building-wide mobile manipulation asks robots to navigate rooms and floors while handling diverse objects over long-horizon tasks. BUMBLE unifies VLM reasoning with open-world perception, a broad skill library, and memory. It scores best across the reported tasks and buildings, with a 47.1% average success rate across tasks in three buildings.

  • Problem

    Building-wide mobile manipulation requires long-horizon navigation across rooms and floors while interacting with diverse objects, and poses challenges in locomotion, open-world perception, and learning from mistakes.

    Source: §I · p. 1

  • Method

    BUMBLE uses a VLM to reason over scenes, experience, and a parameterized skill library, integrated with open-world perception and short- and long-term memory.

    Source: §I · p. 2, §III · p. 3, A. Perception System, Skill Library,… · p. 3

  • Results

    BUMBLE scores best across tasks and buildings, with a 47.1% average success rate across retrieval and chair-rearrangement tasks in three buildings.

    Source: §I · p. 2, §IV · p. 5

  • Takeaways & Limitations

    The authors present BUMBLE as a stepping stone toward autonomous service robots in buildings.

    Source: §V · p. 6

    GoToLandmark assumes access to goal-image landmarks, which requires human effort to collect.

    Source: §V · p. 6

Abstract

from arXiv · show

To operate at a building scale, service robots must perform very long-horizon mobile manipulation tasks by navigating to different rooms, accessing different floors, and interacting with a wide and unseen range of everyday objects. We refer to these tasks as Building-wide Mobile Manipulation. To tackle these inherently long-horizon tasks, we introduce BUMBLE, a unified Vision-Language Model (VLM)-based framework integrating open-world RGBD perception, a wide spectrum of gross-to-fine motor skills, and dual-layered memory. Our extensive evaluation (90+ hours) indicates that BUMBLE outperforms multiple baselines in long-horizon building-wide tasks that require sequencing up to 12 ground truth skills spanning 15 minutes per trial. BUMBLE achieves 47.1% success rate averaged over 70 trials in different buildings, tasks, and scene layouts from different starting rooms and floors. Our user study demonstrates 22% higher satisfaction with our method than state-of-the-art mobile manipulation methods. Finally, we demonstrate the potential of using increasingly-capable foundation models to push performance further. For more information, see https://robin-lab.cs.utexas.edu/BUMBLE/

I. INTRODUCTION

Building-wide mobile manipulation requires robots to combine long-horizon planning, open-world perception, and building-scale navigation and interaction. BUMBLE unifies VLM reasoning with perception, diverse motor skills, and memory; experiments report successful long-horizon tasks and a 47.1% average success rate.

I. INTRODUCTION

  • Building-wide mobile manipulation combines travel across rooms and floors with interaction with diverse, often unseen objects.Tasks may involve navigating to a kitchen, using an elevator, clearing obstacles, and identifying an appropriate object in clutter.Source: §I · p. 1, Figure 1 · p. 1
  • BUMBLE unifies open-world perception, gross-to-fine motor skills, and memory in a VLM framework for reasoning and acting across buildings.Its VLM reasons over the scene, prior experience, and available skills to predict the next parameterized skill.Source: §I · pp. 1–2
  • The framework is designed to adapt to new objects, tasks, and buildings, and to improve as VLM backbones gain perception and reasoning capabilities.The skill library includes navigation between rooms and floors, base adjustment, and contact-rich object interactions.Source: §I · p. 2
  • 47.1% average success rate across tasks like retrieving soda cans and rearranging chairs across three buildings.The tasks include long-horizon trials requiring up to 12 parameterized skills and over 15 minutes per trial, excluding VLM query time.Source: §I · p. 2

II. RELATED WORK

Prior approaches offer pieces of building-wide mobile manipulation but leave gaps in open-world reasoning, skill breadth, or long-horizon memory. BUMBLE combines semantic and geometric reasoning with perception, motor skills, and memory.

II. RELATED WORK

  • TAMP is limited to preprogrammed object categories and scenes, restricting its application to building-scale environments with open-world objects.Earlier work on whole-body control and motion planning addressed low-level control, not long-horizon building-wide reasoning.Source: §II · p. 2
  • LLMs provide semantic planning but lack scene grounding, while VLMs offer a route to grounded reasoning for manipulation and navigation.BUMBLE uses a VLM to unify semantic and geometric reasoning at building-wide scale.Source: §II · pp. 2–3
  • Prior VLM mobile-manipulation methods lack both a diverse coarse-to-fine navigation and manipulation repertoire and long-horizon memory-based reasoning.BUMBLE integrates perceptual and motor skills with short- and long-term memory for diverse objects and scenes across buildings.Source: §II · p. 3, Figure 3 · p. 3
  • Unlike interactive-navigation methods focused on geometry, BUMBLE combines semantic judgments with geometric reasoning when moving obstacles or opening doors.For example, the method considers avoiding delicate objects as well as preventing collisions while pushing an obstacle.Source: §II · p. 3

III. BUMBLE: VLM-BASED BUILDING-WIDE MOMA

BUMBLE integrates perception, motor skills, and memory in a VLM-based system for building-wide mobile manipulation. Its decision module grounds a text instruction in the current scene and predicts the next parameterized skill.

III. BUMBLE: VLM-BASED BUILDING-WIDE MOMA

  • BUMBLE integrates perception, motor skills, and memory in a VLM-based system for high-level reasoning and low-level control.Its technical blocks include open-world perception, physical skills, and short- and long-term memory for adaptation and learning from mistakes.Source: §III · p. 3
  • The architecture iteratively perceives the environment, predicts parameterized skills, executes them, and stores each prediction and its parameters in short-term memory.The stored execution history supports subsequent skill prediction.Source: §III · p. 3
  • The VLM decision module grounds the instruction in the current scene and predicts the next parameterized skill to execute.The framework uses this prediction to connect high-level reasoning with robot action.Source: §III · p. 3

A. Perception System, Skill Library, and Memory

BUMBLE grounds open-world object perception in depth data and uses a broad parameterized skill library for building navigation and manipulation. Short- and long-term memory support execution tracking, recovery, and learning from prediction failures.

A. Perception System, Skill Library, and Memory

  • Grounded-SAM segments interactable objects, while depth back-projection yields point clouds and object distances for skill selection and precise manipulation.The system uses pixel-level masks to localize objects for manipulation.Source: A. Perception System, Skill Library,… · p. 3
  • The skill library spans navigation, base adjustment, pickup, pushing, door opening, and elevator use, with parameters mapped to low-level actions.GoToLandmark uses a topological visual map with landmark images and 2D occupancy maps to generate trajectories between locations.Source: A. Perception System, Skill Library,… · pp. 3–4
  • The execution trace depicts image observations, grounded skill parameters, skill names, and brief step descriptions; failed skills are followed by an adaptive VLM prediction.The figure’s annotations distinguish skill names in blue and failures in red.Source: Figure 4 · p. 4
  • Short-term memory records each step’s scene, subtask, skill, parameter, and detected outcome so the VLM can reason over trial history and recover from failures.Recovery examples include moving an obstacle or adjusting the base after an unsuccessful grasp.Source: A. Perception System, Skill Library,… · p. 4
  • Long-term memory stores human-flagged erroneous VLM predictions with their decision context and a textual failure analysis to inform future predictions.The stated aim is to learn from mistakes and reduce repeated errors and dependence on human annotations.Source: A. Perception System, Skill Library,… · p. 4

B. VLM-based Decision Making

BUMBLE separates each decision into choosing the next subtask and skill, then grounding parameters for that skill. Its VLM reasons over instructions, the current scene, robot state, and execution history.

B. VLM-based Decision Making

  • Each decision first predicts a subtask and selects a skill, then estimates that skill’s parameters and instantiates its motor execution.For example, the first stage may select GoToLandmark for reaching a place to find soda; the second may choose a kitchen image as the goal.Source: B. VLM-based Decision Making · p. 4
  • Subtask and skill selection combines the task instruction, skill descriptions, current RGB scene and robot location, and short- and long-term execution history.BUMBLE queries the VLM for intermediate reasoning with Chain-of-Thought (CoT), avoiding a classical planner’s computationally heavy search.Source: B. VLM-based Decision Making · p. 4
  • Set-of-Mark (SoM) [70] prompting marks foreground objects in RGB and pairs them with perception-derived distances so the VLM can assess reachability.The paper reports that naively prompting with depth images or point clouds does not yield desirable results because VLMs are not trained on those modalities.Source: B. VLM-based Decision Making · p. 4
  • For parameter estimation, the VLM receives the predicted subtask, memories of prior parameter mistakes, and visual markers for candidate choices, then selects a parameter using CoT prompting.Candidate markers can be arrow endpoints for MoveBase or GSAM-generated object candidates for skills such as UseElevator; outcomes return to short-term memory.Source: B. VLM-based Decision Making · p. 4

IV. EXPERIMENTAL EVALUATION

BUMBLE was evaluated on long-horizon building-wide tasks, with comparisons to adapted baselines, component ablations, and tests of VLM scaling. It outperformed the baselines, while analyses also report human ratings, failure modes, and recovery behavior.

IV. EXPERIMENTAL EVALUATION

  • The benchmark covers three tasks requiring up to 12 skills, with free-form task specifications and randomized obstacles across buildings and floors.Evaluations averaged three task specifications; the study spanned three university buildings and 90+ hours, though chair rearrangement was evaluated in only one building. Baselines were extended with BUMBLE’s skill library and short-term memory for comparison.Source: §IV · pp. 4–5
  • BUMBLE scores best in all tasks and buildings and outperforms COME by 12.1% on average; on OfflineSkillDataset, it achieves 80.2% success versus COME (72.6%) and Inner-Monologue (61.7%).Table I reports success rates from 10 trials in building-wide tasks; the OfflineSkillDataset comparison assesses skill-parameter prediction in predefined scenes.Source: Table I · p. 5, §IV · p. 5
  • BUMBLE’s performance scales with VLM capabilities across the Claude, Gemini, and GPT-4o model series on OfflineSkillDataset.The models’ increasing capabilities are ordered from left to right within each series in Fig. 5.Source: Figure 5 · p. 5, §IV · p. 5
  • After NaN depth readings caused failed manipulation attempts, BUMBLE used MoveBase to reposition toward elevator buttons or approach a chair obstacle.These observed recovery behaviors draw on the system’s diverse skill library.Source: §IV · pp. 5–6
  • Of 38 failures, 10 involve sensor failures or GSAM segmentation mistakes, while most are attributed to VLM reasoning; wrong-object selections rise from 10.0% with 5–10 distractors to 38.9% with 20–25.The analysis also notes errors in precise spatial decisions, including pressing the wrong elevator button.Source: §IV · p. 5, Figure 6 · p. 6
  • In the user study, BUMBLE receives a 3.7/5.0 Likert rating versus COME’s 2.6 and has 22% points less irrecoverable and 12% points less recoverable failures.Participants rated BUMBLE as completing tasks suboptimally 33% of the time.Source: Figure 7 · p. 6, §IV · p. 6
  • Without CoT, performance falls by 19.5% points to 60.7%; removing SoM decreases performance by 31% points.The reported SoM comparison replaces visual marking with natural-language object descriptions followed by GSAM segmentation, which struggled to distinguish specific instances such as diet from non-diet soda.Source: §IV · p. 6

V. CONCLUSION

BUMBLE uses a VLM-backed policy with a skill library, open-world perception, and memory to solve building-wide mobile manipulation tasks. Its stated limitations concern scaling long-term memory, collecting navigation landmarks, and interfacing skills that are not readily expressed in language.

V. CONCLUSION

  • BUMBLE combines a VLM-backed central reasoning policy with a diverse skill library, memory, and open-world perception to solve building-wide mobile manipulation tasks.The paper presents the framework as a stepping stone toward autonomous service robots in buildings.Source: §V · p. 6
  • Long-term memory of prior failures may become computationally intractable as it grows; proposed directions include dynamic retrieval or compressing experiences into learnable weights.These are future-work possibilities, not evaluated solutions.Source: §V · p. 6
  • GoToLandmark assumes human-collected landmark goal images, while autonomous exploration and landmark-frame selection are proposed to reduce that effort.Source: §V · p. 6
  • Adding skills or parameters that are difficult to express in language requires additional machinery, such as video conditioning or specialized weights.Source: §V · p. 6

A. Method details

BUMBLE maps scene perception, memory, and a broad set of motor skills to parameterized actions for building-wide mobile manipulation. Its skills range from landmark navigation and local base adjustments to object manipulation, doors, and elevators.

A. Method details

  • Open-world segmentation extracts interactable objects, whose poses are converted into low-level robot base commands.The described perception system uses GroundingDINO for boxes and SAM-HQ for segmentation.Source: A. Method details · p. 9
  • GoToLandmark navigates between same-floor locations using landmark images and an occupancy map, with the VLM selecting a landmark for the subtask.The ROS stack for Tiago includes gmapping, amcl, move_base, global_planning, and teb_local_planner.Source: A. Method details · pp. 9–10
  • NavigateNearObj uses segmented scene objects to move the robot near a visible object for a task.The target pose is planned using the occupancy map and front 2D lidar.Source: A. Method details · p. 9
  • MoveBase adjusts the robot 0.3 meters forward, backward, left, or right for final approach, while pickup uses the left arm and cannot lift heavy objects such as chairs.Source: A. Method details · p. 9
  • PushObjOnGround selects a visible object and push direction, then moves the robot to a target pose and executes a predefined arm trajectory; objects are 2–3 m from the robot.Source: A. Method details · p. 9
  • OpenDoor pushes a door with a selected arm, while CallElevator and UseElevator press segmented buttons to call an elevator and change floors.Elevator button pressing uses a force threshold, and UseElevator localizes on the new floor map.Source: A. Method details · p. 10
  • Short-term memory tracks state-action history to address skill-detected failures, while long-term memory retains erroneous predictions from offline examples for selected skills and skill selection.The retained examples cover MoveBase, UseElevator, PushObjOnGround, and skill selection.Source: A. Method details · p. 10

B. Experimental Evaluations

The evaluation varies language instructions for three retrieval or rearrangement tasks and lists multiple VLM checkpoints. It frames the checkpoint comparison as demonstrating how BUMBLE can leverage advancing VLM capabilities.

B. Experimental Evaluations

Loading 2410.06237v1…