PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

Joshua Jones, Oier Mees, Carmelo Sferrazza, Kyle Stachowicz, Pieter Abbeel, Sergey Levine

arXiv:2501.04693v3cs.ROcs.AI

TL;DR

Generalist robot policies typically lack touch and audio because large heterogeneous datasets are unavailable, limiting multimodal interaction. FuSe finetunes pretrained policies with language-based cross-modal grounding using contrastive and language-generation losses. It enables zero-shot multimodal and compositional capabilities across policy architectures and improves success rates by over 20% compared with considered baselines.

  • Problem

    Generalist robot policies usually rely on visual and proprioceptive data, while datasets with touch, audio, and joint multimodal action reasoning are scarce.

  • Method

    FuSe finetunes pretrained generalist policies on heterogeneous sensor data by grounding modalities in natural language with contrastive and sensory-grounded language-generation losses.

  • Results

    FuSe enables zero-shot reasoning across vision, touch, and sound, including multimodal prompting, compositional prompting, and object descriptions, across different policy backbones.

  • Takeaways & Limitations

    Language-grounded multimodal finetuning extends pretrained generalist policies to challenging heterogeneous-sensor manipulation tasks using smaller-scale datasets.

  • Takeaways & Limitations

    Additional modalities increase training-resource requirements and currently limit observation history to 0.4s.

Abstract

from arXiv · show

Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities -- including vision, touch, and audio -- to fill in gaps from partial observation. For example, when vision is occluded reaching into a bag, a robot should rely on its senses of touch and sound. However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations. In this work, we propose FuSe, a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities for which large datasets are not readily available by leveraging natural language as a common cross-modal grounding. We combine a multimodal contrastive loss with a sensory-grounded language generation loss to encode high-level semantics. In the context of robot manipulation, we show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound in a zero-shot setting, such as multimodal prompting, compositional cross-modal prompting, and descriptions of objects it interacts with. We show that the same recipe is applicable to widely different generalist policies, including both diffusion-based generalist policies and large vision-language-action (VLA) models. Extensive experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.

I. INTRODUCTION

Generalist robot policies typically lack heterogeneous sensory data and joint multimodal reasoning, despite touch and audio complementing vision. FuSe addresses this gap by grounding additional modalities in language during finetuning, enabling zero-shot multimodal capabilities across policy architectures.

  • Motivation: Generalist robot policies typically rely on visual and proprioceptive observations, although touch and audio provide complementary information when vision is insufficient.This limitation is especially relevant in partially observable manipulation settings.
  • Research gap: Sparse datasets containing touch or audio make it difficult to connect pretrained generalist policies with heterogeneous sensory data and low-level actions.The challenge includes a lack of joint multimodal-perceptual and action data.
  • Approach: FuSe finetunes generalist policies on smaller multimodal datasets and uses natural language as a common grounding across sensory modalities.The approach is designed for modalities complementary to vision, including touch and sound.
  • Capabilities: Language grounding enables zero-shot reasoning over vision, touch, and sound for multimodal prompting, object descriptions, and compositional cross-modal prompting.Examples include selecting objects by color, tactile feel, and sound, or describing a grasped object.
  • Results: Multimodal finetuning consistently outperforms vision-only finetuning and heterogeneous-data training from scratch across generalist policy architectures.The recipe applies to Octo and a 3B PaliGemma-based vision-language-action model.
  • Experimental basis: The experiments use 27K trajectories with vision, touch, audio, proprioception, and language instructions across three real-world manipulation tasks.The dataset includes robot action data and is released with code and models.

II. RELATED WORK

Generalist robot policies benefit from large-scale datasets but remain primarily visual because touch and audio are rarely available. This work adds heterogeneous sensory data and actions to extend their multimodal reasoning capabilities.

  • Generalist policies: Large generalist robot datasets and models primarily support visual inputs because touch and audio are scarce in available collections.This limits multimodal capabilities despite some models accepting flexible observations.
  • FuSe contribution: The work uses smaller robotic datasets containing additional heterogeneous modalities to enable joint reasoning over vision, touch, and sound in zero-shot settings.This extends capabilities beyond the dominant visual-input regime.

B. Multimodal Reasoning in Robotics

Multimodal sensing can improve robotic performance, generalization, and robustness, but datasets with nonvisual sensors and robot actions remain uncommon. FuSe addresses this by extending pretrained policies with language-grounded multimodal finetuning.

  • Motivation: Multimodal sensing exploits complementary information across sensors and has been associated with improved performance, generalization, and robustness.
  • Data gap: Most robotics datasets omit touch and sound, while many multimodal perception datasets lack robot actions needed for physically grounded policy training.The work introduces a dataset combining vision, touch, audio, inertial measurements, proprioception, actions, and language.
  • FuSe finetuning: FuSe finetunes pretrained policies to extend semantic understanding to touch and sound while retaining pretrained knowledge.The method links heterogeneous observations with natural language through contrastive and generative auxiliary losses.
  • Implementation challenges: Finetuning must address small datasets for new encoders, reliance on pretrained modalities, and modality-specific annotations for cross-modal prompting.The implementation uses pretrained tactile encoding, spectrogram-based audio encoding, and templated language rephrasing.
  • Architecture: The architecture tokenizes heterogeneous modalities, processes them with a pretrained transformer, and applies contrastive and language-generation losses.The contrastive loss aligns modalities and scene semantics, while generation predicts high-level semantics for modality combinations.

IV. EXPERIMENTS

The experiments evaluate whether FuSe supports zero-shot multimodal prompting, object discrimination under ambiguous single modalities, compositional reasoning, and the necessity of its auxiliary language-grounding losses.

  • Evaluation questions: The experiments test zero-shot multimodal prompting in partially observable environments.
  • Evaluation questions: The evaluation examines whether multimodal prompting distinguishes objects that are ambiguous when described through a single modality.
  • Evaluation questions: The experiments assess compositional reasoning and whether auxiliary cross-modal language-grounding losses are necessary for high finetuning performance.

A. Real Robot Setup and Training Data

The real-robot experiments use a multimodal sensor suite and evaluate grasping on seen and unseen objects across repeated rollouts. The dataset contains 26,866 teleoperated trajectories with visual, tactile, audio, proprioceptive, and language data.

  • Hardware and control: The WidowX 250 robot uses third-person and wrist RGB cameras, two DIGIT tactile sensors, a microphone, and a 9-DoF IMU.It is controlled through delta end-effector position commands at 5 Hz.
  • Evaluation objects: Evaluation covers 24 training objects and 32 unseen test objects for grasping, plus six buttons and 13 distractors for button tasks.The experiments visualize the training and test objects in Figure 4.
  • Object diversity: The real-world objects vary in shape, appearance, material, hardness, and surface properties.
  • Evaluation protocol: Each evaluation scenario is run for five different rollouts.
  • Training data: 26,866 teleoperated trajectories combine vision, touch, audio, proprioception, and templated language instructions.Grasping tasks include visual, tactile, and action data, while button tasks additionally include sound.

B. Evaluation Tasks

The evaluation tests zero-shot multimodal reasoning in tabletop grasping, occluded shopping-bag grasping, and button pressing with cross-modal sound and visual relationships. Figure 5 compares FuSe with baselines and distinguishes full grasping from objects merely touched.

  • Tabletop Grasping: Tabletop grasping asks the robot to select and grasp a prompted object among multiple objects on a tray.
  • Shopping Bag: Shopping-bag grasping introduces occlusions and poor lighting as objects are placed inside a paper bag.
  • Button Pressing: Button-pressing tasks require selecting buttons using visual, sound, and cross-modal relationships.One task matches an object's visual characteristics to a button, while another matches a training button's sound to an unseen button.
  • Comparisons: Figure 5 compares FuSe with baselines trained from scratch or finetuned with vision only across the evaluation tasks.Lighter shades denote intermediate success in which an object was touched but not fully grasped.

C. Finetuning Performance

Finetuning a pre-trained generalist policy on multimodal data outperforms training from scratch and vision-only finetuning, with the largest advantage under shopping-bag visual occlusion. The results indicate that pre-trained knowledge adapts to new tasks and modalities using limited additional data.

  • Pre-training comparison: Pre-trained Octo finetuned with the multimodal recipe significantly outperforms both training Octo from scratch and a ResNet26 baseline on all three tasks.The from-scratch model is challenged by the limited size of the multimodal dataset.
  • Role of pre-training: The multimodal recipe leverages knowledge acquired during pre-training to adapt to new tasks and modalities with a smaller amount of additional data.
  • Vision-only comparison: Vision-only finetuning is competitive on tabletop and button pressing but considerably inferior on the shopping-bag task.Visual occlusions make visual features less discriminative when the gripper enters the bag.

D. Multimodal Prompting

FuSe uses multimodal instructions to disambiguate objects through visual, tactile, and audio properties, then extends this grounding to compositional button-pressing tasks. Its generative head supports a multi-step chain from visual button selection to audio-based action.

  • D. Multimodal Prompting: FuSe distinguishes objects using multimodal prompts when descriptions based on one modality are ambiguous.Examples combine properties such as being round, feeling squishy, or producing a particular sound.
  • E. Compositional Capabilities: The simpler compositional task asks the model to grab an object matching the color of a button associated with a specified sound.
  • E. Compositional Capabilities: The multi-step task first presses an unseen button from visual instructions, generates an audio-related instruction, and then presses the training button with the same sound.The generative head connects the subtasks through the resulting sound and its corresponding language instruction.
  • E. Compositional Capabilities: FuSe outperforms all baselines on the simpler compositional task and outperforms the from-scratch comparison particularly on full multi-step completion.The from-scratch model shows poor language grounding and does not fulfill the audio-based instruction.
  • D. Multimodal Prompting: Table I evaluates multimodal prompting on objects sharing visual features and on objects sharing tactile features.The caption reports improvement over ambiguous descriptions.
  • E. Compositional Capabilities: Figure 6 contrasts object selection by matching color to a sound-associated button with multi-step button selection by matching sound across buttons.

F. Ablation Study

FuSe’s auxiliary losses are evaluated in the partially observable shopping bag task, where using both losses improves performance, especially on unseen objects.

  • Including both auxiliary losses is key to exploiting heterogeneous robot feedback in the shopping bag task.Performance deteriorates particularly for baselines on unseen test objects.

G. Vision-Language-Action Model Results

FuSe extends to a 3B-parameter PaliGemma-based VLA and remains effective across policy backbones. The model uses both contrastive and language-generative losses, with strong performance in the shopping bag task.

  • FuSe’s PaliGemma-based VLA uses both contrastive and language-generative losses in the multimodal finetuning setup.
  • The PaliGemma FuSe VLA achieves robust performance, demonstrating applicability to a different generalist policy architecture.
  • The VLA FuSe policy is competitive with its Octo-based counterpart and outperforms it on the challenging shopping bag task.
  • VLA FuSe is the first open-source VLA finetuned on heterogeneous non-visual sensory inputs, to the authors’ knowledge.

V. CONCLUSIONS

FuSe finetunes large pretrained robot policies on heterogeneous sensing by grounding modalities in natural language, enabling zero-shot multimodal reasoning and capabilities across policy architectures. The approach is limited by training-resource requirements that restrict observation history to 0.4s.

  • FuSe finetunes large pretrained robot policies on touch and audio using natural language as common cross-modal grounding.
  • FuSe enables zero-shot reasoning over vision, touch, and sound, including multimodal prompting, compositional cross-modal prompting, and object descriptions.
  • The recipe applies across generalist policies including transformer-based Octo and a policy finetuned from a generative VLM base model.
  • Additional modalities increase training-resource requirements and currently limit the observation history to 0.4s.Longer context could improve reasoning about sparse signals such as tactile data, according to the authors.
Loading 2501.04693v3…