PaperlayerView exact revision on arXiv

Source-linked AI summary

VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulation

Wei Zhao, Pengxiang Ding, Min Zhang, Zhefei Gong, Shuanghao Bai, Han Zhao, Donglin Wang

arXiv:2502.13508v2cs.RO

TL;DR

Existing VLAs largely depend on text instructions or external speech recognition, which complicates the system and can lose useful information in raw speech. VLAS integrates speech directly into an end-to-end robot policy, adds multimodal tuning datasets and Voice RAG, and reports effective performance on speech-driven and customized manipulation tasks.

  • Problem

    Existing VLAs mainly support text instructions, while external speech recognition adds complexity and can discard voice information relevant to customized robot manipulation.

  • Method

    VLAS directly processes speech, text, and visual observations through three-stage tuning with SQA and CSI, and uses Voice RAG to retrieve individual-specific knowledge.

  • Results

    VLAS significantly outperforms cascading speech-understanding methods and achieves an average success rate of over 86% on customized tasks, versus below 20% for the text-only VLA baseline.

  • Takeaways & Limitations

    Direct speech integration enables VLAS to simplify robot-control pipelines and handle customized tasks requiring individual-specific knowledge.

Abstract

from arXiv · show

Vision-language-action models (VLAs) have become increasingly popular in robot manipulation for their end-to-end design and remarkable performance. However, existing VLAs rely heavily on vision-language models (VLMs) that only support text-based instructions, neglecting the more natural speech modality for human-robot interaction. Traditional speech integration methods usually involves a separate speech recognition system, which complicates the model and introduces error propagation. Moreover, the transcription procedure would lose non-semantic information in the raw speech, such as voiceprint, which may be crucial for robots to successfully complete customized tasks. To overcome above challenges, we propose VLAS, a novel end-to-end VLA that integrates speech recognition directly into the robot policy model. VLAS allows the robot to understand spoken commands through inner speech-text alignment and produces corresponding actions to fulfill the task. We also present two new datasets, SQA and CSI, to support a three-stage tuning process for speech instructions, which empowers VLAS with the ability of multimodal interaction across text, image, speech, and robot actions. Taking a step further, a voice retrieval-augmented generation (RAG) paradigm is designed to enable our model to effectively handle tasks that require individual-specific knowledge. Our extensive experiments show that VLAS can effectively accomplish robot manipulation tasks with diverse speech commands, offering a seamless and customized interaction experience.

1 INTRODUCTION

VLAS addresses the limitations of text-only and cascaded speech pipelines by directly integrating speech into an end-to-end robot policy. It supports multimodal instructions and customized manipulation through new datasets and Voice RAG.

  • Speech offers a more natural and customizable communication modality for robots assisting users with varied physical abilities and preferences.
  • External speech recognition complicates robot systems and can discard identity, emotion, and intonation information needed to interpret unstructured instructions.
  • VLAS directly processes textual and speech instructions with visual observations in an end-to-end policy model.It is built on LLaVA and trained through three phases that align speech representations with the model’s language space before multimodal and behavior-cloning tuning.
  • The SQA and CSI datasets support multimodal tuning for speech-image understanding and robot manipulation trajectories.The resulting VLAS-Base handles text-image and speech-image instructions, while CSI supports behavior cloning for manipulation.
  • VLAS is presented as a speech-capable VLA that avoids external speech recognition and supports customized robot tasks requiring individual-specific knowledge.The paper also introduces Voice RAG and releases VLAS-Base, SQA, and CSI for further study.

2 RELATED WORK

Prior work established vision-language models and vision-language-action models for multimodal understanding and robot manipulation, but speech remains insufficiently integrated into VLA systems. Existing speech-enabled approaches commonly rely on external speech-to-text conversion, which adds complexity and loses auxiliary speech information.

  • Vision-Language Model: Vision-language models developed from large instruction-following models support text-image question answering and multimodal instruction following.LLaVA is highlighted for accessibility, reproducibility, performance, and two-stage visual instruction tuning.
  • Vision-Language Model: LLaVA lacks support for speech instructions despite its strong visual question-answering performance.
  • Vision-Language-Action Model: VLMs have been applied to robotics through approaches that use foundation models for high-level planning or robot manipulation.
  • Vision-Language-Action Model: Few studies investigate speech inputs for VLA models, and the common external speech-recognition approach is complex and loses auxiliary speech information.

3 METHOD

VLAS directly integrates speech, vision, personalized knowledge retrieval, and robot action generation in an end-to-end policy. Its SQA and CSI datasets support a three-stage training process that aligns speech, develops multimodal understanding, and fine-tunes manipulation.

  • Overall Framework: VLAS takes speech instructions and visual observations as inputs, then directly generates robot actions through a LLaMA-based policy.Visual and speech features are projected into a shared language space before action generation.
  • Action Generation: VLAS autoregressively predicts discretized action tokens, which are detokenized into continuous robot-control values across seven action dimensions.The action dimensions include end-effector position, rotation, and gripper state.
  • Voice RAG: The Voice RAG module uses speaker identification to retrieve personalized knowledge, tokenizes it, and supplies it alongside visual and speech embeddings.The retrieved information provides customized background knowledge for instructions requiring individual-specific information.
  • Datasets: The SQA and CSI datasets provide speech-based supervision for visual question answering and robot manipulation, respectively.SQA converts textual questions into speech, while CSI adds speech instructions to CALVIN manipulation trajectories using 500 voices.
  • Training Paradigm: The model uses three training stages: speech alignment, speech question-answering fine-tuning, and robot manipulation fine-tuning.The stages progressively train speech-text alignment, multimodal comprehension, and manipulation using speech or text instructions.
  • Model Resources: VLAS-Base extends LLaVA to accept speech instructions and is presented as a resource for multimodal large-language-model research.The model is built on LLaVA and uses a speech encoder with an MLP projector to share semantic space with visual and text inputs.

4 EXPERIMENTS AND RESULTS

Experiments evaluate VLAS on standard robot manipulation, customized tasks, real-world operation, and its VLAS-Base foundation model. VLAS outperforms comparable speech-understanding pipelines and achieves substantially higher success on customized tasks.

  • 4.1 ROBOT MANIPULATION WITH SPEECH INSTRUCTIONS: VLAS is evaluated on CALVIN, a benchmark of 1,000 long-horizon tasks, against a traditional VLA baseline fine-tuned from LLaVA without speech support.Each CALVIN task contains five successive subtasks and a human command.
  • 4.1 ROBOT MANIPULATION WITH SPEECH INSTRUCTIONS: VLAS with textual or speech instructions significantly outperforms MCIL, HULC, and RT-1 on CALVIN, while textual VLAS remains comparable to the baseline VLA.Speech evaluations include synthetic voices and real recordings, with external Whisper large-v2 transcription used for baseline speech systems.
  • 4.1 ROBOT MANIPULATION WITH SPEECH INSTRUCTIONS: VLAS significantly outperforms two cascading speech-understanding methods, which the authors associate with specialized instruction recognition and reduced error propagation.The authors state that VLAS can also be combined with other VLA models.
  • 4.2 ROBOT MANIPULATION FOR CUSTOMIZED TASKS: Over 86% average success for VLAS versus below 20% for the VLA baseline on customized tasks requiring individual-specific knowledge.VLAS uses raw speech and Voice RAG, whereas the baseline relies on text instructions and lacks background knowledge.
  • 4.2 ROBOT MANIPULATION FOR CUSTOMIZED TASKS: Removing Voice RAG significantly degrades VLAS performance on the customized benchmark, while adding it to VLA significantly improves performance.The ablations support the effectiveness of Voice RAG for customized tasks.
  • 4.4 ANALYSIS FOR THE VLAS-BASE FOUNDATION MODEL: VLAS-Base achieves nearly the same performance as LLaVA across general multimodal benchmarks and significantly outperforms other prevalent VLMs.These results indicate that adding speech does not degrade the foundation model’s performance.
  • 4.4 ANALYSIS FOR THE VLAS-BASE FOUNDATION MODEL: VLAS-Base achieves comparable performance to Whisper large-v2 on LibriSpeech and surpasses BLIP-2 while remaining behind LLaVA with ground-truth text on SGQA.The SGQA benchmark evaluates image-speech question answering.

5 CONCLUSION

The paper presents VLAS as an end-to-end speech-enabled VLA for customized robot manipulation. It combines direct raw-speech processing with Voice RAG to use speaker-specific information and simplify control.

  • 5 CONCLUSION: VLAS understands speech instructions for robot manipulation without relying on an external speech recognition system.Raw speech is directly input to the model, allowing auxiliary information such as voiceprint to be used.
  • 5 CONCLUSION: Voice RAG improves VLAS performance on spoken instructions that require extensive individual-specific knowledge.The paper reports that speech integration enables customized tasks across a range of user-specific scenarios.
  • 5 CONCLUSION: The authors identify other auxiliary information in human speech or environmental sounds as future directions for complex tasks.

A TRAINING DETAILS

VLAS is trained through three sequential fine-tuning stages using speech recognition, speech question answering, instruction-following, and robot manipulation data. Training uses different compute configurations across stages.

  • A TRAINING DETAILS: Stage I fine-tunes on LibriSpeech train-clean-100 for 5 epochs with learning rate 1e-3 and batch size 16.
  • A TRAINING DETAILS: Stage II fine-tunes on SQA, LLaVA’s 665K instruction-following data, and LibriSpeech train-clean-360 for 1 epoch with learning rate 2e-5 and batch size 16.
  • A TRAINING DETAILS: Stage III fine-tunes on the CSI robot manipulation dataset for 1 epoch.
  • A TRAINING DETAILS: All models use 8× A100 GPUs except Stage I, where a single GPU performs coarse-grained speech alignment.The authors report better performance for Stage I with one GPU.

B.1 FAILURE CASES OF VLAS AND VLA ON THE CUSTOMIZATION BENCHMARK

Failure analysis finds that VLAS failures concentrate in preference tasks and the second phase of compound tasks, whereas VLA failures show diverse patterns on the customization benchmark.

  • B.1 FAILURE CASES OF VLAS AND VLA ON THE CUSTOMIZATION BENCHMARK: VLAS failures mainly occur in preference tasks and the second phase of compound tasks.
  • B.1 FAILURE CASES OF VLAS AND VLA ON THE CUSTOMIZATION BENCHMARK: VLAS shows a more consistent error pattern, suggesting instruction understanding but unsuccessful action execution.The authors conjecture that policy architecture and training refinements could address this issue.
  • B.1 FAILURE CASES OF VLAS AND VLA ON THE CUSTOMIZATION BENCHMARK: VLA exhibits diverse failure patterns because it accesses only superficial semantic information and relies on random attempts for personalized tasks.

B.2 COMPARISON WITH ROBOFLAMINGO ON THE CALVIN BENCHMARK

VLAS is compared with RoboFlamingo on CALVIN using textual instructions. VLAS performs slightly worse, with the difference attributed mainly to its lack of historical information when predicting actions.

  • VLAS performs slightly behind RoboFlamingo on the CALVIN benchmark with textual instructions.
  • The reported gap is mainly attributed to VLAS lacking historical information during action prediction.
  • Removing RoboFlamingo’s LSTM policy head causes its performance to deteriorate significantly.

B.3 COMPARISON WITH OPENVLA ON THE CALVIN BENCHMARK

VLAS is evaluated against pretrained and fine-tuned OpenVLA models on CALVIN. The pretrained model fails at zero-shot generalization, while the fine-tuned model completes some early tasks but remains suboptimal overall.

  • The comparison uses a pretrained OpenVLA checkpoint tailored to the Franka Emika Panda in simulation and an officially recommended fine-tuning setup.
  • The pretrained OpenVLA model fails to achieve zero-shot generalization on the CALVIN benchmark.
  • The fine-tuned OpenVLA model completes some initial long-horizon tasks but remains suboptimal overall.
  • The authors hypothesize that OpenVLA’s poor performance is related to its support for only third-person-view input.

B.4 EXPERIMENTAL EVALUATION ON THE CALVIN BENCHMARK USING ABC/D SPLITS

ABC-to-D evaluation tests generalization to novel scenes under a domain gap, while a personalization benchmark evaluates customized-task performance. VLAS remains comparable to RoboFlamingo and outperforms the other models, and it handles novel scenes in personalization tests.

  • ABC/D split evaluation: The ABC-to-D experiment evaluates generalization by training on ABC splits and testing on the D split.
  • ABC/D split evaluation: Despite performance degradation from the domain gap, VLAS performs comparably to RoboFlamingo and outperforms the other models on ABC-to-D evaluation.
  • Benchmark settings: The CALVIN policy-model comparison evaluates textual and speech instructions, while the customized-task benchmark uses Voice RAG for individualized knowledge.
  • Personalization benchmark: On the personalization benchmark, VLAS demonstrates the ability to handle novel scenes.

B.5 INFERENCE EFFICIENCY ANALYSIS

VLAS accelerates inference through speech-spectrogram downsampling and multi-step action prediction and execution. Setting r=5 improves speed and CALVIN performance while achieving the best balance between efficiency and manipulation performance.

  • Two optimizations accelerate VLAS inference: speech-spectrogram downsampling and multi-step prediction and execution.
  • x = 5 downsamples adjacent spectrogram frames, while r=5 sets the number of predicted and executed action steps for both VLA and VLAS.
  • With r=5, both VLA and VLAS achieve significant speedups and improved CALVIN benchmark performance.
  • r=5 achieves the optimal balance between inference efficiency and manipulation performance.
Loading 2502.13508v2…