PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboOmni: Proactive Robot Manipulation in Omni-modal Context

Siyin Wang, Jinlan Fu, Feihong Liu, Xinzhe He, Huangxuan Wu, Junhao Shi, Kexin Huang, Zhaoye Fei, Jingjing Gong, Zuxuan Wu, Yu-Gang Jiang, See-Kiong Ng, Tat-Seng Chua, Xipeng Qiu

arXiv:2510.23763v3cs.ROcs.CLcs.CV

TL;DR

Robotic manipulation systems largely assume explicit instructions, leaving proactive intent inference from real-world multimodal context underexplored. RoboOmni introduces cross-modal contextual instructions and an end-to-end omni-modal framework with the OmniAction dataset, and it outperforms text- and ASR-based baselines across success rate, inference speed, intention recognition, and proactive assistance.

  • Problem

    Existing robotic manipulation systems generally assume explicit instructions and lack joint reasoning over speech, environmental sounds, and visual observations for proactive intent recognition.

    Source: §1 · pp. 1–2

  • Method

    RoboOmni is a Perceiver-Thinker-Talker-Executor framework that fuses auditory and visual inputs, supports direct speech interaction, and unifies intention recognition, confirmation, and action execution.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    RoboOmni outperforms text- and ASR-based baselines in simulation and real-world settings across accuracy, inference speed, proactive assistance, and intention recognition.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Cross-modal contextual instructions and OmniAction provide a setting and data resource for proactive robotic manipulation from multimodal context.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.

1 Introduction

RoboOmni addresses the gap between explicit-instruction robotics and real-world interactions by inferring and verifying intent from speech, environmental sounds, and vision. It combines this capability with a new dataset and reports stronger performance than text- and ASR-based baselines.

Results

  • Experiments in simulation and real-world settings show higher accuracy, faster inference, more effective proactive assistance, and improved intention recognition than text- and ASR-based baselines.The reported gains span both evaluation settings and multiple performance axes.Source: §1 · pp. 2–3

Motivation

  • Cross-modal contextual instructions require robots to proactively infer user commands from vision, environmental sounds, and speech rather than await explicit instructions.The setting targets latent intent conveyed by multimodal context and verified through interaction.Source: §1 · p. 2

RoboOmni

  • RoboOmni is an end-to-end Perceiver-Thinker-Talker-Executor framework that unifies intent recognition, confirmation, and action execution.It fuses auditory and visual inputs for intent reasoning within an omni-modal LLM framework.Source: §1 · p. 2

Dataset

  • OmniAction contains 140k episodes, over 5k speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types.The dataset addresses the lack of training data for proactive intention reasoning and is paired with OmniAction-LIBERO for simulation-based evaluation.Source: §1 · p. 2

2 Related Work

Prior omni-modal and vision-language-action systems improve multimodal processing but generally remain limited by fragmented pipelines, explicit commands, or action-only outputs. RoboOmni extends end-to-end omni-modality to embodied action and proactive contextual interaction.

2.1 Omni-Modal LLMs

  • End-to-end omni-modal models jointly represent speech, vision, and text, but prior systems remain oriented toward linguistic outputs rather than embodied actions.Earlier modular pipelines also face temporal-alignment difficulties and limits in situated semantic understanding.Source: §2.1 · p. 3

2.2 Vision-Language-Action Model

  • Vision-language-action models typically map visual and linguistic inputs to actions but assume short explicit commands and struggle with compositional or context-dependent tasks.Cascaded and hierarchical extensions decompose instructions into sub-goals but remain fragmented and rigid.Source: §2.2 · p. 3
  • ASR/TTS pipelines discard paralinguistic cues, add latency, and disrupt temporal alignment, while direct-speech VLA systems remain restricted and output actions without speech responses.These limitations motivate direct integration of speech, environmental sounds, vision, and text for natural interaction.Source: §2.2 · p. 3

3 OmniAction Dataset Construction

OmniAction is a multimodal dataset for proactive intent reasoning, combining contextual instructions with speech, environmental sounds, and visual observations. Its construction pipeline turns scripted tasks into verified audio-visual interaction data and simulation benchmarks.

3.1 Overview

  • OmniAction is designed to address the lack of datasets combining audio, visual context, and inferential instructions for proactive intent reasoning.Its contextual instructions encode latent intents grounded in speech, sound events, environmental audio, and vision.Source: §3.1 · p. 3
  • The dataset covers six contextual instruction types, including sentiment, overlapping voices, non-verbal cues, identity cues, dyadic dialogue, and triadic dialogue.Single-person text instructions are also included to preserve general command-following ability.Source: §3.1 · pp. 3–4
  • OmniAction contains 141,162 multimodal episodes spanning 112 skills, 748 objects, 5,096 speaker timbres, 2,482 sound events, and 640 environmental backgrounds.Each sample pairs multi-turn conversation, visual observations, and a 7-D end-effector action trajectory.Source: §3.1 · p. 4

3.2 Construction Process

  • The three-stage construction pipeline comprises textual scripting, auditory realization, and verification.Textual scripting transforms Open-X Embodiment tasks into contextual dialogues; auditory realization adds synthesized speech, overlaps, sound events, and backgrounds; sampled dialogues achieved 98.7% intent-recovery agreement in manual evaluation.Source: §3.2 · pp. 4–5

3.3 Simulation Dataset: OmniAction-LIBERO

  • OmniAction-LIBERO provides 240 simulated evaluation tasks across four LIBERO suites and six contextual instruction types, plus a real-speech evaluation with 10 volunteers.The TTS variant expands 40 manipulation tasks into six variants per task, while the real variant evaluates speech collected in real environments.Source: §3.3 · p. 5

4 Methods

RoboOmni unifies multimodal perception, reasoning, speech interaction, and robotic control in a Perceiver–Thinker–Talker–Executor architecture. It generates conversational and action outputs autoregressively within a shared token space and trains them with a unified objective.

4 Methods

  • RoboOmni uses unified tokenization and specialized decoders to process speech, environmental audio, vision, text, and actions within one end-to-end framework.The Perceiver encodes modalities, the Thinker produces shared representations, the Talker generates speech, and the Executor renders executable actions.Source: §4 · p. 5

4.1 Architecture Components

  • The Perceiver concatenates visual, audio, and textual embeddings into a unified representation that feeds the Thinker.At timestep t, Xt = [vt; st; ct] combines visual, audio, and textual context.Source: §4.1 · p. 5
  • The Thinker autoregressively interleaves text tokens, speech representations, and action tokens in the joint vocabulary space V ∪ A.This shared sequence space supports reasoning across perception, language, and robotic control.Source: §4.1 · p. 6
  • The Executor converts discrete FAST+ action-token sequences back into continuous robot commands, representing 7-DoF vectors with short symbol sequences from a 2,048-token action vocabulary.This bridges language-model generation and robotic control in a single autoregressive sequence.Source: §4.1 · p. 6

4.2 Dual-Mode Generation

  • RoboOmni supports dual-mode generation: the Thinker produces text for conversation, optionally converted to speech by the Talker, or predicts action-token chunks decoded into continuous actions.Conversational and manipulation outputs therefore share the same multimodal reasoning backbone while using different output pathways.Source: §4.2 · p. 6

4.3 Training Paradigms

  • Training combines conversational and action supervision through batch interleaving under one autoregressive maximum-likelihood objective over V ∪ A.The model learns either text responses or action sequences from multimodal context within the same framework.Source: §4.3 · p. 6

5 Experiment

RoboOmni is evaluated from simulation through real-world manipulation, proactive assistance, and efficiency analyses. Across these settings, it outperforms text-, ASR-, and cascaded baselines while preserving multimodal contextual information.

5.2 Evaluation on Cross-modal Contextual Instructions

  • 85.6% overall success rate surpasses NORA’s 25.9% and all cascaded methods below 10% on OmniAction-LIBERO.RoboOmni exceeds 76% across all audio types, while text-only baselines fail to capture prosody and overlapping speech.Source: §5.2 · p. 8

5.3 Evaluation on Real Human Audio Direct Instructions

  • 76.6% average performance on OmniAction-LIBERO-Real exceeds π0 at 73.8%, OpenVLA at 40.1%, and NORA at 17.4%.Direct speech processing avoids ASR errors from accents, coarticulation, and background noise.Source: §5.3 · p. 8, Table 2 · p. 8

5.4 Real-World Experiments

  • RoboOmni transfers to WidowX 250S by recognizing intent, asking for confirmation, and reliably executing confirmed actions.The real-world demonstrations use speech recorded by 10 volunteers in real environments and include distractor objects.Source: §5.4 · p. 8, Figure 5 · p. 9

5.5 Evaluation of Proactive Assistance Capabilities

  • 88.9% intent-recognition accuracy is highest among the compared models, while results show that contextual instructions require reasoning beyond acoustic modeling.GPT-4o reaches 55.6%, compared with 27.8% and 50.0% for the smaller Omni models in the reported comparison.Source: §5.5 · p. 9
  • RoboOmni proactively clarifies ambiguous requests, integrates speech with environmental sounds, and sustains dialogue that respects human agency.Baselines often ignore auditory cues, respond irrelevantly, or execute without clarification.Source: §5.5 · pp. 9–10, Figure 6 · p. 9

5.6 Further Analysis

  • OmniAction pretraining reaches nearly 90% accuracy within 2k steps, whereas from-scratch SFT reaches about 30% after 20k steps and degrades at 30k steps.This comparison supports faster and more stable adaptation with minimal fine-tuning.Source: Figure 7 · p. 9, §5.6 · p. 10
  • End-to-end RoboOmni outperforms planner–controller pipelines by avoiding semantic drift and preserving speaker-related paralinguistic cues.The cascaded comparison spans six contextual instruction types.Source: §5.6 · p. 10, Figure 8 · p. 10
  • 0.49× latency relative to ASR + OpenVLA’s 1.0× shows that RoboOmni removes the ASR bottleneck, while other cascades remain near 1×.ASR + NORA is 1.02× and ASR + π0 is 0.96× on a single RTX 4090 GPU.Source: §5.6 · p. 10

6 Conclusion

The paper concludes that cross-modal contextual instructions enable robots to infer intent from vision, environmental sounds, and speech rather than await explicit commands. RoboOmni combines this inference with confirmation and execution, and experiments show gains over text- and ASR-based baselines.

6 Conclusion

  • RoboOmni integrates auditory and visual inputs to unify intention recognition, confirmation, and action execution in manipulation.The framework is described as a Perceiver–Thinker–Talker–Executor system built on end-to-end omni-modal LLMs.Source: §6 · p. 10
  • Simulation and real-world experiments show higher success rates, faster inference, more effective proactive assistance, and improved intention recognition than text- and ASR-based baselines.The conclusion presents these results as evidence for RoboOmni’s broad performance advantage within the evaluated settings.Source: §6 · p. 10, Figure 9 · p. 10

A.1 Data Statics

OmniAction expands base robot trajectories into a diverse multimodal corpus spanning speakers, contextual audio, and manipulation skills. Its statistics characterize the dataset’s scale, acoustic diversity, temporal structure, and action vocabulary.

A.1 Data Statics

  • 74,645 base trajectories are expanded into 141,162 multimodal episodes from the Open-X dataset.Source: §A.1 · p. 19
  • OmniAction contains 5,096 speaker timbres across six demographic categories, including senior, adult, and child male and female voices.Source: §A.1 · p. 19
  • Most audio clips last 10–80 seconds; overlapping dialogues are shorter, while non-verbal sequences are longer because they include sound events.Source: §A.1 · p. 19
  • Processing 70k trajectories yields 112 unique skills and 748 manipulable objects from extracted verb–noun pairs.Source: §A.1 · p. 19

A.2 Data Example

The examples illustrate contextual instructions embedded in varied dialogue and audio phenomena, while verification checks whether intent is recoverable and the targeted phenomenon is faithfully realized.

A.2 Data Example

  • The examples cover non-verbal, overlapping, dyadic, triadic, sentiment, and identity cues linked to manipulation instructions.These scenarios embed actions in conversations, sound events, emotional cues, or speaker characteristics rather than presenting only direct commands.Source: §A.2 · pp. 20–21
  • Conditional sound cues can determine whether an action occurs, such as moving the pan only after a ceramic collision and not after a doorknob sound.Source: §A.2 · p. 20
  • Dialogue examples require the robot to infer an action, ask for confirmation, and execute after the user responds.The examples include placing a lid, moving a pot, and responding to conversationally embedded object references.Source: §A.2 · pp. 20–21

A.3 Details of Verfication

  • Human verification evaluates whether the canonical atomic intent is recoverable from dialogue and multimodal context.Annotators answer Yes or No, with Yes requiring reconstruction without external information.Source: §A.3 · p. 21
  • Verification also checks phenomenon fidelity for sentiment, overlap, non-verbal sounds, identity, and multi-party dialogue.Positive and negative examples are supplied for calibration before annotation.Source: §A.3 · p. 21

B.1 Success Cases

RoboOmni demonstrates successful real-world intent inference across visual, non-verbal, and dialogue contexts, while interacting naturally and executing the inferred actions. It also clarifies ambiguous situations, integrates environmental sounds, and preserves human agency through collaborative dialogue.

B.1 Success Cases

  • RoboOmni accurately infers intent from visual and audio cues, asks interactive questions, and reliably executes corresponding actions in real-world scenarios.The demonstrations cover Identity Cues, Non-verbal cues, Dyadic Dialogues, and Triadic Dialogues.Source: §B.1 · p. 21, Figure 12 · p. 22, Figure 13 · p. 22

B.2 Interaction Capability

  • RoboOmni provides proactive clarification for ambiguous contextual instructions instead of assuming the intended action or executing blindly.For example, it asks whether to put an egg dumpling into the hot pot when no explicit command is given.Source: §B.2 · p. 23
  • RoboOmni combines speech context with environmental sounds, such as a doorbell, to formulate contextually relevant confirmation questions.Baseline models are described as ignoring auditory cues or producing irrelevant responses in this scenario.Source: §B.2 · p. 23
  • RoboOmni uses collaborative questions such as “Would you like me to...?” to maintain a natural dialogue flow and respect human agency.This contrasts with baselines that often issue direct commands or statements.Source: §B.2 · p. 23

C Prompt Template

This section presents prompt templates for generating and structuring multimodal family dialogues and robot interactions. The templates support recoverable implicit actions, natural annotated exchanges, instruction consistency, and explicit execution confirmation.

C.1 Prompts for Data Generation

  • Non-verbal-cue prompts generate ambiguous family dialogue in which one selected sound determines between two action options and matches the robot instruction exactly.The dialogue uses [S1]/[S2] turns, inserts the selected sound with a [Sound] tag, and requires 4–5 rounds without revealing that speakers heard a sound.Source: §C.1 · p. 23
  • Sentiment-cue prompts require the target action or item to be inferred only from emotional or onomatopoeic cues that exclude incorrect options without directly stating the instruction.They require natural 4–6-round human dialogue, prohibit imperative refusals and explicit target mentions, and preserve exact consistency with the input instruction.Source: §C.1 · p. 25
  • Overlapping-cue prompts keep textually ambiguous choices resolvable only through interruption markers and visual context, with the resolved action matching the instruction.The format uses speaker labels, [Overlap] within interrupted speech, and [Overlap_Sx] for the interrupting speaker.Source: §C.1 · p. 26
  • Identity-cue prompts make speaker age, gender, and family role necessary for selecting among multiple possible intentions while keeping the dialogue text itself insufficient.Three speakers discuss everyday goals without mentioning the agent or directly restating the action, and the final instruction is conveyed through agreement with another speaker.Source: §C.1 · pp. 27–28
  • Identity-based dialogue construction grounds conversations in manipulable objects, named family characters, plausible underlying goals, and tones consistent with each speaker’s identity.The prompts encourage warm everyday small talk, fillers, jokes, teasing, and lifelike polishing rather than mission-like planning.Source: §C.1 · pp. 27–28
  • Dyadic templates specify two named speakers, a JSON conversation field, at most five short sentences, and an ending where one speaker clearly states what they want without directly instructing the agent.The associated guidelines require identifying manipulable objects, setting a plausible goal, and matching natural dialogue to speaker identities.Source: §C.1 · pp. 28–29
  • Triadic templates extend the same implicit-action design to three named speakers using [S1], [S2], and [S3], with natural identity-aware dialogue and a clear final desire.The output remains a JSON object containing the conversation, and the dialogue begins with a user utterance without a greeting.Source: §C.1 · p. 29

C.2 Prompts for Interaction Extension

  • Interaction-extension prompts convert an original two-person dialogue and instruction into a 2–4-turn JSON human–robot exchange ending with explicit action confirmation and an [ACT] tag.The first turn contains the original dialogue placeholder; later user turns use one speaker’s utterance, while robot replies remain short and service-oriented.Source: §C.2 · pp. 29–30
  • The template begins with a scene description: {scene_description}.Source: §C.2 · p. 30
  • The scene description is represented by the placeholder {scene_description}.Source: §C.2 · p. 30
  • The template includes the original dialogue: {conversation}.Source: §C.2 · p. 30
  • The original dialogue is represented by the placeholder {conversation}.Source: §C.2 · p. 30
  • The template includes a robot execution instruction: {instruction}.Source: §C.2 · p. 30
  • The execution instruction is represented by the placeholder {instruction}.Source: §C.2 · p. 30

D.1 Data Example

The data examples pair contextual audio with LIBERO manipulation tasks and conversational transcripts, covering identity, sentiment, dialogue, overlapping-voice, and non-verbal cues. These examples show how indirect conversations and environmental sounds specify the intended action.

D.1 Data Example

  • Sentiment Cues are included as an audio category in the contextual instruction examples.The supplied example identifies the audio type but does not provide a corresponding transcript or task instance.Source: §D.1 · p. 30
  • Overlapping Voices are represented in a LIBERO Object task where speakers clarify which cream cheese item should be placed in the basket despite an overlap marker.The dialogue includes explicit overlap annotations during the object clarification.Source: §D.1 · p. 30
  • Non-Verbal Cues connect environmental sounds to alternative actions, such as selecting alphabet soup after a drawer-closing sound or dressing after a coffee-machine sound.The example’s target task is to pick up the alphabet soup and place it in the basket.Source: §D.1 · p. 30
  • Identity Cues pair speaker identity information with a LIBERO Spatial task and a multi-person conversation about moving the black bowl onto the plate.The example lists male-child, female-senior, and female-child speakers and the corresponding task instruction.Source: §D.1 · p. 31
  • Dyadic Dialogue and Triadic Dialogue examples use conversational context to specify opening the middle drawer or placing soup and cream cheese in the basket.The dyadic example involves a mother and grandmother, while the triadic example involves a daughter, grandpa, and mother.Source: §D.1 · p. 31

Related papers

Loading 2510.23763v3…