PaperlayerView exact revision on arXiv

Source-linked AI summary

LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

Yifan Dai, Zhenhua Wu, Bohan Zeng, Daili Hua, Jialing Liu, Bozhou Li, Yuran Wang, Chengzhuo Tong, Hao Liang, Xiaochen Ma, Junbo Niu, Tianyu Guo, Yang Shi, Yue Ding, Yiyan Ji, Bingyin Mei, Yushuo Guan, Yuanxing Zhang, Pengfei Wan, Fangcheng Fu, Wentao Zhang

arXiv:2605.22012v1cs.CLcs.CV

TL;DR

Audio-visual MLLMs struggle to reason over fine-grained evidence across modalities. LatentOmni interleaves textual reasoning with synchronized audio-visual latent states, and achieves the best performance among evaluated open-source models across four benchmarks while outperforming Explicit Text CoT.

  • Problem

    Audio-visual MLLMs remain limited on reasoning problems requiring fine-grained integration of temporal and semantic evidence across modalities.

  • Method

    LatentOmni interleaves textual reasoning with audio-visual latent states using feature-level supervision, OSPE temporal alignment, and the LatentOmni-Instruct-35K dataset.

  • Results

    Across four omnimodal benchmarks, LatentOmni achieves the best performance among evaluated open-source models and consistently outperforms the Explicit Text CoT baseline.

  • Takeaways & Limitations

    The results support latent-space joint reasoning as a promising approach for stronger omnimodal understanding.

  • Takeaways & Limitations

    LatentOmni does not yet cover broader physical and interactive signals such as 3D spatial representations, tactile physics, and motor action commands.

Abstract

from arXiv · show

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose \textbf{LatentOmni}, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct \textbf{LatentOmni-Instruct-35K}, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.

1 Introduction

LatentOmni addresses the limitations of text-based chain-of-thought for fine-grained audio-visual reasoning by interleaving textual reasoning with audio-visual latent states. It combines feature-level supervision, OSPE, and LatentOmni-Instruct-35K, achieving strong performance across audio-visual benchmarks.

  • Motivation: Current MLLMs struggle with reasoning that requires integrating fine-grained evidence across audio and visual modalities.Although they have progressed on captioning and grounding, joint interpretation remains constrained for reasoning tasks.
  • Motivation: Explicit text-based CoT compresses temporally aligned audio-visual evidence into discrete tokens, encouraging reliance on language priors over sensory evidence.This textual bottleneck causes models to under-attend to original audio-visual modalities during reasoning.
  • Method: LatentOmni interleaves textual reasoning with audio-visual latent states in a unified latent space to preserve fine-grained sensory features.The framework is presented as a post-training approach for joint audio-visual reasoning.
  • Data: LatentOmni-Instruct-35K fills the shortage of tailored training data for complex cross-modal latent reasoning with audio-visual interleaved CoT trajectories.The dataset is produced through a scalable audio-video interleaved CoT data synthesis pipeline.
  • Results: LatentOmni achieves the best results among evaluated open-source models on all four benchmarks and substantially outperforms the Explicit Text CoT baseline.The introduction attributes these gains to improved attention to original audio-visual modalities, particularly on audio-visual alignment tasks.
  • Method: Feature-level supervision and Omni-Sync Position Embedding facilitate cross-modal temporal alignment while preserving attention to audio-visual modalities.OSPE helps bridge audio-visual and textual semantics, while Figure 1 reports higher audio-visual token attention than the Explicit Text CoT baseline.

2 Related work

Related work has advanced from explicit text-based chain reasoning toward continuous latent-space reasoning, but text bottlenecks can lose audio-visual information and existing latent methods rarely address unified dynamic audio-visual reasoning. LatentOmni is positioned to fill this gap by extending continuous latent reasoning to omnimodal scenarios with temporal and semantic alignment.

  • Explicit cross-modal reasoning: MLLM research has shifted from adding diverse perceptual capabilities toward improving reasoning for complex real-world tasks through explicit chain techniques.These approaches use text as the primary semantic bridge for cross-modal integration and decompose complex tasks through natural-language reasoning.
  • Explicit cross-modal reasoning: Text-centric discrete reasoning constrains complex cross-modal inference by forcing high-dimensional audio-visual signals through a narrow textual bottleneck.The resulting information loss and reduced attention to raw audio-visual signals can produce sensory detachment and multimodal hallucinations.
  • Continuous latent reasoning: Recent work mitigates discrete-token constraints by performing reasoning directly in continuous latent spaces, with Coconut executing intermediate reasoning entirely in hidden states.This paradigm bypasses autoregressive generation of intermediate textual tokens and has been extended toward multimodal settings.
  • Continuous latent reasoning: Existing latent reasoning methods mainly target pure text or single-modality extensions, leaving unified continuous-space reasoning over dynamic audio-visual signals underexplored.LatentOmni addresses this gap by extending continuous latent reasoning to omnimodal scenarios and explicitly targeting temporal and semantic alignment of cross-modal audio-visual information.

3 Method

LatentOmni interleaves textual deduction with continuous audio-visual latent reasoning in a shared space, using modality-specific latent allocation and OSPE for temporal consistency. Its training combines trajectory supervision, sensory grounding, temporal synchronization, and next-token prediction, supported by the LatentOmni-Instruct-35K dataset.

  • Interleaved latent reasoning: LatentOmni alternates explicit textual deduction with continuous latent phases that revisit dense audio-visual evidence, preserving text as the scaffold for high-level logic.The hybrid trajectory inserts latent states between text segments and returns to textual generation after each continuous reasoning phase.
  • Latent representation: The model allocates Kv visual and Ka audio latent positions within the same continuous space Rd, enabling modality-specific capacity control.Each generated latent state is fed back as the next input embedding, forming an autoregressive trajectory of length K.
  • Temporal synchronization: Omni-Sync Position Embedding assigns shared physical timestamps to corresponding visual frames and audio segments, aligning sequentially generated latents within the same time window.OSPE extends time-aligned multimodal RoPE to the unified latent space so later reasoning can attend to temporally consistent evidence.
  • Dataset construction: LatentOmni-Instruct-35K synthesizes segment-grounded interleaved reasoning trajectories through AVQA filtering, localized caption synthesis, and trajectory auditing.The pipeline uses temporally aligned source data, separate audio and video captions, explicit segment markers, and auditing to correct citation errors and remove inconsistent branches.
  • Training objectives: Training optimizes complementary objectives that synchronize audio-visual features, ground latent embeddings in sensory anchors, and preserve explicit language generation.The temporal synchronization loss pulls together co-occurring features and separates asynchronous pairs, while next-token prediction applies to discrete vocabulary elements in the hybrid sequence.

4 Experiments

LatentOmni is evaluated across four complementary omnimodal benchmarks and achieves the best performance among evaluated open-source methods on all four. Controlled comparisons and ablations show consistent gains over text-only reasoning and identify synchronized multimodal latent representations, OSPE, and a 40-token configuration as important design choices.

  • Benchmarks: The evaluation spans Daily-Omni, WorldSense, OmniVideoBench, and LVOmniBench, covering everyday events, spatiotemporal commonsense, cross-modal QA, and long-form multisensory understanding.The suite includes fine-grained audio-type and video-duration splits and sustained reasoning over longer inputs.
  • Main Results: LatentOmni achieves the best performance among evaluated open-source methods on all four omnimodal benchmarks.The main comparison focuses on open-source models, text-only reasoning variants, and latent reasoning baselines, while proprietary systems provide context.
  • Comparison with Open-Source Models: 4.5%, 3.5%, 6.1%, and 3.1% are LatentOmni’s absolute gains over Qwen2.5-Omni-7B on Daily-Omni, WorldSense, OmniVideoBench, and LVOmniBench, respectively.LatentOmni also outperforms OmniVinci and HumanOmniV2-7B on benchmarks where they report results.
  • Comparison with Text CoT: 1.8%, 2.3%, 2.2%, and 3.0% are LatentOmni’s accuracy gains over Explicit Text CoT on Daily-Omni, WorldSense, OmniVideoBench, and LVOmniBench, respectively.Relative to Vanilla SFT, the largest reported improvements are +5.4% on Daily-Omni and +4.9% on OmniVideoBench.
  • Fine-Grained OmniVideoBench Analysis: 35.4% is LatentOmni’s highest average accuracy among open-source methods on OmniVideoBench, improving over the base model by 6.1pp.It leads on music and speech questions, all short-to-medium duration buckets, and ties for the best score on the longest videos.
  • Ablation Studies: Removing OSPE reduces accuracy on every benchmark, while removing either modality also degrades performance, confirming the importance of multimodal temporal alignment.Examples include 67.4 → 66.0 on Daily-Omni and 35.1 → 33.1 on LVOmniBench; 40 total latent tokens with 32 visual and 8 audio tokens performs best.

5 Conclusion

LatentOmni addresses audio-visual reasoning in MLLMs by interleaving explicit textual reasoning with synchronized latent audio-visual states. Its design keeps intermediate reasoning grounded in native sensory evidence through feature-level supervision, OSPE, and LatentOmni-Instruct-35K.

  • 5 Conclusion: LatentOmni interleaves explicit textual reasoning with synchronized latent audio-visual states for audio-visual reasoning in MLLMs.The framework is designed to avoid forcing every reasoning step through a text-only bottleneck.
  • 5 Conclusion: The framework keeps intermediate reasoning grounded in native sensory evidence rather than compressing every step into text.This grounding is presented as the key idea behind LatentOmni’s approach.
  • 5 Conclusion: LatentOmni introduces feature-level latent supervision, Omni-Sync Position Embedding (OSPE), and LatentOmni-Instruct-35K.These components support latent supervision, cross-modal temporal alignment, and supervision for the proposed framework, respectively.

Appendices … A.6 AV Interleaved Reasoning Trajectory Synthesis Instruction

The appendices detail the datasets and instruction pipelines used to build temporally aligned audiovisual captions, AVQA pairs, segment captions, and interleaved reasoning trajectories. These procedures emphasize cross-modal alignment, quality control, modality-specific evidence, narrative coherence, and step-by-step multimodal integration.

  • A.1 Caption Database: AvoCaDO contains 107K high-quality, temporally aligned audiovisual video captions emphasizing synchronized visual and auditory events.It is designed to improve temporal coherence, dialogue accuracy, and comprehensive multimodal alignment in audiovisual video captioning.
  • A.1 Caption Database: ASID-1M provides one million structured, fine-grained audiovisual instruction annotations spanning scenes, objects, actions, speech, camera movements, and narrative elements.An automated verification and refinement pipeline supports controllable, reliable video understanding and mitigates hallucinations.
  • A.2 AVQA Synthesis Instruction: AVQA synthesis instructions generate open-ended and multiple-choice question-answer pairs with strict cross-modal dependencies and diverse coverage of source captions.The complete open-ended and MCQ prompts are presented in Fig. 5 and Fig. 6.
  • A.3 AVQA Classification and Filtering Instruction: A joint GLM-4.7 instruction classifies each preliminary AVQA pair’s reasoning type and assesses quality for threshold-based filtering.The detailed classification and evaluation prompt is shown in Fig. 7.
  • A.4 Segment Level Caption Synthesis Instruction: Two modality-specific prompts synthesize concise, objective visual and audio segment captions strongly correlated with QA pairs.Separating observable visual elements from identifiable audio elements helps prevent information omission; the prompts appear in Fig. 8 and Fig. 9.
  • A.5 Segment Level Caption Fusion and Refinement Instruction: Sequential fusion and refinement instructions produce cohesive audiovisual captions while resolving narrative fragmentation caused by shot transitions.The fusion and refinement prompts are detailed in Fig. 10 and Fig. 11.
  • A.6 AV Interleaved Reasoning Trajectory Synthesis Instruction: A GLM-4.7 instruction combines generated AVQA pairs with aligned segment-level captions into cohesive, logical, step-by-step audiovisual reasoning trajectories.The trajectory-synthesis prompt is illustrated in Fig. 12.

B Implementation Details

LatentOmni uses constrained optimization settings and a staged prompt pipeline to construct, evaluate, refine, and supervise audio-visual reasoning data. The prompts emphasize temporally aligned multimodal evidence, complex cross-modal reasoning, structured outputs, and grounded interleaved trajectories.

  • Training Configuration: Training caps each sample at 256 frames, uses a 10^-5 learning rate with 0.05 warmup, sets λ1 = 0.005 and λ2 = 1.0, and pairs batch size 1 with 12 gradient accumulation steps.These settings address limited computational resources while maintaining the specified optimization schedule and loss weighting.
  • Dataset Synthesis: Open-ended and multiple-choice AVQA prompts require exactly one complex, multi-step cross-modal question-answer pair from temporally aligned audio-video captions.The multiple-choice format additionally requires four distinct options, a correct answer, and the employed reasoning type.
  • Caption Processing: Additional prompts generate visual-only and audio-only captions, fuse segment captions, and refine fragmented video segments against full captions while preserving temporal constraints.The refinement step addresses shot fragmentation by cross-referencing fragmented segments with the full video caption.
  • Reasoning Trajectory Synthesis: The interleaved-trajectory prompt treats detailed audio-video segments as direct sensory perception and requires step-by-step analysis with explicit segment citations and concise grounded observations.This prompt frames the task as describing raw video and audio rather than analyzing text.

C Case Study

The case study uses DailyOmni examples to illustrate LatentOmni’s reasoning across diverse audio-visual tasks without projecting latent embeddings into discrete language space. Attention visualizations show latent states dynamically tracking fine-grained audio-visual evidence during generation.

  • Case Study: DailyOmni case studies demonstrate LatentOmni’s reasoning capabilities across diverse audio-visual tasks.The examples cover three representative scenarios, including AV Event Alignment.
  • Case Study: The generated latent embeddings are represented as <Unified_Latent><latent_embeddings></Unified_Latent> because projecting them into discrete language space would produce uninterpretable tokens.This presentation avoids converting latent reasoning segments into discrete textual tokens.
  • Case Study: Attention maps visualize interactions between latent reasoning tokens, original audio-visual inputs, and generated outputs.The visualizations show latent states dynamically tracking and anchoring to fine-grained multimodal evidence during generation.

D Limitation

LatentOmni establishes unified latent reasoning across visual, auditory, and textual modalities but shares current multimodal systems’ limitation in modality coverage. Real-world environments include additional sensory and control signals, whose integration remains unresolved.

  • Modality coverage: LatentOmni’s modality coverage remains bounded similarly to current state-of-the-art multimodal systems.The framework spans visual, auditory, and textual modalities, but the passage identifies modality coverage as a common boundary.
  • Modality coverage: Real-world environments encompass broader sensory and control signals, including 3D spatial representations, tactile physics, and motor action commands.These signals extend beyond the modalities addressed by LatentOmni.
  • Modality coverage: Mapping extended physical and interactive signals remains a current limitation.The supplied passage ends while describing this mapping challenge, without specifying a completed solution.
Loading 2605.22012v1…