PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Speech-Centric Omni Understanding with Frozen VLMs

Ankan Deria, Hanoona Rasheed, Xilin He, Fahad Shahbaz Khan, Salman Khan

arXiv:2609.04242v1eess.AScs.CVcs.SD

TL;DR

Audio-visual understanding requires models to connect speech, visual events, and temporal relationships, while native omni systems rely on costly backbone-specific audio training. TFO instead routes confidence-filtered, timestamped Whisper transcripts through a frozen VLM’s language interface, preserving its visual pathway. Across broad matched evaluations, it is competitive for speech-centric omni understanding and generally preserves stronger original VLM capabilities, but remains limited when essential evidence is non-speech acoustic content.

  • Problem

    Native omni models require costly audio encoders and multimodal alignment, motivating evidence on whether speech-centric capability can be added without retraining each VLM backbone.

  • Method

    TFO converts a frozen VLM into a speech-centric omni model by routing confidence-filtered, timestamped Whisper transcripts through its existing language interface.

  • Results

    Across 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves audio-only and multilingual speech understanding, and generally preserves broader VLM capabilities.

  • Takeaways & Limitations

    Strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.

  • Takeaways & Limitations

    TFO adds ASR latency and remains limited when tasks require music, environmental sounds, or other acoustic evidence absent from speech transcripts.

Abstract

from arXiv · show

Audio-visual understanding remains challenging because models must jointly interpret spoken content, visual events, and their temporal relationships. Existing omni models typically introduce dedicated audio encoders and rely on expensive audio-video-text training, tightly coupling omni capability to specific VLM backbones and potentially weakening their existing visual and reasoning abilities. This raises three questions: whether native omni training is necessary for every new VLM, whether speech-centric omni capability can be added while preserving the original backbone, and where richer acoustic representations remain essential. We introduce Training-Free Omni (TFO), a plug-and-play framework that converts a frozen VLM into a speech-centric omni model without architectural modification, or multimodal re-alignment. TFO uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, while leaving its visual pathway unchanged. Across matched comparisons with native omni models on 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves average audio-only performance across all five model settings, and achieves substantial multilingual speech gains. Freezing the VLM also generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical question answering than the corresponding native omni checkpoints. These results show that strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.

1. Introduction

Native omni training is costly, backbone-specific, and may weaken existing VLM capabilities, leaving open whether speech-centric omni understanding can be added through a matched training-free alternative. TFO addresses this question by routing filtered, timestamped speech transcripts through a frozen VLM while preserving its visual pathway.

  • Motivation: Native omni models require dedicated audio encoders and large-scale multimodal alignment, tying omni capability to particular VLM backbones.As newer VLMs improve, their perception, reasoning, and knowledge capabilities may not transfer automatically to existing omni models.
  • Motivation: Audio integration is difficult because signals are noisy, temporally dense, and must align spoken content with visual events.Prior studies report that omni models may overlook relevant audio, infer sounds from visual cues, or miss subtle cross-modal relationships.
  • Research question: Existing evaluations do not establish whether learned audio pathways are necessary for every task because they lack matched training-free comparisons.The paper asks whether a simple alternative can provide comparable speech-centric omni understanding for new VLM backbones.
  • Approach: TFO converts a frozen VLM into a speech-centric omni model by routing Whisper transcripts through its existing language interface without architectural or parameter updates.Confidence filtering removes unreliable transcripts, while timestamps preserve associations between spoken segments and visual events.
  • Evaluation: Across 56 benchmarks and 21 languages, the study compares native omni models, original VLM backbones, and training-free conversions across multiple model families and scales.The evaluation examines audio-visual and audio-only understanding, visual capabilities, reasoning, medical question answering, and multilingual speech.
  • Findings: TFO is competitive for spoken-content audio evidence and generally retains stronger visual, reasoning, coding, medical, and grounding capabilities than native omni models.Its limitations include additional ASR latency and weaker handling of music, environmental sounds, and other non-speech acoustic cues.

2. Related Work

Prior omni systems commonly add acoustic encoders and train cross-modal connectors, creating backbone-specific alignment costs. TFO differs by freezing the entire VLM and supplying speech through its existing language interface.

  • Existing omni approaches: Audio-language and omni models typically extend an LLM or VLM with acoustic encoders and connectors followed by audio-text or audio-video-text alignment.These audio pathways remain tied to particular backbones and training recipes.
  • Existing omni approaches: Each new VLM generation may therefore require costly multimodal training and cross-modal re-alignment.Approaches that freeze parts of the model still train connectors or alignment modules.
  • Backbone preservation: Existing VLMs already provide strong image and video understanding, visual grounding, language reasoning, and domain knowledge.Prior work has explored freezing language backbones to reduce capability drift, but speech modules and alignment stages remain trained.
  • TFO distinction: TFO instead freezes the entire VLM rather than training speech modules or alignment stages.This design directly supports matched comparisons of acquired audio capability and preservation of original VLM abilities.

3. Methodology

TFO combines Whisper-derived speech evidence with visual inputs and user queries in a standard prompt, then runs a frozen VLM without changing its visual pathway. The pipeline filters unreliable segments, preserves timestamps for temporal reasoning, and optionally synthesizes spoken responses.

  • Pipeline overview: TFO accepts speech, image, video, and text inputs and combines their representations with the user query.Whisper supplies transcript, language, confidence, and timestamp information before prompt construction.
  • Output: Generated text can optionally be converted into speech with CosyVoice3, which is used only for response generation and not benchmark audio construction.The spoken-output component therefore does not affect reported benchmark results.
  • Multimodal fusion: Audio is routed through language rather than a learned audio-token pathway, conditioning the frozen VLM through its standard prompt interface.The visual pathway, architecture, and parameters remain unchanged.
  • Audio-to-language routing: Whisper decomposes audio into segment-level transcriptions containing spoken text, detected language, start and end times, and confidence.This representation retains speech content, temporal boundaries, and transcription reliability.
  • Audio-to-language routing: TFO retains only segments above a confidence threshold, using the filtered transcript as the audio-derived evidence for fusion.The threshold is τ = 0.65; if no segment passes, the audio context is omitted.
  • Multimodal fusion: The retained transcript becomes a language-side audio context inserted into the VLM prompt alongside visual input and the user query.Timestamps are included when temporal audio-video reasoning is required.

4. Experiments

Across matched evaluations, TFO provides competitive speech-centric Omni understanding through language-level audio routing while generally preserving the frozen VLM’s visual, reasoning, and domain capabilities. Its benefits are strongest for spoken content and multilingual speech, whereas non-speech acoustic reasoning remains a limitation and inference cost increases.

  • Experimental setup: TFO compares frozen VLM conversions with native Omni counterparts across 56 benchmarks spanning audio-visual, audio-only, visual, reasoning, medical, grounding, and multilingual tasks.The evaluation covers five matched comparisons across four model families and 21 CoVoST2 languages.
  • Audio-visual and audio-only understanding: TFO is competitive with native Omni training when audio evidence is primarily spoken content, but native models can retain an advantage for richer acoustics or tightly learned audio-visual alignment.Transcript-based routing is therefore most suitable when speech carries the task-relevant evidence.
  • Audio-visual and audio-only understanding: +3.0 and +2.2 points are the Qwen2.5 3B and 7B audio-visual average gains, with the strongest improvements in speech-conditioned and temporal video reasoning.WorldSense, Video-Holmes, AVUT-Human, and Daily-Omni show especially large gains across the two Qwen2.5 scales.
  • Audio-visual and audio-only understanding: +1.5, +1.4, +4.0, +13.5, and +1.4 points are the audio-only average gains across all five matched settings, while MMAR-Bench declines by 1.5–9.5 points.The contrast highlights strong speech-dominant performance but limited coverage of music, sound events, and broader non-speech acoustics.
  • Multilingual speech: +7.9, +13.5, +18.4, +11.1, and +6.8 points are the multilingual speech average gains across five settings and 21 languages.The largest reported gain is MiniCPM4.5, increasing from 45.6 to 64.0.
  • Backbone preservation and trade-offs: TFO cannot represent acoustic evidence absent from transcripts, and auxiliary textual predictions from MELLOW and SenseVoice do not consistently improve over Whisper alone.This limits transcript-based routing for non-speech audio and richer acoustic evidence.
  • Backbone preservation and trade-offs: Segment-level timestamps provide targeted benefits for temporal audio-video alignment, while the frozen VLM supplies video reasoning and Whisper supplies speech understanding.Removing timestamps reduces performance on AVUT-Gemini and Daily-Omni by 2.3 and 1.4 points, respectively.

5. Conclusion

TFO routes confidence-filtered, timestamped speech transcripts through a frozen VLM’s language interface without modifying its architecture or visual pathways. Across 56 benchmarks and 21 languages, it supports speech-centric omni understanding while generally preserving stronger capabilities than corresponding native Omni models.

  • TFO routes confidence-filtered, timestamped speech transcripts through a frozen VLM’s language interface without modifying its architecture or visual pathways.
  • Across 56 benchmarks and 21 languages, TFO performs similarly to or better than native Omni models when spoken content and temporal speech-video evidence are central.
  • Freezing the backbone generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, medical knowledge, and downstream capabilities than corresponding native Omni models.

A. Benchmark Suite

The benchmark suite evaluates TFO across capability axes using distinct underlying datasets, with related subsets and variants counted once. It spans multimodal understanding, reasoning, medical question answering, and multilingual speech.

  • The evaluation suite counts each underlying benchmark dataset once rather than separately counting every subset, split, or evaluation variant.
  • The suite groups related evaluations such as UnoBench, AVMeme, VideoMME, MBPP, MMMU-Med, and Medbullets under single dataset counts.
  • CoVoST2 is counted once despite being used in two evaluation contexts.

B. Evaluation Protocol

The evaluation uses task-specific metrics and standardized, deterministic inference across multimodal benchmarks. OmniEvalKit provides a unified interface for loading data, constructing prompts, running models, parsing outputs, and computing metrics.

  • Multiple-choice tasks use exact-match accuracy, while long-form generation is judged by GPT-5.6 using reference responses and model predictions.
  • Each model is evaluated once per benchmark with greedy decoding at temperature zero for repeatable outputs.
  • Experiments run on AMD Instinct MI210 GPUs with 64 GB of memory per GPU.
  • OmniEvalKit standardizes dataset loading, prompt construction, inference, output parsing, and metric computation across task formats.

C. Additional Implementation Details

TFO converts retained transcripts into a language-side audio context and combines that context with the original visual input and user query. Explicit transcript labels preserve audio-source identity while timestamps support temporal speech-video reasoning.

  • TFO converts the confidence-filtered transcript into a language-side audio context containing retained spoken segments and, when needed, their timestamps.
  • The final prompt combines the system instruction, unchanged visual pathway, routed speech context, and user query; without retained audio, the original image/video-text prompt is used.
  • Separate audio files and embedded video audio receive explicit transcript labels, and both blocks are included when both sources are available.

C.2. Audio Preprocessing.

TFO preprocesses audio into confidence-filtered Whisper transcripts, preserving language and timestamps for insertion into generalized VLM prompts. The evaluation suite covers 56 underlying datasets, while prompt blocks include only available transcript sources.

  • Audio preprocessing: Audio is transcribed with Whisper after conversion to mono 16-kHz waveforms, language identification, and confidence filtering at 0.65.Long Qwen-based audio inputs use 30-second chunks with 3-second overlap; unreliable or absent speech can produce no transcript context.
  • Evaluation scope: The benchmark suite contains 56 distinct datasets spanning audio-visual, image/video, audio-only, coding/math, medical QA, and grounding tasks.The 21 CoVoST2 language splits are not counted as separate datasets.
  • Prompt construction: The prompt template combines dataset-specific system instructions with language and confidence metadata and transcript text.The supplied template shows a dedicated audio-file transcript block.
  • Prompt construction: The generalized prompt uses separate or embedded audio labels when both sources exist and includes only available, confidence-filtered transcripts.With one source, the prompt uses a single Audio Transcript block.

D. Additional Ablations

The ablations identify reliable timestamped Whisper transcription as TFO’s main source of gains, while auxiliary acoustic descriptions help selectively but do not generalize consistently. Timestamps matter most for temporally grounded audio-visual reasoning, and confidence filtering balances noise suppression against speech retention.

  • Ablation overview: Whisper plus timestamps provides the strongest overall TFO configuration, while auxiliary audio models offer no consistent improvement.This pattern is summarized for both audio-visual and audio-only ablations.
  • Audio-visual ablation: Removing timestamps reduces AVUT-Gemini by 2.3, AVMeme-Full by 1.7, and Daily-Omni by 1.4 points.These drops confirm the value of temporal anchors for grounded reasoning.
  • Audio-visual ablation: SenseVoice improves WorldSense by +5.3 points, but auxiliary descriptions regress several other benchmarks and none consistently outperforms timestamped Whisper.MELLOW improves WorldSense by +4.3 but reduces Video-Holmes by 1.8, while AF2 drops Daily-Omni by 21.1.
  • Audio-only ablation: Whisper alone outperforms Qwen2.5-Omni on eight of ten audio-only benchmarks, including Audio Web by +5.3 and CoVoST2 by +5.4.Its largest reported weakness is a -5.7-point gap on MMAR-Bench.
  • Audio-only ablation: Auxiliary acoustic outputs can benefit isolated tasks but reduce performance elsewhere, with their combination lowering CoVoST2 by 9.7 and LiveSports3K by 9.5 points.The results indicate that simple textual concatenation does not consistently recover non-speech information.
  • Confidence filtering: Moderate confidence filtering reduces unreliable ASR outputs without sacrificing useful speech, whereas excessive filtering can remove informative speech.Thresholds 0.60 and 0.65 perform similarly, with 0.65 selected by default.

E. Benchmark-Specific Evaluation Metrics and Aggregation

The evaluation uses task-specific accuracy, semantic, translation, BLEU, and judge-based metrics, with aggregation rules designed to avoid overweighting dataset variants. Multilingual translation is averaged across 21 languages, while benchmark-specific protocols define how predictions count as correct.

  • Audio-visual evaluation: The audio-visual macro-average weights each of nine underlying benchmarks once, excluding UnoBench-MC and AVMeme-Main diagnostic subsets.Closed-form questions use exact option accuracy, while UnoBench open-ended samples use binary semantic correctness.
  • Audio-only evaluation: Audio benchmarks use higher-is-better metrics, with spoken questions presented as audio only and semantic correctness judged by GPT-5.6 Sol where specified.The stored text question is not used as model input for Audio Trivia and Audio Web Questions.
  • Multilingual translation: CoVoST2 reports thresholded translation accuracy and BLEU-1 across 21 languages using an unweighted macro-average.Thresholded correctness accepts exact normalized matches or ROUGE-L F1 of at least 0.40.
  • Multilingual translation: TFO improves average CoVoST2 BLEU-1 across all five model settings by +3.4, +7.4, +12.7, +6.5, and +2.5 points.The largest gain is on MiniCPM4.5, increasing from 33.6 to 46.3.
  • Visual evaluation: Image and video benchmarks use LLM-judged semantic correctness or exact programmatic and multiple-choice accuracy, depending on the task.The listed video benchmarks use exact multiple-choice accuracy.
  • Reasoning evaluation: Coding and mathematical reasoning are reported as binary LLM-judge correctness or exact option-match accuracy, not official execution-based pass@1.Coding scores judge whether responses specify complete and logically correct solutions.
  • Medical evaluation: Medical QA uses MedEvalKit with exact option matching for multiple-choice tasks and benchmark-specific normalization for open-ended VQA.All medical results are reported as accuracy percentages.
  • Reproducibility: CosyVoice3 is used only for optional spoken response generation and does not affect evaluation audio or benchmark results.The complete evaluation codebase is included for reproducibility.

F. Qualitative Results

The qualitative examples show TFO using timestamped transcripts for temporal grounding and translated speech text for multilingual audio-visual reasoning. In both cases, the VLM combines language-side audio evidence with visual context.

  • Temporal grounding: Timestamped transcript segments let the VLM match a queried utterance to the correct audio interval and select the corresponding answer.The example uses the utterance “welcome back we’re back from executive session” as the temporal cue.
  • Multilingual understanding: Chinese speech paired with an English question enables the VLM to use textual audio evidence and visual context to identify humor from contrasting polite language and violent actions.The example demonstrates cross-lingual audio-visual reasoning without native audio-language training.
Loading 2609.04242v1…