Training-Free Speech-Centric Omni Understanding with Frozen VLMs

• Native Omni training adds dedicated audio encoders and costly backbone-specific alignment, may weaken VLM skills, and existing evaluations lack matched training-free comparisons despite diverse audio needs and audio-visual alignment failures.Source: §1 · pp. 1–2
• TFO uses Whisper to extract confidence-filtered transcripts and routes them through the VLM’s language interface, retaining timestamps for temporal reasoning and omitting audio context when no transcript passes the threshold.The VLM architecture and parameters remain unchanged, and TFO requires no audio-video-text training.Source: §1 · p. 2
• The matched study compares native Omni models, original VLMs, and TFO conversions across model families and scales on 56 benchmarks, including multilingual speech across 21 languages.Evaluation spans audio-visual and audio-only understanding, image and video understanding, visual grounding, coding, mathematical reasoning, and medical question answering.Source: §1 · p. 2
• TFO matches or beats native Omni models on several speech-centric audio-visual benchmarks, improves audio-only and multilingual speech, and generally preserves VLM capabilities better, but adds ASR latency and misses non-speech cues.The limitation applies to music, environmental sounds, and other acoustic cues that transcript-based routing cannot preserve.Source: §1 · p. 2

2.1. Native Omni Models and Backbone-Specific Alignment

• Native Omni models [18] [64] [65] [66] [68] [71] add backbone-tied acoustic pathways and alignment; TFO instead leaves the complete VLM unchanged.Earlier transcript and subtitle approaches [8] [40] [73] were not systematically evaluated as matched training-free controls for native Omni training or backbone preservation; partial-freezing methods [74] [62] still train connectors or alignment modules.Source: §2.1 · p. 2

2.2. Frozen VLMs and Capability Preservation

• TFO evaluates both speech-centric Omni capability gained through audio-to-language routing and original VLM capabilities retained by freezing the entire VLM.These capabilities include visual understanding, visual grounding, language reasoning, and domain knowledge [3] [4] [46] [70]; prior work froze the language backbone [62] but still trained speech modules and alignment stages.Source: §2.2 · p. 2

3.1. Overview

• TFO routes audio as language-level evidence to a frozen VLM, leaving its architecture, parameters, and visual pathway unchanged.The framework uses the VLM’s existing capacity to reason over language-conditioned visual inputs rather than adding a learned audio-token pathway.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2. Audio-to-Language Routing

• Whisper [56] produces segment-level transcripts with detected language, timestamps, and confidence, and TFO retains only segments above the 0.65 threshold.If no segment passes, the audio context is omitted and the VLM receives its original visual-language input.Source: §3.2 · p. 3

3.3. TFO Multimodal Fusion

• TFO inserts retained speech segments and, for temporal audio-video reasoning, their timestamps into the standard prompt alongside visual input and the user query.If there are no retained segments, audio context is omitted; generated text can optionally be converted to speech with CosyVoice3 [22].Source: §3.3 · p. 3

4.1. Evaluation Design

• The evaluation comprises five matched comparisons across four model families and 56 benchmark datasets, including multilingual speech evaluation in 21 CoVoST2 languages.Coverage includes audio-visual, image/video, audio-only, coding, mathematical reasoning, medical question answering, and visual grounding.Source: §4.1 · p. 4

4.2. When Is Language-Level Audio Routing Sufficient?

• TFO improves audio-only averages in all five settings (+1.5, +1.4, +4.0, +13.5, and +1.4 points) and raises Qwen2.5 audio-visual averages by +3.0/+2.2, while other comparisons vary.VILA gains +0.4 points; MiniCPM4.5 and Qwen3 decline by 2.2 and 0.8 points, respectively. All variants decline on MMAR-Bench by 1.5–9.5 points, showing transcript routing’s limits for non-speech acoustic reasoning.Source: §4.2 · pp. 4–5, Table 2 · p. 5
• Across 21 languages, TFO improves CoVoST2 averages in all five settings by +7.9, +13.5, +18.4, +11.1, and +6.8 points.The largest average gain is MiniCPM4.5, from 45.6 to 64.0; these results show multilingual speech coverage can transfer from ASR to a frozen VLM without backbone-specific audio alignment.Source: §4.2 · p. 5, Table 3 · p. 5

4.3. Does TFO Preserve the VLM Backbone?

• TFO improves image averages in all five settings and video averages in four, while broadly retaining coding, medical QA, and visual grounding; NVILA’s coding/math average declines by 2.8 points.VideoMME improves across every model family by +0.4 to +10.5 points; coding averages rise in three of four settings, medical averages rise in all five, and PixMo-Count improves in all five.Source: §4.3 · p. 6, Table 4 · p. 6

4.4. Practical Trade-offs and Limitations

• On AVMeme, TFO latency rises from approximately 0.7–2.4 s for native Omni models to 1.3–3.1 s, while transcripts can be reused across questions sharing an input.Transcript routing cannot verify sounds absent from speech text, though TFO performs better than native omni models on audio-driven video hallucination across all four model families. MELLOW and SenseVoice yield isolated gains but neither consistently improves over Whisper alone.Source: §4.4 · pp. 6–8, Table 8 · p. 8

4.5. Ablation Study

• In the Qwen2.5-VL-3B ablation, adding Whisper improves all six audio-visual benchmarks and surpasses Qwen2.5-Omni on five; removing timestamps lowers AVUT-Gemini and Daily-Omni by 2.3 and 1.4 points.The ablation attributes video reasoning to the frozen VLM, speech understanding to Whisper, and targeted temporal alignment benefits to segment-level timestamps.Source: §4.5 · p. 8, Table 9 · p. 8

• Across 56 benchmarks and 21 languages, TFO performs similarly to or better than native omni models when spoken content and temporal speech-video evidence are central.The paper reports strong results on audio-visual, audio-only, and multilingual speech tasks.Source: §5 · p. 8
• Freezing the backbone preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical knowledge than corresponding native omni checkpoints in most comparisons.Source: §5 · p. 8
• TFO routes confidence-filtered, timestamped speech transcripts through a frozen VLM’s language interface without modifying its architecture or visual pathways.This removes backbone-specific audio training and re-alignment, enabling speech-centric omni capability to transfer to stronger VLMs.Source: §5 · p. 8
• TFO’s remaining gap is non-speech acoustic understanding, including music, environmental sounds, vocal tone, and emotion, where native acoustic training remains important.The paper proposes combining external ASR-based speech routing with careful acoustic training while preserving the underlying VLM’s mature capabilities.Source: §5 · p. 8

A. Benchmark Suite

• The core suite covers 56 datasets: 10 AV omni, 14 image/video, 9 audio-only, 7 coding/math, 12 medical QA, and 4 grounding.Datasets are counted by underlying benchmark identity; CoVoST2 counts once across audio-only and multilingual evaluations, while LiveSports3K is excluded from the core total because it appears only in a supplementary ablation. Including LiveSports3K, the paper reports results on 57 distinct datasets overall.Source: A. Benchmark Suite · p. 13

B. Evaluation Protocol

• Multiple-choice tasks use exact-match accuracy, long-form generation uses GPT-5.6 as an evaluator, translation assesses lexical accuracy, meaning, and quality, and open-ended answers require semantic consistency.The GPT-5.6 evaluator receives both the reference response and model prediction; open-ended answers may differ in wording.Source: B. Evaluation Protocol · p. 13
• Each model is evaluated once per benchmark using greedy decoding at temperature zero for consistent repeated outputs, subject to software and hardware determinism.Source: B. Evaluation Protocol · p. 13
• Inference runs on AMD Instinct MI210 GPUs with 64 GB per GPU, using OmniEvalKit for dataset loading, prompts, inference, output parsing, and metric computation.The toolkit is planned for open-source release to support reproducibility and future evaluation.Source: B. Evaluation Protocol · p. 13

C.1. Prompt Construction.

• For speech-content reasoning, the routed audio context contains retained spoken segments; for temporal audio-video reasoning, it also preserves their timestamps as anchors to visual events.The transcript is confidence-filtered before being converted into language-side audio context.Source: §C.1 · p. 13
• The prompt combines dataset-specific instructions, backbone-processed visual input, routed speech context, and the user query, including answer choices and output constraints when applicable.Source: §C.1 · p. 13
• Speech conditions the VLM only through its language interface, while its original visual pathway remains unchanged.Source: §C.1 · p. 13
• If the confidence-filtered transcript is empty, TFO omits the audio context and uses the original image/video-text prompt.Source: §C.1 · p. 13
• When separate audio files and embedded video audio are both available, TFO transcribes them independently and includes explicitly labeled blocks for each source.Labels include “Audio file transcript:” and “Video audio transcript:”.Source: §C.1 · p. 13
• When only one audio source is available, its transcript appears under a single “Audio Transcript:” field, preserving its origin in the language context.Source: §C.1 · p. 13

C.2. Audio Preprocessing.

• TFO transcribes mono 16-kHz audio with Whisper-large-v3turbo at temperature zero, identifies language first, and discards segments below confidence 0.65.Unless specified in ablations, SenseVoice and MELLOW are disabled.Source: §C.2 · p. 13
• Qwen-based implementations process long audio in 30-second chunks with 3-second overlap, and pass only visual input and the task prompt when reliable speech is absent.The generalized prompt labels separate and embedded video audio independently when both are available; a single source uses one Audio Transcript block containing only available confidence-filtered transcripts.Source: §C.2 · p. 13, Figure 2 · p. 14
• The core evaluation suite contains 56 distinct benchmark datasets: 10 AV omni, 14 image/video, 9 audio-only, 7 coding/math, 12 medical QA, and 4 grounding.Dataset subsets, splits, and alternative evaluation variants count once; 21 CoVoST2 language splits are not counted separately, and supplementary-ablation-only LiveSports3K is excluded.Source: Table 10 · p. 14

Overall ablation findings

• Timestamped Whisper is the strongest overall configuration; auxiliary audio-model descriptions offer no consistent gains on audio-visual or audio-only tasks.The VLM remains unchanged, and the authors conclude that direct textual concatenation can add noisy or irrelevant context without reliably recovering non-speech information.Source: Table 11 · p. 15, Table 12 · p. 15, D. Additional Ablations · pp. 15–16

Audio-visual ablation

• Whisper improves all six audio-visual benchmarks over the original VLM and surpasses Qwen2.5-Omni on five, while AVMeme and MMAR-Bench remain weaker than the omni model.This supports the benefit of speech transcription alongside the preserved visual backbone, while exposing remaining gaps for emotion, vocal attributes, music, environmental sounds, and other non-speech cues.Source: D. Additional Ablations · p. 15
• Removing timestamps causes the largest reported drops on AVUT-Gemini (-2.3), AVMeme-Full (-1.7), and Daily-Omni (-1.4), supporting their role in temporal grounding.Auxiliary models sometimes improve individual benchmarks, such as SenseVoice on WorldSense (+5.3), but also cause regressions and do not consistently outperform timestamped Whisper.Source: D. Additional Ablations · p. 15, Table 11 · p. 15

Audio-only ablation

• Whisper alone outperforms Qwen2.5-Omni on eight of ten audio-only benchmarks, including Audio Web (+5.3), CoVoST2 (+5.4), LiveSports3K (+5.8), and Voice-CMMLU (+3.3).It still trails by 5.7 points on MMAR-Bench, indicating transcription remains insufficient for mixed and non-speech acoustic reasoning.Source: D. Additional Ablations · p. 15, Table 12 · p. 15

Language-confidence threshold ablation

• At τ = 0.60 and τ = 0.65 results are similar; raising the threshold to τ = 0.75 lowers UnoBench-MC from 35.1 to 34.3 and AV-Odyssey from 26.8 to 26.2.The default τ = 0.65 more reliably suppresses spurious outputs from music or non-speech regions, while stricter filtering may reject informative speech.Source: D. Additional Ablations · p. 16

E. Benchmark-Specific Evaluation Metrics and Aggregation

• Closed-form audio-visual questions use exact option accuracy, while the nine-benchmark macro-average excludes diagnostic subsets and weights each underlying benchmark equally.UnoBench combines multiple-choice and open-ended correctness, with UnoBench-MC reported separately; AVMeme-Main is a diagnostic subset of AVMeme-Full. Human and AVUT-Gemini count as distinct benchmarks because their samples and annotation settings differ.Source: E. Benchmark-Specific Evaluation… · p. 16
• Audio-only evaluation uses GPT-judged semantic correctness for spoken QA, LLM-judged transcription accuracy for LibriSpeech/FLEURS, weighted F1 for MELD, and exact match for MMAR-Bench, Voice-CMMLU, and VoiceBench.Audio Trivia and Audio Web Questions use the released spoken audio alone; LibriSpeech and FLEURS count transcripts correct when they preserve at least 80% of reference spoken content.Source: E. Benchmark-Specific Evaluation… · p. 16
• CoVoST2 BLEU-1 averages improve by +3.4, +7.4, +12.7, +6.5, and +2.5 points across five settings; MiniCPM4.5 rises 33.6→46.3.Thresholded translation accuracy and BLEU-1 both support the overall finding; thresholded accuracy uses exact normalized match or ROUGE-L F1 of at least 0.40, with an unweighted 21-language macro-average. BLEU-1 is computed without that threshold.Source: E. Benchmark-Specific Evaluation… · pp. 16–17, Table 13 · p. 17
• Visual benchmarks use semantic judging or exact accuracy; code and some math tasks use binary judge correctness, while other math tasks use exact option-match accuracy.Coding scores measure semantic and algorithmic solution correctness, not official execution-based pass@1; medical benchmarks use MedEvalKit's benchmark-specific correctness rules and report accuracy percentages.Source: E. Benchmark-Specific Evaluation… · pp. 17–18

Temporal grounding

• Matching the queried utterance to Whisper’s timestamped transcript interval lets the VLM correctly select the answer covering the beginning of the audio.The example shows how explicit transcript timestamps support temporal grounding.Source: F. Qualitative Results · p. 18

Multilingual understanding

• With Chinese speech and an English question, the VLM combines Whisper’s preserved Chinese text with visual context to identify the humor’s contrast between polite philosophy and violent actions.The example demonstrates cross-lingual audio-visual reasoning without native audio-language training.Source: F. Qualitative Results · p. 18