PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

Che Liu, Lichao Ma, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Xuerui Yang, Fei Tian

arXiv:2605.12034v2cs.MMcs.AIcs.CV

TL;DR

Omni-modal benchmarks can reward visual shortcuts rather than genuine audio-visual-language integration. The paper introduces visually debiased evaluation and staged post-training, finding that mixed-modality RLVR provides the clearest broad gains while self-distillation remains useful but profile-dependent.

  • Problem

    Omni-modal benchmark scores may reflect visual shortcut exploitation when visual evidence and the question suffice without audio.

  • Method

    The paper constructs OmniClean through visual-only probing over nine benchmarks and evaluates staged post-training with mixed-modality RLVR and filtered self-distillation SFT.

  • Results

    Mixed-modality RLVR provides the clearest benchmark-level macro improvement, while self-distillation leads under query-weighted aggregation but remains profile-dependent.

  • Takeaways & Limitations

    Visually debiased evaluation makes omni-modal progress easier to interpret, while synthetic paired audio-video supervision can improve a 3B base model without a stronger external omni teacher.

  • Takeaways & Limitations

    Filtered-score views are not uniformly defined across the audited benchmarks, and some visual-only filtered entries are omitted.

Abstract

from arXiv · show

Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision. Project page: https://cheliu-computation.github.io/omni/

1 Introduction

Omni-modal benchmark gains can arise from visual shortcuts rather than genuine audio-visual-language integration. The paper introduces OmniClean and studies whether staged post-training with explicit omni-modal signals improves a Qwen2.5-Omni-3B model.

  • Motivation: Visual evidence and the question can suffice to answer many audio-visual-language tasks, allowing strong scores without using audio.Consequently, raw benchmark gains may reflect visual shortcut exploitation rather than improved omni-modal understanding.
  • OmniClean: OmniClean is a visually debiased evaluation view constructed over nine existing omni benchmarks using visual-only probing.The protocol removes visually solvable queries and retains full subsets when filtering is undefined or would destabilize score comparisons.
  • OmniClean: 16,968 queries are audited and 8,551 queries retained under the OmniClean protocol.OmniClean reduces visual shortcuts under a fixed protocol but does not prove that retained queries are causally audio-dependent.
  • OmniBoost: The paper studies OmniBoost, a staged post-training recipe based on Qwen2.5-Omni-3B, asking whether bi-modal strengthening alone is sufficient for omni-modal understanding.The study compares balanced mixed bi-modal supervised fine-tuning with approaches involving explicit omni-modal data and optimization signals.

2 Background and Related Work

Omni-modal models extend multimodal interfaces to jointly consume text, images, video, and audio, but visually dominant evidence can inflate evaluations without demonstrating true modality integration. Recent benchmarks and post-training methods therefore target audio-visual-language reasoning while probing whether explicit omni-modal signals are necessary beyond composing vision-language and audio-language competence.

  • Omni-modal systems: Omni-modal systems consume text, images, video, and audio within a single model, alongside continued advances in vision-language and audio-language models.Representative systems include Qwen2.5-Omni, Qwen3-Omni, HumanOmniV2, NEXUS-O, and Nemotron 3 Nano Omni.
  • Evaluation challenges: Visually dominant evidence can make queries answerable without audio, causing evaluations to overestimate omni-modal capability.This motivates protocols that distinguish genuine omni-modal use from performance explained by visual shortcuts.
  • Benchmarks: Recent audio-visual-language benchmarks test joint understanding of audio-visual events and language queries grounded in omni-modal evidence.They cover temporal alignment, intent and social reasoning, counting, complex video reasoning, and open-world audio-visual question answering.
  • Post-training: Multimodal post-training uses supervised fine-tuning, reinforcement-learning-style optimization with verifiable or task-aligned rewards, and distillation or self-distillation.For omni-modal models, it remains unresolved whether vision-language and audio-language competence can simply compose or require explicit omni-modal signals.

3 Probing Visual Leakage and Constructing a Cleaned Evaluation View

This section audits omni-modal benchmarks for visual leakage by testing whether queries remain solvable from visual input and text alone, then constructs OmniClean through query-level filtering and benchmark-specific retention exceptions. Across 16,968 audited queries, 8,551 are retained, yielding a visually debiased evaluation view over nine benchmarks.

  • Visual-only audit: The audit marks a query visually answerable when a strong model verifies the correct answer using only its image or video and text question, then excludes it.This criterion reduces visual shortcuts under the protocol but does not prove exclusive audio dependence.
  • Cleaning protocol: Pass@16 cleaning generates 16 visual-only rollouts per query and removes a query if at least one rollout is verified correct.The rule constructs the cleaned split and visual-only histograms, not reported model scores.
  • Benchmark variation: Visual-only solvability varies substantially: Daily-Omni and OmniBench show extensive leakage, while Video-Holmes retains a larger visually unsolved core.This benchmark variation motivates query-level cleaning instead of relying only on aggregate scores.
  • Retention exceptions: AV-Odyssey is omitted from filtered-score summaries because some answer options require audio, while CG-AV-Counting is probed diagnostically but retained fully for score stability.Filtered-score views are therefore not uniformly defined across the audited suite and are used for leakage diagnosis rather than aggregate ranking.
  • OmniClean: 16,968 queries are audited and 8,551 retained, producing OmniClean, a cleaned evaluation view over the same nine audited omni benchmarks.Retention follows cleaning or full-retention rules, with the filtering unit defined as the query rather than the underlying media item.

4 OmniBoost: A Staged Post-Training Study

OmniBoost studies three staged post-training interventions from Qwen2.5-Omni-3B on the cleaned evaluation view: mixed bi-modal SFT, mixed-modality RLVR, and self-distillation SFT. RLVR produces the strongest benchmark-level macro average, while self-distillation remains useful but changes performance unevenly across benchmarks.

  • Study design: OmniBoost starts from Qwen2.5-Omni-3B and compares mixed bi-modal SFT, mixed-modality RLVR, and self-distillation SFT in one initialization lineage.The recipe tests whether balanced bi-modal supervision suffices and whether explicit omni-modal data plus self-distillation improves capability.
  • Mixed Bi-modal SFT: The balanced SFT pool mixes audio-text, image-text, video-text, and pure-text supervision, sampling each source to 1B output tokens.This output-token balancing is intended to isolate modality composition from simple data imbalance.
  • Mixed-Modality RLVR: RLVR targets queries spanning text-only, image-text, video-text, audio-image-text, and audio-video-text settings while retaining visual and textual replay queries.The stage uses DAPO, 16 rollouts per query, 32 queries per update, and a 1200-step run from the mixed-SFT checkpoint.
  • Stage comparison: 31.43 is the benchmark-level macro average for Stage 2: Mixed-Modality RLVR, compared with 26.49 for Stage 1: Mixed Bi-modal SFT and 31.03 for Stage 3: Self-Distillation SFT.The comparison shows that balanced mixed bi-modal SFT alone is insufficient for consistent omni gains on OmniClean, while RLVR delivers the strongest reported macro average.
  • Self-Distillation SFT: Self-distillation uses synthetic audio-visual-text queries generated from the 3B RLVR lineage, with filtered rollouts providing supervision that stabilizes and amplifies existing reasoning patterns.F2-retained data gives the strongest macro average in the fixed ablation, while OmniVideoBench declines under all three fixed ablation datasets.

5 Conclusion

The conclusion argues that visually answerable queries can overstate omni-modal understanding and presents OmniClean, a visually debiased evaluation view built through query-level visual-only probing. It further finds that balanced mixed bi-modal SFT is insufficient for consistent gains, whereas mixed-modality RLVR provides the clearest benchmark-level macro.

  • Visually answerable queries can make omni-modal benchmarks overstate omni-modal understanding.
  • OmniClean is a visually debiased evaluation view created through query-level visual-only probing over nine existing benchmarks.
  • Balanced mixed bi-modal supervised fine-tuning is a useful control but insufficient for consistent omni-modal gains.
  • Mixed-modality RLVR provides the clearest benchmark-level macro.

Author List · A Detailed Synthetic Query Graphic Description

The supplied author information credits Che Liu and collaborators from StepFun and five academic institutions. No passage describing the synthetic query graphic is provided.

  • Author List: Che Liu is listed as the first author and is affiliated with StepFun and Imperial College London.
  • Author List: Lichao Ma is listed as an author affiliated with StepFun and Peking University.
  • Author List: Xiangyu Tony Zhang is listed as an author affiliated with StepFun and The University of New South Wales.
  • Author List: Yuxin Zhang is listed as an author affiliated with StepFun and Shanghai Jiao Tong University.
  • Author List: Haoyang Zhang and Xuerui Yang are listed as authors affiliated with StepFun.
  • Author List: Fei Tian is listed as the corresponding author, marked with an asterisk, and affiliated with StepFun.

B Full Section 3 Regression Plots

This section presents benchmark-by-benchmark regression panels comparing vision and audio scores against omni scores across nine datasets. It also documents the Synthetic Query construction process and notes omitted benchmarks lacking filtered-score views under the protocol.

  • Synthetic Query Construction: Figure 7 details Synthetic Query construction from segmented seed videos, captions, entity-relation scaffolds, and answer-format constraints supplied to gpt-oss-120b [18].The process produces a Synthetic Query with a verifiable answer.
  • Benchmark Regression Panels: Figures 8 and 9 show paired vision-score and audio-score regression views against omni scores for six benchmarks.The panels cover Daily-Omni [6], IntentBench [3], Video-Holmes [28], WorldSense [25], OmniBench [26], and UNO-Bench [29].
  • Benchmark Regression Panels: Figure 10 presents the final regression panels for OmniVideoBench [31].AV-Odyssey [27] and CG-AV-Counting [30] are omitted because filtered-score views were not reported under the protocol.

C Cleaned-View Stage Delta Visualization

The cleaned-view delta visualization shows that post-training gains redistribute across benchmarks: Stage 2 is strongest on Video-Holmes, OmniBench, and Daily-Omni, while Stage 3 favors five other benchmark profiles.

  • Benchmark-level score deltas: Stage 2 produces the largest gains on Video-Holmes, OmniBench, and Daily-Omni relative to Qwen2.5-Omni-3B.Figure 11 presents benchmark-level score deltas on the cleaned evaluation view.
  • Benchmark-level score deltas: Stage 3 shifts strength toward AV-Odyssey, Daily-Omni, IntentBench, OmniVideoBench, and UNO-Bench.The visualization emphasizes redistribution rather than uniform improvement across benchmarks.

D Original-View Results Across the Three OmniBoost Stages

Original-view results preserve the cleaned-view stage ordering, with Mixed-Modality RLVR strongest overall and Self-Distillation SFT improving only selected datasets. Because visual leakage remains, these supplementary scores are not the primary basis for interpretation.

  • Interpretation: Original-view stage results can appear competitive with larger open-source omni models, but visual leakage makes such comparisons shortcut-sensitive.The original view is therefore supplementary rather than the main basis for conclusions.
  • Stage comparison: Stage 2: Mixed-Modality RLVR has the strongest macro and original-weighted averages across the three OmniBoost stages.Stage 1: Mixed Bi-modal SFT is clearly weaker.
  • Stage comparison: Stage 3: Self-Distillation SFT improves a subset of datasets, especially AV-Odyssey and OmniVideoBench, but RLVR remains stronger on most other benchmarks.

E Source Uni-modal Benchmark Pools for the Regression Analysis

This section documents the published uni-modal source scores used to construct the regression-analysis axes, rather than presenting additional evaluations of the staged post-training variants.

  • Source Scores: Published scores from Qwen2.5-Omni and Qwen3-Omni reports form the source pool for the regression analysis.These scores support the Average Vision Performance and Average Audio Performance axes in Section 3.
  • Benchmark References: The benchmark columns reference the corresponding public benchmark definitions.The tables document the uni-modal reference pools behind the regression plots.
  • Scope: These tables are documentation of reference pools, not additional evaluations of the staged post-training variants.

E.1 Vision and General Benchmarks · E.2 Audio

The paper uses published Qwen2.5-Omni and Qwen3-Omni report scores to construct regression-analysis axes for vision/general performance and audio performance. The vision/general axis covers nine benchmarks, while the audio axis covers seven.

  • E.1 Vision and General Benchmarks: Table 5 provides published source scores for the average vision/general-performance axis used in Section 3’s regression analysis.The scores come from the Qwen2.5-Omni and Qwen3-Omni reports.
  • E.1 Vision and General Benchmarks: The vision/general benchmark set includes MMMU, MMMU-Pro, MathVista, MathVision, AI2D, ChartQA, MMStar, MM-MT-Bench, and Video-MME.
  • E.1 Vision and General Benchmarks: Table 5’s scores are sourced from the Qwen2.5-Omni and Qwen3-Omni reports.
  • E.2 Audio: Table 6 provides published source scores for the average audio-performance axis used in Section 3’s regression analysis.The scores come from the Qwen2.5-Omni and Qwen3-Omni reports.
  • E.2 Audio: The audio benchmark set includes SD-QA, MMSU, OpenBookQA, IFEval, AdvBench, VoiceBench, and MMAU.
  • E.2 Audio: Table 6’s scores are sourced from the Qwen2.5-Omni and Qwen3-Omni reports.
Loading 2605.12034v2…