PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

Che Liu, Lichao Ma, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Xuerui Yang, Fei Tian

arXiv:2605.12034v2cs.MMcs.AIcs.CV

TL;DR

Omni-modal benchmark scores may reflect visual shortcuts rather than audio-visual-language integration. The paper introduces OmniClean through visual-only probing and evaluates OmniBoost, finding RLVR produces the clearest macro gains while self-distillation leads under query-weighted aggregation.

  • Problem

    Visual evidence and a question can suffice to answer audio-visual-language queries, leaving unclear whether benchmark performance reflects genuine omni-modal integration.

    Source: §1 · pp. 1–2

  • Method

    The paper builds OmniClean through query-level visual-only probing across nine benchmarks and evaluates staged OmniBoost, including mixed-modality RLVR and self-distillation.

    Source: §4.1.3 · p. 10, §4.1.4 · p. 12, §5 · p. 15

  • Results

    31.43 macro average for Stage 2: Mixed-Modality RLVR exceeded Stage 3’s 31.03, while Stage 3 led under query-weighted aggregation.

    Source: §4.2 · p. 13, §5 · p. 15

  • Takeaways & Limitations

    Visually answerable queries can make omni-modal benchmarks overstate omni-modal understanding, underscoring the value of visually debiased evaluation.

    Source: §5 · p. 15

    Filtered-score views are not uniformly defined across audited benchmarks, so some comparisons retain full evaluation subsets.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision. Project page: https://cheliu-computation.github.io/omni/

1 Introduction

OmniClean reduces visual shortcuts in omni-modal evaluation, and OmniBoost studies staged post-training on the cleaned view. Its final self-distillation stage brings the 3B model to performance comparable to larger references and slightly above Qwen3-Omni-30B-A3B-Instruct under both aggregate summaries.

Motivation

  • Visual evidence and the question can suffice to answer some audio-visual-language queries, so benchmark gains may reflect visual shortcuts rather than genuine omni-modal integration [5] [6].The paper reports that balanced bi-modal SFT alone yields limited and uneven transfer, while the first broad improvement appears after training with explicit omni-modal data.Source: §1 · pp. 1–2

OmniClean

  • OmniClean audits 16,968 queries across nine benchmarks, removes visually solvable queries where filtering is defined and stable, and retains 8,551 queries.It is an operational evaluation view that reduces visual shortcuts under a fixed protocol, not proof that every retained query is causally audio-dependent.Source: §1 · p. 2

OmniBoost

  • After SFT on self-distilled data, the 3B model becomes comparable to larger open-source references and slightly exceeds Qwen3-Omni-30B-A3B-Instruct under both OmniClean aggregate summaries.The self-distilled queries are constructed without a stronger external omni-modal teacher.Source: §1 · p. 2
  • OmniBoost compares balanced mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data, whose synthetic audio-visual-text queries use entity-based spatial and temporal relations.The entity-based procedure draws on video clips, audio and video captions, and entity scaffolds, then filters model-generated reasoning traces.Source: §1 · p. 2

2 Background and Related Work

Omni-modal models combine audio, visual inputs, and language, but access to multiple modalities does not ensure that evaluations measure their integration. Benchmarks can remain vulnerable to visual shortcuts, while multimodal post-training offers several approaches to improve instruction following and reasoning.

2.1 Omni-modal LLMs

  • Omni-modal models integrate audio, visual inputs, and language in one interface, but visually dominant evidence can let queries be answered without audio and inflate estimates of omni-modal capability.This concern motivates evaluation protocols that distinguish genuine omni-modal use from unimodal competence.Source: §2.1 · pp. 2–3

2.2 Audio-Visual-Language Evaluation

  • Audio-visual-language benchmarks span temporal alignment, social reasoning, counting, complex video reasoning, and open-world QA, yet some queries remain solvable from visual content and the question alone.Verifiable targets such as multiple-choice answers or numeric outputs do not by themselves prevent modality leakage.Source: §2.2 · p. 3

2.3 Post-Training for Multimodal Models

  • Multimodal post-training uses supervised fine-tuning [7] [8] [9], reward-based optimization [10] [11] [12] [32], and distillation [13] [14], while whether vision-language and audio-language competence compose or require explicit omni-modal signals remains unresolved.Recent multimodal RL work [19] [33] [34] [35] [36] suggests targeted optimization can improve reasoning.Source: §2.3 · p. 3

3 Probing Visual Leakage and Constructing a Cleaned Evaluation View

The section audits visual shortcuts in omni-modal benchmarks and constructs OmniClean by removing queries answerable under a visual-only probe where filtering is defined. It also examines how cleaning changes benchmark score relationships with uni-modal capability.

3.1 Visual-Only Probing and a Cleaned Evaluation View

  • The operational pass@16 screen removes a query if any visual-only rollout verifies correct; AV-Odyssey [27] has no defined filtered subset, and CG-AV-Counting [30] remains full for score stability.The screen uses Qwen3-VL-30B-A3B-Thinking [17]; retained queries are not thereby proven impossible to answer from vision under every prompt or decode.Source: §3.1 · pp. 4–6
  • Visual-only solvability varies sharply: Daily-Omni [6] and OmniBench [26] have substantial visually solvable shares, while Video-Holmes [28] retains a larger visually unsolved core.The histogram motivates query-level cleaning rather than relying only on aggregate benchmark scores.Source: §3.1 · p. 4, Figure 1 · p. 5
  • Daily-Omni [6] and OmniBench [26] lose a large fraction of apparent omni performance after filtering, whereas Video-Holmes [28] preserves a larger retained core.The comparison is a leakage diagnostic, not an aggregate model ranking, because filtered-score views are not uniformly defined across the audited suite.Source: §3.1 · pp. 4–5, Figure 2 · p. 5, Table 1 · p. 6
  • 8,551 of 16,968 audited queries are retained in OmniClean, a cleaned evaluation view spanning nine existing omni-modal benchmarks.Filtering operates at the query level; some benchmarks are retained in full under the protocol’s exceptions.Source: §3.1 · p. 7

3.2 Correlation Shifts After Cleaning

  • WorldSense [25] shows the largest post-cleaning drop in vision- and audio-side correlations; Daily-Omni [6], IntentBench, OmniBench [26], and UNO-Bench [29] also become less tied to uni-modal strength.Video-Holmes [28] and OmniVideoBench [31] show smaller or mixed shifts, so cleaning changes benchmark meaning unevenly.Source: §3.2 · p. 7, Figure 3 · p. 8

3.3 How Uni-modal Capabilities Predict Omni Scores

  • Visual strength often predicts original-view omni scores, but this relationship weakens or shifts after filtering for several benchmarks.The regression gallery is reported in Appendix B.Source: §3.3 · pp. 7–8

3.4 Toward a Cleaned Evaluation View

  • The audit recommends reporting visual-shortcut sensitivity and comparing original with cleaned views where defined; OmniClean preserves verifiable answer formats while reducing shortcuts under its probe.Section 4 uses OmniClean to evaluate post-training signals in a less shortcut-sensitive setting.Source: §3.4 · pp. 8–9

4 OmniBoost: A Staged Post-Training Study

OmniBoost tests a single Qwen2.5-Omni-3B [1] lineage through balanced mixed bi-modal SFT, mixed-modality RLVR, and self-distillation SFT on OmniClean. RLVR leads on benchmark-level macro average, while Stage 3 is strongest under query-weighted averaging; filtered synthetic-data SFT also helps in a fixed ablation.

4.1 Staged Post-Training Study Design

  • The study tests whether balanced bi-modal supervision suffices for cleaned omni-modal gains and whether explicit omni-modal data plus later self-distillation can improve capability.Its three completed stages share one initialization lineage: mixed bi-modal SFT, mixed-modality RLVR, then self-distillation SFT.Source: §4.1 · p. 9

4.1.1 Data Construction Across the Staged Study

  • The study constructs three training pools: balanced bi-modal SFT, mixed-modality RLVR, and synthetic audio-visual-text data for self-distillation SFT.The SFT pool allocates 1B output tokens each to audio-text, image-text, video-text, and pure text; the synthetic pool uses seed videos, segment captions, entity-relation records, and filtered traces from the Stage 2 checkpoint.Source: §4.1.1 · p. 9

4.1.2 Stage 1: Mixed Bi-modal SFT

  • The Stage 1 control tests whether large-scale balanced dual-modal supervision transfers to OmniClean without explicit omni-modal instruction data in that stage.It starts from Qwen2.5-Omni-3B [1] and balances audio-text, image-text, video-text, and pure-text sources at 1B output tokens each; video data are deduplicated and rewritten with captions and answer checks.Source: §4.1.2 · pp. 9–10

4.1.3 Stage 2: Mixed-Modality RLVR

  • RLVR is the first stage to deliver substantial benchmark-level macro gains, using a mixed-modality query pool that targets omni-grounded evidence integration while retaining visual and text replay.The pool is 54.8% audio-video-text, 17.4% audio-image-text, 9.0% video-text, 9.4% image-text, and 9.4% text-only; every category includes a text question.Source: §4.1.3 · p. 10, Figure 4 · p. 10

4.1.4 Stage 3: Self-Distillation SFT with Filtered Synthetic Queries

  • Stage 3 builds verifiable audio-visual-text queries from caption and entity records, then distills eight-rollout traces generated by the RLVR checkpoint.Filtering removes queries that are all-wrong or solved in 7/8 or 8/8 rollouts, drops perception-defective or malformed traces, and retains reasoning-answer-consistent outputs after normalization.Source: §4.1.4 · pp. 11–12

4.2 Main Staged Results on OmniClean

  • Macro average rises from 26.49 for Stage 1 to 31.43 for Stage 2, while Stage 3 reaches 31.03; query-weighted averages instead favor Stage 3, 32.15 versus 30.74 for Stage 2.Stage 2 remains stronger on four benchmarks, while Stage 3 improves five; AV-Odyssey contributes 4,555 of 8,551 retained queries, helping explain the query-weighted reversal. Stage 3 uses self-generated traces rather than a stronger external omni-model teacher.Source: Table 2 · p. 13, §4.2 · p. 13, Figure 6 · p. 14

4.3 Data-Centric Self-Distillation Filtering Ablation

  • Under a fixed SFT setup, all three filtered synthetic-data sets improve both aggregate views, with F2 strongest on macro average and F3 slightly strongest on query-weighted average.Video-Holmes, Daily-Omni, and UNO-Bench improve most clearly, while OmniVideoBench declines under all three datasets; this data-centric ablation is distinct from the main Stage 3 trajectory.Source: §4.3 · pp. 14–15, Table 3 · p. 14

5 Conclusion · A Detailed Synthetic Query Graphic Description

Visually answerable queries can overstate omni-modal benchmark performance, while OmniClean and staged OmniBoost provide a visually debiased evaluation view and post-training recipe. RLVR gives the clearest macro improvement, while self-distillation’s benefits depend on aggregation and benchmark profile.

5 Conclusion

  • Mixed-modality RLVR [10] [11] [12] provides the clearest benchmark-level macro improvement, while Stage 3 leads under query-weighted aggregation but does not uniformly dominate RLVR across benchmark families.The conclusion scopes these findings to one Qwen2.5-Omni-3B lineage [1] and the authors’ visual-only leakage protocol.Source: §5 · p. 15
  • Visually answerable queries can make omni-modal benchmarks overstate omni-modal understanding; OmniClean addresses this with query-level visual-only probing across nine existing benchmarks.Source: §5 · p. 15
  • Balanced mixed bi-modal supervised fine-tuning [7] [8] [9] is a useful control but is insufficient for consistent omni-modal gains.Source: §5 · p. 15
  • Fixed ablations show that synthetic queries built from paired audio and video captions can directly improve the 3B base model.Source: §5 · p. 15
  • Self-distillation [13] [14] remains useful but profile-dependent, with Stage 3 leading under query-weighted aggregation rather than uniformly dominating RLVR across benchmark families.Source: §5 · p. 15
  • The findings support controlling visual leakage to interpret omni-modal progress and suggest small models can benefit substantially from staged post-training with explicitly constructed omni-modal supervision.The evidence is limited to one Qwen2.5-Omni-3B lineage [1] and the authors’ visual-only leakage protocol.Source: §5 · p. 15

B Full Section 3 Regression Plots

The section presents benchmark-by-benchmark regression panels pairing vision and audio scores against omni scores, alongside a detailed graphic of the synthetic-query construction process. The final panels cover OmniVideoBench; AV-Odyssey and CG-AV-Counting are omitted because filtered-score views are unavailable under the protocol.

Full Section 3 Regression Plots

  • Figure 7 details synthetic-query construction, from seed-video segmentation and caption records through an entity-relation scaffold to verifiable-answer composition by gpt-oss-120b.The graphic shows captions, the entity graph, and answer-format constraints provided to the model, and expands the process summarized in Figure 5.Source: Figure 7 · p. 19
  • Figures 8–10 show paired vision-score and audio-score views against omni score for Daily-Omni, IntentBench, Video-Holmes, WorldSense, OmniBench, UNO-Bench, and OmniVideoBench.AV-Odyssey and CG-AV-Counting are omitted because the protocol has no reported filtered-score views for them.Source: Figure 8 · p. 20, Figure 9 · p. 21, Figure 10 · p. 21

C Cleaned-View Stage Delta Visualization

On the cleaned evaluation view, benchmark-level score deltas relative to Qwen2.5-Omni-3B show different strength patterns for Stages 2 and 3.

C Cleaned-View Stage Delta Visualization

  • Relative to Qwen2.5-Omni-3B [1], Stage 2 gains are largest on Video-Holmes [28], OmniBench [26], and Daily-Omni [6]; Stage 3 shifts strength toward AV-Odyssey [27], Daily-Omni, IntentBench [3], OmniVideoBench [31], and UNO-Bench [29].Figure 11 presents these benchmark-level deltas as a visual companion to Table 2.Source: Figure 11 · p. 22

D Original-View Results Across the Three OmniBoost Stages

On the original, uncleaned evaluation view, RLVR remains the broad-gain stage, while self-distillation improves selected benchmarks without uniformly surpassing RLVR. These supplementary results remain vulnerable to visual leakage and are not the paper’s primary basis for interpretation.

D Original-View Results Across the Three OmniBoost Stages

  • The original-view results are supplementary because visually answerable queries remain, so visual leakage can make strong scores reflect shortcut-sensitive benchmark structure.The comparison checks whether the stage ordering seen after cleaning also appears before cleaning; it includes reference scores for four open-source omni models.Source: D Original-View Results Across the… · pp. 22–23, Table 4 · p. 23
  • Original-view stage results can look competitive against larger open-source omni models, but self-distillation mainly redistributes strengths across benchmarks rather than becoming a uniformly dominant endpoint.The table includes Qwen2.5-Omni and Qwen3-Omni reference columns.Source: D Original-View Results Across the… · p. 23, Table 4 · p. 23
  • Stage 2: Mixed-Modality RLVR has the strongest macro and original-weighted averages; Stage 1 is clearly weaker, while Stage 3 improves selected datasets but not uniformly.Stage 3 gains are most visible on AV-Odyssey and OmniVideoBench, whereas RLVR remains stronger on most other benchmarks.Source: D Original-View Results Across the… · pp. 22–23, Table 4 · p. 23

E Source Uni-modal Benchmark Pools for the Regression Analysis

This section documents the published uni-modal benchmark scores used to form the average vision and audio performance axes in the Section 3 regression analysis.

E Source Uni-modal Benchmark Pools for the Regression Analysis

  • The reference pools use scores from the Qwen2.5-Omni and Qwen3-Omni reports to define the regression axes; they are not evaluations of OmniBoost variants.Benchmark columns cite the corresponding public benchmark definitions.Source: E Source Uni-modal Benchmark Pools for… · p. 23

E.1 Vision and General Benchmarks

Table 5 lists published benchmark scores used to compute the vision/general-performance axis average for regression analysis, drawing on Qwen2.5-Omni [1] and Qwen3-Omni [2] reports.

E.1 Vision and General Benchmarks

  • Table 5’s published scores are used to compute the average vision/general-performance axis for the regression analysis in Section 3.Source: Table 5 · p. 23
  • The source scores come from the Qwen2.5-Omni [1] and Qwen3-Omni [2] reports.Source: Table 5 · p. 23
  • The benchmark columns cover MMMU [41], MMMU-Pro [42], MathVista [43], MathVision [44], AI2D [45], ChartQA [46], MMStar [47], MM-MT-Bench [48], and Video-MME [49].Source: Table 5 · p. 23

E.2 Audio

The audio regression axis is computed as an average of published Qwen2.5-Omni [1] and Qwen3-Omni [2] scores across seven benchmarks.

E.2 Audio

  • Table 6 lists source scores for SD-QA [50], MMSU [51], OpenBookQA [52], IFEval [53], AdvBench [54], VoiceBench [55], and MMAU [56].These scores are used to compute the average audio-performance axis for the regression analysis.Source: Table 6 · p. 24

Related papers

Loading 2605.12034v2…