PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-Centric Post-Training for Omni-Modal Reasoning

Ziyang Cheng, Yuhao Wang, Hongcheng Liu, Qimin Wu, Jingru Fan, Chen Qian, Yanfeng Wang, Yu Wang

arXiv:2610.02819v1cs.CL

TL;DR

Omni-LLMs still struggle with joint audio-visual reasoning, and native multimodal data construction and training are costly. The paper studies text-only reasoning training followed by reduced-data native audio-visual RL for perception. The best text-only pipeline improves reasoning over the base model with lower compute than the complete native route, while the staged approach restores perception and retains most reasoning gains.

  • Problem

    Omni-LLMs can have strong perceptual capabilities without reliable reasoning across audio-visual evidence over modalities and time, while native multimodal data construction and training costs limit scalability.

    Source: §1 · p. 1

  • Method

    The paper organizes post-training by capability, using text-only SFT and RL for main reasoning optimization followed by reduced-data native audio-visual RL for perceptual refinement.

    Source: §1 · p. 2

  • Results

    25.83%: The best text-only configuration raises Qwen2.5-Omni-7B’s geometric mean of nine reasoning scores over the base model and outperforms the complete native audio-visual route with 56.6% fewer GPU-hours.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Reduced-data audio-visual refinement restores perception above the base level while retaining 93.5% of the best text-only pipeline’s reasoning gain.

    Source: §1 · p. 2, SFT GRPO SFT GRPO · p. 9

Abstract

from arXiv · show

Improving joint audio-visual reasoning in Omni Large Language Models typically incurs substantial data construction and training costs. Our diagnostics reveal multi-hop reasoning difficulties despite correct answers to all corresponding single-hop questions and suggest partial decoupling in the local optimization of perception and reasoning objectives. This motivates post-training with different emphases on these capabilities. Text-only reasoning training yields gains across data sources, model scales, and families. With the best-performing text-only configuration, supervised fine-tuning followed by reinforcement learning (RL) raises Qwen2.5-Omni-7B's geometric mean of nine reasoning scores by 25.83% over the base model, outperforming the complete native audio-visual route with 56.6% fewer GPU-hours. Training on data synthesized entirely by a text-only LLM raises this geometric mean by 21.01% without audio-visual data in construction or training. However, text-only training degrades perception. We therefore propose a text-centric post-training paradigm: text-only training provides the main reasoning optimization, and reduced-data native audio-visual RL then refines perception. Refinement uses about 90% fewer input tokens than full-data audio-visual RL, restores perception above the base level, and retains 93.5% of the best-performing text-only pipeline's reasoning gain.

1 INTRODUCTION

Omni-LLMs can answer individual audio-visual questions yet still struggle to combine evidence across modalities and time, while constructing native multimodal training data is costly. This section motivates separating reasoning and perception optimization, using text-only training for reasoning and reduced-data audio-visual RL for perceptual refinement.

1 INTRODUCTION

  • Native audio-visual reasoning data requires content-consistent annotations and coverage of infrequent long-tail categories, while multimodal training costs limit scalability.Prior text-only transfer to separate visual or audio tasks does not establish effectiveness for joint audio-visual reasoning across modalities and time.Source: §1 · p. 1
  • Across nine Omni-LLMs, multi-hop reasoning remains difficult even when models answer all corresponding single-hop perception questions correctly.The diagnostic case is illustrated in Figure 1(a).Source: §1 · p. 1, Figure 1 · p. 2
  • Projected gradient updates retain most target-loss reduction while substantially reducing changes to the other objective, supporting partial decoupling in local optimization.This motivates post-training strategies with different emphases on perception and reasoning.Source: §1 · p. 2
  • 25.83%: Text-only SFT followed by RL raises Qwen2.5-Omni-7B’s geometric mean of nine reasoning scores over the base model, with 56.6% fewer GPU-hours than the complete native audio-visual route.Gains extend across evaluated model scales and families; fully synthetic text-only training also raises the geometric mean by 21.01%.Source: §1 · p. 2
  • Reduced-data native audio-visual RL after text-only SFT and RL restores perception above the base level while retaining 93.5% of the text-only pipeline’s reasoning gain.Refinement uses about 90% fewer input tokens than full-data audio-visual RL.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses Omni-LLM reasoning with multimodal training data or active inspection, while text-only transfer has been studied on separate visual and audio tasks. This paper studies text-only post-training for joint audio-visual reasoning without native audio-visual training data.

Text-only reasoning transfer

  • Text-only transfer work has addressed visual or audio tasks, whereas this work studies joint audio-visual reasoning across modalities and time.The fully synthetic setting uses textual SFT and RL data generated by a text-only LLM without native audio-visual inputs.Source: §2 · p. 3

Perception and reasoning in multimodal models

  • Prior approaches to perception and reasoning limitations use multimodal training or active inspection; this paper’s diagnostics motivate text-only post-training without native audio-visual training data.The cited prior approaches include cross-modal evaluations and VLM post-training that balances perception and reasoning.Source: §2 · p. 3

3 PRELIMINARY STUDY

The preliminary study tests whether correct single-hop perception predicts multi-hop success and whether perception and reasoning objectives respond differently to local gradient updates.

3 PRELIMINARY STUDY

  • Questions built from shared audio-visual annotations combine facts across two or three steps, using inputs from a common subset answered correctly by all nine models.Each multi-hop question is derived from corresponding single-hop questions.Source: §3 · p. 3

3.1 THE GAP BETWEEN PERCEPTION AND REASONING

  • 94.7% accuracy on the original 1-hop set contrasts with 64.1% on 2-hop and 62.3% on 3-hop questions, despite correct answers to every corresponding single-hop question.The gap appears across all five domains.Source: §3.1 · p. 3

3.2 PARTIAL DECOUPLING IN PERCEPTION AND REASONING OPTIMIZATION

  • The optimization study defines mean cross-entropy losses for single-hop perception and multi-hop reasoning, then compares original updates with updates constrained to leave the other training-fold loss unchanged to first order.It uses five-fold cross-validation by video and computes gradients for the Thinker’s Transformer decoder layers and final normalization layer, with other parameters fixed.Source: §3.2 · pp. 3–4
  • Projected reasoning updates reduce reasoning loss by 6.01% while reducing perception loss by 0.06%; projected perception updates retain 74.3% of original target-loss reduction and reduce reasoning loss by 0.36%.These responses support partial decoupling of the two mean-loss objectives in local optimization.Source: §3.2 · p. 4

4 STUDY DESIGN

The study addresses five questions about text-only reasoning gains, data choices, synthetic data, perceptual effects, and training strategy. Its experiments use multiple model families and benchmark suites to assess reasoning, perception, and training cost.

4 STUDY DESIGN

  • The study asks five research questions spanning reasoning gains and compute, data choice, synthetic training data, perceptual effects, and a text-centric training paradigm.The questions are framed around improving omni-modal reasoning with lower training compute.Source: §4 · p. 4

4.1 TRAINING SETUP

  • Experiments center on Qwen2.5-Omni-7B and apply text-only training to Qwen2.5-Omni-3B and Gemma-4 models, with text-only SFT also evaluated on Qwen3-Omni-30B-A3B-Instruct.Text-only and native audio-visual post-training start from the same base model and follow SFT with GRPO; the default text-only configuration uses Script-based MCQs and CoT supervision for SFT.Source: §4.1 · p. 4

4.2 EVALUATION SETUP

  • Evaluation covers seven omni-modal reasoning benchmarks, five perception benchmarks, and joint reasoning-perception evaluation for the training-paradigm question.The study compares the base model and six representative omni-modal reasoning baselines; it reports official benchmark metrics, Macro Avg., geometric mean improvement, input tokens, and GPU-hours for specified questions.Source: §4.2 · pp. 4–5

5 FINDINGS

Text-only reasoning training produces broad omni-modal gains, including when the training data are synthesized entirely from text. The findings also examine training-data choices, perceptual effects, and a text-centric staged-training paradigm.

5.1 RQ1: REASONING GAINS AND TRAINING COMPUTE

  • Qwen2.5-Omni-7B’s text-only SFT–GRPO raises Macro Avg. from 39.80 to 49.39, a 9.59-point gain and the highest result among compared methods.Its Macro Avg. and geometric-mean improvement of +25.83% both rank first.Source: §5.1 · p. 5, Table 2 · p. 6
  • Text-only SFT–GRPO improves reasoning across model scales and families, including Qwen2.5-Omni-3B, Gemma-4 variants, and Qwen3-Omni-30B-A3B-Instruct.Qwen2.5-Omni-3B’s Macro Avg. rises from 37.23 to 44.93, while text-only SFT raises Qwen3-Omni-30B-A3B-Instruct’s Macro Avg. from 50.49 to 53.07.Source: §5.1 · pp. 5–6, Table 3 · p. 6
  • The complete text-only route reaches a 49.39 Macro Avg. versus 47.37 for the native audio-visual route, using 56.6% fewer GPU-hours.It processes 0.505B versus 1.306B training input tokens.Source: §5.1 · p. 6

5.2 RQ2: CHOICE OF TRAINING DATA

  • Script-based MCQs yield the strongest reported reasoning transfer, with final Macro Avgs. of 49.39 or 48.91 depending on the SFT data.Contextual QA in both training phases yields 42.97; both MCQ configurations improve all nine scores over the base model.Source: §5.2 · p. 6, Table 4 · p. 7

5.3 RQ3: SYNTHETIC TRAINING DATA

  • Training on text-only-LLM-synthesized data improves omni-modal reasoning across four models without native audio-visual data in construction or training.Qwen2.5-Omni-7B gains 7.70 Macro Avg. points, from 39.80 to 47.50.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

5.4 RQ4: PERCEPTUAL EFFECTS

  • The findings include a comparison of perceptual effects from text-only and audio-visual training in Table 5.The supplied passage identifies the table’s scope but does not state its results.Source: Table 5 · p. 8

5.5 RQ5: TRAINING PARADIGM

  • The staged-training study tests reduced-data native audio-visual training after text-only training and reports reasoning, perception, and training budgets.The supplied passages describe the study setup and table scope, not its outcome.Source: §5.5 · p. 8, Table 6 · p. 9

SFT GRPO SFT GRPO

Staged training follows text-only reasoning optimization with reduced-data native audio-visual refinement. The reported configurations compare reasoning and perception outcomes against text-only and full-data audio-visual routes.

SFT GRPO SFT GRPO

  • With the complete text-only SFT–GRPO initialization, audio-visual GRPO reaches a reasoning Macro Avg. of 48.48, versus 47.42 for audio-visual SFT, with nearly identical perception averages.The perception averages are 78.50 and 78.53, respectively.Source: SFT GRPO SFT GRPO · p. 9
  • Compared with the complete text-only route, staged refinement raises perception Macro Avg. by 2.09 points while reducing reasoning by 0.91 points.It retains 93.5% of the text-only route’s geometric-mean reasoning gain: 24.15% versus 25.83%.Source: SFT GRPO SFT GRPO · p. 9
  • The staged configuration exceeds the full-data native audio-visual SFT–GRPO route by 1.11 reasoning Macro Avg. points, with a 0.57-point lower perception average.The refinement uses about 90% fewer training input tokens than full-data native audio-visual GRPO.Source: SFT GRPO SFT GRPO · p. 9

6 CONCLUSION

The conclusion presents text-only training as a route to omni-modal reasoning gains across data sources, model scales, and families, including fully synthetic text data. It describes a text-centric sequence that uses reduced-data native audio-visual refinement after text-only reasoning training.

6 CONCLUSION

  • Text-only reasoning training improves omni-modal reasoning across data sources, model scales, and families, including with fully synthetic text data.The synthetic-data route requires no audio-visual data in construction or training.Source: §6 · p. 9
  • On Qwen2.5-Omni-7B, the best-performing text-only pipeline outperforms the complete native audio-visual route with less compute.Source: §6 · p. 9
  • The conclusion assigns text-only training to reasoning optimization and reduced-data native audio-visual supervision to perceptual refinement.The reduced-data refinement restores perception while retaining most reasoning gains.Source: §6 · p. 9

A.1 DIAGNOSTIC QUESTION CONSTRUCTION AND EVALUATION

The diagnostic study constructs multi-hop questions from facts in corresponding single-hop audio-visual questions, then evaluates models on matched inputs without prior-question hints. Its taxonomy and protocol define the scope of the comparison.

A.1 DIAGNOSTIC QUESTION CONSTRUCTION AND EVALUATION

  • Single-hop questions test direct audio-visual information extraction, while multi-hop questions combine facts across audio and vision in two or three reasoning steps.The questions cover Causal, Character, Modality, Temporal, and Theme domains.Source: §A.1 · p. 16
  • Multi-hop questions are built only from single-hop questions answered correctly by all nine models, with ambiguous, underspecified, unsupported, or single-cue-solvable items removed.The resulting set contains approximately 5,000 questions from 2,703 videos, equally divided between two-hop and three-hop questions.Source: §A.1 · p. 16
  • Each model answers multi-hop and corresponding single-hop questions separately using the same audio-visual material and preprocessing, without prior questions, answers, evidence hints, or conversation history.Table 7 reports accuracy by model and question hop count.Source: §A.1 · p. 16, Table 7 · p. 16

A.2 LOCAL OPTIMIZATION ANALYSIS

The analysis tests whether perception and reasoning objectives can be optimized with limited cross-effects. Projected updates preserve target-loss reduction while constraining changes to the other objective, with held-out results supporting partial local decoupling.

A.2 LOCAL OPTIMIZATION ANALYSIS

  • Five-fold video-level cross-validation constructs update directions and calibrates norms on training folds, then evaluates each update on its held-out fold.The higher norm is calibrated until both original directions satisfy the Armijo condition; the lower norm is half as large and is also checked.Source: §A.2 · pp. 17–18
  • The projected update removes the first-order change to the other training objective while maximizing target-loss decrease per unit update norm.The original and projected update directions are unit norm, so the step size is the parameter-change norm.Source: §A.2 · p. 17
  • Across five splits, projected updates retain a mean 84.23% of first-order target-loss reduction at the same update norm, with mean gradient cosine similarity 0.5391.Training-gradient orthogonality does not impose a zero first-order change on held-out objectives.Source: §A.2 · p. 17
  • Cross-task projections show higher target retention and lower retention on the other objective than projections between the two perception halves.Table 8 reports first-order retention on held-out objectives, averaged over three seeds.Source: §A.2 · p. 17, Table 8 · p. 17
  • At the lower update norm, projected perception updates retain 67.48% of target loss reduction and decrease reasoning loss by 0.006%, while projected reasoning updates retain 85.89% and increase perception loss by 0.040%.This supports local partial decoupling of the two mean-loss objectives.Source: §A.2 · p. 18

B.1 TRAINING SETUP

Training uses text-only and audio-visual data routes with shared Qwen2.5-Omni-7B hyperparameters for comparison, while evaluation follows benchmark protocols and reports defined reasoning and perception aggregates.

B.1 TRAINING SETUP

  • Training applies LoRA to language-model components while freezing audio and vision encoders and multimodal projection modules.SFT and GRPO use AdamW with warmup followed by cosine decay; GRPO combines accuracy and format rewards as R = Racc + 0.1Rformat.Source: §B.1 · p. 19
  • Qwen2.5-Omni-7B uses the same SFT and GRPO hyperparameters for text-only and audio-visual training, with audio and video frames sampled at 1 FPS in the latter.For synthetic RQ3 data, a text-only LLM generates fictional scripts and MCQs without native audio-visual inputs, followed by programmatic, model-based, and subset manual checks.Source: §B.1 · p. 19

B.2 EVALUATION SETUP

  • Evaluation follows each benchmark’s official settings; Macro Avg. is the unweighted mean of nine reasoning scores or five perception scores.AVHBench Overall pools audio-driven and video-driven hallucination accuracies; Video-MME-v2 is reported as Avg Acc.Source: §B.2 · p. 20

C.1 TRAINING DATA MIXTURES

The paper evaluates SFT data mixtures and transfer to agentic tasks, then documents prompt protocols, synthetic-data prompts, and examples from the final refined model.

C.1 TRAINING DATA MIXTURES

  • With subsequent GRPO, SFT on Script-based MCQs alone reaches reasoning Macro Avg. 49.39, above 46.26 for the open-ended mixture and 47.90 for the General reasoning mixture.Although mixtures raise SFT Macro Avg., their final scores are lower, and both mixtures improve all nine evaluation dimensions over the base model.Source: Table 12 · p. 20, §C.1 · p. 20

C.2 AGENTIC REASONING TRANSFER

  • Text-only SFT followed by GRPO raises OmniGAIA Overall Pass@1 from 3.6 to 13.1, nearly matching 13.3 from native omni-modal training.The text-only route scores 15.0 and 6.4 on the Medium and Hard subsets, respectively.Source: Table 13 · p. 21, §C.2 · p. 21

D.1 TRAINING AND EVALUATION

  • The Qwen-Omni and Gemma-4 prompt figures cover training and evaluation templates, with video-only evaluation for NExT-QA and added reasoning instructions for SocialOmni How.Omni-Cloze uses its official prompts.Source: §D.1 · p. 21, Figure 5 · p. 21, Figure 6 · p. 22

D.2 SYNTHETIC DATA GENERATION

  • The synthetic-data prompt figures cover episode planning, visual expansion, MCQ and reasoning generation, and sample-quality assessment.The prompts use brace-enclosed named placeholders for variable content and JSON blocks to illustrate output formats.Source: §D.2 · p. 23, Figure 9 · p. 23, Figure 10 · p. 24

E CASE STUDIES

  • Figures 11–14 show examples from the final Qwen2.5-Omni-7B model after text-only reasoning training followed by reduced-data audio-visual refinement.The examples are drawn from Daily-Omni, Video-Holmes, WorldSense, and MUSIC-AVQA v1.Source: E CASE STUDIES · p. 25, Figure 11 · p. 25, Figure 12 · p. 26
Loading 2610.02819v1…