PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering

Yiduo Jia, Muzhi Zhu, Hao Zhong, Mingyu Liu, Yuling Xi, Hao Chen, Bin Qin, Yongjie Yang, Zhenbo Luo, Chunhua Shen

arXiv:2604.08209v1cs.CV

TL;DR

OmniJigsaw addresses the difficulty of scaling reinforcement-learning post-training for omni-modal reasoning without massive annotated data. It uses temporal reordering of shuffled audio-visual clips with three modality-orchestration strategies and a two-stage filtering pipeline, achieving gains across 15 benchmarks while CMM mitigates JMI’s bi-modal shortcut phenomenon. Its current evidence is limited by evaluation on a single base model and offline data curation.

  • Problem

    Massive high-quality annotated omni-modal data and effective supervisory signals are costly, leaving simultaneous video-audio reasoning insufficiently explored.

  • Method

    OmniJigsaw trains omni-modal models to restore shuffled audio-visual clips chronologically using JMI, SMS, CMM, and two-stage data filtering.

  • Results

    Across 15 benchmarks, OmniJigsaw improves video, audio, and collaborative omni-modal reasoning; CMM gains +4.38 on MLVU-Test over the Qwen3-Omni-30B-A3B-Instruct baseline.

  • Takeaways & Limitations

    Fine-grained clip-level modality masking mitigates the bi-modal shortcut phenomenon and supports scalable self-supervised omni-modal learning on massive unannotated data.

  • Takeaways & Limitations

    The study uses a relatively conservative training setup on a single base model, leaving scalability and transferability across models, data scales, and architectures unexplored.

Abstract

from arXiv · show

To extend the reinforcement learning post-training paradigm to omni-modal models for concurrently bolstering video-audio understanding and collaborative reasoning, we propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task. Centered on the chronological reconstruction of shuffled audio-visual clips, this paradigm strategically orchestrates visual and auditory signals to compel cross-modal integration through three distinct strategies: Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking. Recognizing that the efficacy of such proxy tasks is fundamentally tied to puzzle quality, we design a two-stage coarse-to-fine data filtering pipeline, which facilitates the efficient adaptation of OmniJigsaw to massive unannotated omni-modal data. Our analysis reveals a ``bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue while outperforming sample-level modality selection. Extensive evaluations on 15 benchmarks show substantial gains in video, audio, and collaborative reasoning, validating OmniJigsaw as a scalable paradigm for self-supervised omni-modal learning.

1 Introduction

OmniJigsaw extends reinforcement-learning post-training to omni-modal reasoning through a temporal reordering proxy task and coordinated audio-visual strategies. It addresses puzzle quality and modality shortcuts while improving video, audio, and collaborative reasoning.

  • Motivation: The framework addresses the shortage of massive, high-quality annotated omni-modal data by using a self-supervised proxy task over unannotated data.The paper frames this as a response to costly data collection and weak supervisory signals.
  • Framework: OmniJigsaw extends jigsaw-based reinforcement-learning post-training to omni-modal reasoning through chronological reordering of shuffled audio-visual clips.The framework is self-supervised and annotation-free.
  • Modality orchestration: Joint Modality Integration exposes both streams, while Sample-level Modality Selection and Clip-level Modality Masking orchestrate modality access at different granularities.These strategies are designed to manage modality interference and encourage cross-modal reasoning.
  • Data filtering: The two-stage coarse-to-fine filtering pipeline combines signal-based filtering with semantic Chain-of-Thought screening to improve puzzle quality and scalability.It targets temporal irreversibility, clear state transitions, omni-modal integrity, and dynamism.
  • Results: +4.38 on MLVU-Test, +2.50 on MMAR, and +1.70 on OmniVideoBench are absolute gains from CMM over the Qwen3-Omni-30B-A3B-Instruct baseline.The evaluations report improvements across uni-modal video reasoning, audio comprehension, and collaborative omni-modal reasoning.
  • Analysis: The study identifies a bi-modal shortcut phenomenon in JMI and reports that finer-grained modality masking mitigates it while outperforming sample-level selection.The analyses also examine sensitivity to data quality, reward mechanisms, and orchestration granularity.

2 Related Work

Prior work has strengthened reasoning through reinforcement learning in textual and visual domains, but simultaneous video-audio reasoning in omni-modal models remains insufficiently explored. Self-supervised jigsaw tasks provide an annotation-independent proxy-task direction for extracting supervision from data structure.

  • Reinforcement learning: Reinforcement-learning post-training has advanced reasoning in textual and visual domains, while simultaneous video and audio reasoning remains insufficiently explored.The related work positions OmniJigsaw at the intersection of these capabilities.
  • Self-supervised proxy tasks: Self-supervised learning seeks proxy tasks that derive supervisory signals from innate data topology without manual annotation.Jigsaw-style tasks are characterized as concise, computationally efficient, and independent of auxiliary generative models.

3 Method

OmniJigsaw formulates temporal reordering as permutation prediction over synchronized video-audio clips, then controls modality access through JMI, SMS, and CMM. Data filtering and composite rewards support solvable puzzles and precise, structurally valid reorderings.

  • OmniJigsaw Formulation: OmniJigsaw segments synchronized video-audio input into trimmed clips, shuffles them with a random permutation, and trains the model to restore chronological order.The model predicts an index sequence aligned with the ground-truth permutation.
  • OmniJigsaw Formulation: The orchestration function Φ governs modality accessibility and masking for each clip, while the model outputs chain-of-thought reasoning followed by predicted indices.This provides the common interface for the three modality strategies.
  • Joint Modality Integration: JMI retains synchronized visual and acoustic information for every shuffled clip and applies temporal visual downsampling while preserving audio synchronization.The strategy is intended to support joint analysis of scene evolution and auditory cues.
  • Sample-level Modality Selection: SMS globally selects either video or audio as the primary temporal carrier, then performs reordering using only that modality to reduce interference from less informative signals.The selection is based on a dominance analysis of the complete audio stream and downsampled visual stream.
  • Clip-level Modality Masking: CMM evaluates each clip’s semantic density and selectively retains video, audio, or both, replacing unselected modalities with null tensors.This clip-specific masking imposes an information bottleneck and dynamically shifts attention between visual and acoustic clues.
  • Data filtering: The filtering pipeline first removes samples lacking modal integrity or dynamism, then uses semantic screening to identify irreversible temporal flows and clear state transitions.Signal heuristics include visual motion and audio-quality checks; semantic screening uses lightweight MLLM Chain-of-Thought reasoning.
  • Reward design: The composite reward combines repetition penalties, format rewards, positional accuracy, continuity accuracy, and an accuracy-dependent discount factor.The design rewards exact clip placement, preserved adjacent pairs, structured outputs, and non-repetitive reasoning.

4 Experiments

OmniJigsaw is evaluated across video, audio, and omni-modal reasoning benchmarks, with ablations examining data quality, reward design, and modality orchestration. Results show broad gains, while analyses identify a bi-modal shortcut in JMI and favor clip-level masking over sample-level selection.

  • Ablations and Analysis: Random unfiltered training data performs worse than OmniJigsaw-8K by -3.99 on MLVU-Test, -2.10 on MMAR, and -2.12 on IntentBench.The analysis attributes this gap to visually redundant or otherwise ill-posed reordering samples that lack identifiable state evolution.
  • Ablations and Analysis: Including a discount factor supports continued optimization, whereas fixing λ at 1 causes premature convergence and performance degradation of -2.09 on TUNA-Bench and -1.70 on MMAR.The proposed reward design amplifies differences between sub-optimal and optimal sequences, encouraging later exploration.
  • Modality Orchestration: JMI can underperform uni-modal Jigsaw because redundant audio-visual streams let the model rely on a dominant modality instead of analyzing both.The resulting bi-modal shortcut reduces the need to learn from the weaker modality.
  • Modality Orchestration: CMM mitigates the bi-modal shortcut by dynamically masking less salient modalities at the clip level, enforcing cross-modal switching and integration.Its information bottleneck prevents task completion through single-modality reliance.
  • Modality Orchestration: CMM consistently outperforms SMS across fine-grained video, audio, and omni-modal sub-capabilities because local orchestration preserves temporally changing high-value cues.SMS instead arbitrates globally at the sample level and can omit locally informative modal cues.

5 Conclusion

OmniJigsaw is presented as a scalable, annotation-free self-supervised framework for improving video, audio, and collaborative omni-modal reasoning. Its clip-level masking strategy mitigates the bi-modal shortcut phenomenon, while two-stage filtering supports adaptation to massive unannotated data.

  • OmniJigsaw orchestrates audio-visual signals through three strategies within a temporal reordering proxy task.
  • Evaluations across 15 benchmarks show substantial improvements in video, audio, and omni-modal collaborative reasoning.
  • Clip-level Modality Masking mitigates the bi-modal shortcut phenomenon caused by redundant modality participation.
  • A two-stage coarse-to-fine data filtering pipeline enables efficient adaptation to massive unannotated omni-modal data.

A Appendix

The appendix documents the paper’s uni-modal formulations, implementation and evaluation settings, additional results, limitations, future work, and prompts. It organizes these materials across Sections A.1–A.5.

  • A.1 Uni-Modal Jigsaw Formulation: Section A.1 defines the VideoJigsaw and AudioJigsaw uni-modal references.
  • A.2 Additional Implementation Details: Section A.2 covers data filtering, training, and evaluation implementation details.
  • A.3 More Results: Section A.3 provides semantic-filtering cases, fine-grained sub-capability evaluations, and qualitative baseline comparisons.
  • A.4 Limitations and Future Work: Section A.4 discusses OmniJigsaw’s current limitations and possible future directions.
  • A.5 Prompts: Section A.5 presents semantic-screening, training, and evaluation prompts.

A.1 Uni-Modal Jigsaw Formulation

The uni-modal references assess how much visual or acoustic information contributes to temporal reordering. VideoJigsaw uses only visual clips, AudioJigsaw only audio clips, and both recover the chronological sequence from shuffled clips.

  • VideoJigsaw and AudioJigsaw are established as uni-modal references for measuring modality-specific enhancements.
  • An omni-modal sample X = (V, A) is segmented into N synchronized clips, each containing visual and acoustic information.
  • The uni-modal objective is to recover the ground-truth chronological sequence after clips are randomly permuted.
  • VideoJigsaw: VideoJigsaw reconstructs temporal order from visual clips while explicitly excluding acoustic signals.
  • AudioJigsaw: AudioJigsaw reorders shuffled audio clips without visual assistance.

A.2.1 Details on Data Filtering Pipeline

The data-filtering pipeline first removes ill-posed videos using signal-level integrity, duration, visual-dynamics, and audio-quality checks, then applies MLLM semantic screening for causal temporal progression. Surviving candidates are standardized for training.

  • Signal-based Heuristic Filtering: The heuristic stage prunes samples lacking modal integrity or irreversible transitions to ensure temporal-puzzle solvability.
  • Signal-based Heuristic Filtering: Videos exceeding Dmax = 200 seconds or lacking a valid visual or audio stream are discarded, with the duration limit adjustable to computational resources.
  • Signal-based Heuristic Filtering: Videos with a static-frame ratio above 70% are pruned after frames are sampled every 1.0 second and compared using MAD with threshold τv = 5.0.
  • Signal-based Heuristic Filtering: Audio filtering removes videos with silence ratios above 70% and requires Spectral Flux variance of at least 0.5; speech must occupy 30%–80% of video length.
  • Output Standardization: All surviving candidates are standardized as MP4 files with H.264 video and AAC audio encodings.
  • MLLM-based Semantic Screening: Qwen2.5-VL-7B-Instruct performs semantic screening focused on high-level causal progression and logically deterministic state transitions.

A.2.2 Details on Training

The training pipeline constructs OmniJigsaw-8K from filtered video sources, partitions videos into temporal clips, orchestrates modality access, and applies GRPO post-training. Strategy-specific masking and selection regulate which visual and acoustic information reaches the jigsaw rollout.

  • OmniJigsaw-8K aggregates YouCook2, FineVideo, and LLaVA-Video-178K data, then removes ill-posed instances through two-stage filtering.
  • Each video is uniformly divided into N = 6 non-overlapping clips, with trimming applied to prevent solutions based on low-level boundary continuity.
  • JMI retains synchronized visual and acoustic tensors for every clip, exposing the model to both modalities throughout the rollout.
  • SMS globally identifies the dominant modality and discards the unselected stream, whereas CMM masks less salient modalities independently at clip level.
  • GRPO post-training uses Qwen3-Omni-30B-A3B-Instruct as the initial policy with efficiency-oriented frozen components and configured KL regularization.

A.2.3 Details on Evaluation

Evaluation covers 15 benchmarks spanning video, audio, and omni-modal reasoning, using standardized preprocessing and deterministic decoding. Inputs are processed differently for audio-enabled, audio-free, and audio-only settings while preserving reproducibility.

  • OmniJigsaw is evaluated on 15 diverse benchmarks, primarily reporting top-1 accuracy for multiple-choice question answering.
  • Evaluation input protocols vary by reasoning mode, with video sampled to a maximum of 200 frames and audio capped at 600 seconds when paired with video.
  • Audio-only reasoning permits up to 3,600 seconds of waveform input, while missing audio is padded with zero arrays.
  • All benchmark evaluations use greedy decoding, a 1.05 repetition penalty, and a maximum of 2,048 newly generated tokens.

A.3.1 Cases of Semantic-based Data Filtering

Semantic screening rejects videos whose temporal variation does not provide a directionally recoverable chronology, including indistinct state changes and visually diverse but disconnected narratives. Qualitative comparisons further examine downstream grounding and coherence after CMM post-training.

  • Indistinct State Changes: Semantic screening rejects dynamic videos without distinct visual states, environmental transitions, or reliable temporal markers for chronological reordering.
  • Disjointed Narrative: A video with noticeable scene changes is rejected when its scenes lack causal progression and can be rearranged without changing the meaning.
  • Fine-grained Evaluation: Sub-capability results show relatively consistent gains on several Video-MME and MMAU-test-mini capabilities, while OmniVideoBench gains are more heterogeneous.
  • Qualitative Comparisons: CMM post-training produces more faithful video-grounded responses, reducing unsupported speculation and improving long-horizon semantic coherence in captioning and summarization.
  • Screening Criteria: The screening criteria require causal progression, visual state changes, logical narrative flow, and temporal markers, while excluding repetitive, static, unrelated, or ambiguous content.
Loading 2604.08209v1…