PaperlayerView exact revision on arXiv

Source-linked AI summary

FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

Qian Chen, Jinlan Fu, Changsong Li, Min Zhang, See-Kiong Ng, Xipeng Qiu

arXiv:2601.13836v2cs.CLcs.CVcs.MM

TL;DR

FutureOmni addresses the limited evaluation of audio-visual future forecasting beyond retrospective multimodal understanding. It introduces a human-in-the-loop benchmark and OFF training strategy, finding that current models struggle while OFF improves forecasting and generalization.

  • Problem

    Existing benchmarks largely emphasize retrospective understanding, leaving audio-visual future forecasting and joint causal reasoning underexplored.

  • Method

    FutureOmni combines 919 videos and 1,034 QA pairs across 8 domains with a human-in-the-loop construction pipeline, and OFF uses 7K instruction-tuning samples.

  • Results

    Current omni-modal and video-only models struggle, with the best performance reaching 64.8% by Gemini 3 Flash, while OFF improves forecasting and generalization.

  • Takeaways & Limitations

    The benchmark exposes substantial room for improvement in omni-modal future forecasting, and OFF provides gains across FutureOmni and out-of-domain benchmarks.

Abstract

from arXiv · show

Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).

1 Introduction

FutureOmni addresses the underexplored problem of forecasting future events from audio-visual context, where existing benchmarks largely emphasize retrospective reasoning. It introduces a broad benchmark and finds that current models remain limited, while OFF improves forecasting and generalization.

  • Benchmark motivation and design: FutureOmni evaluates cross-modal causal forecasting, temporal reasoning, and internal-knowledge use for future events from audio-visual context.The benchmark requires models to select future events using audio and video while resisting visual-only, audio-only, delayed, and reverse-causal distractors.
  • Benchmark motivation and design: 919 videos and 1,034 multiple-choice QA pairs span 8 domains in the FutureOmni benchmark.The dataset uses an AI-assisted, human-in-the-loop construction pipeline and includes diverse audio types and videos up to 20 minutes.
  • Experiments and findings: 64.8% was the best reported performance on omni-modal future forecasting, achieved by Gemini 3 Flash.The evaluation covers both omni-modal and video-centric models, including proprietary and open-source systems.
  • Experiments and findings: OFF substantially improves open-source performance on FutureOmni and enhances out-of-domain generalization across audio-visual and video-only benchmarks.Attention visualizations indicate improved identification of critical keyframes alongside better generalization and reasoning performance.

2 Related Work

Prior multimodal benchmarks expanded audio-visual understanding but mainly evaluate retrospective tasks, while future-prediction benchmarks are predominantly vision-centric. FutureOmni targets the resulting gap by requiring both omni-modal perception and causal future reasoning.

  • Multimodal benchmarks: Omni-modal benchmarks such as WorldSense and DailyOmni incorporate audio but predominantly emphasize retrospective perception, captioning, or question answering.Their task scope differs from future-event forecasting.
  • Future prediction benchmarks: Future-prediction benchmarks such as VLEP and IntentQA typically focus on vision and may mute or disregard audio cues.This limits evaluation of cases where sound acts as the primary precursor to a future event.
  • Research gap: FutureOmni addresses the lack of benchmarks that jointly demand omni-modal perception and causal future reasoning.Its design is positioned as a bridge between audio-visual understanding and future prediction.

3 The FutureOmni Benchmark

FutureOmni constructs an audio-visual future-forecasting benchmark through filtering, multimodal temporal localization, causal-pair discovery, verification, and distractor design. The resulting benchmark compares future-forecasting coverage and reasoning patterns against existing video and audio-visual datasets.

  • 3.1 Audio Coordinated Video Selection: Approximately 18K YouTube videos are collected, then low-quality samples are filtered using visual similarity and audio-visual correlation criteria.Videos with limited scene changes or weakly informative audio are removed before downstream annotation.
  • 3.2 Audio-Visual Temporal Localization and Calibration: Gemini 2.5 Flash grounds events in videos, while boundary checks use MFCC differences exceeding a threshold of 2.0.The pipeline also annotates synchronized dialogue, sound effects, and background music within temporal segments.
  • 3.3 Audio-Visual QA Construction: Causal-pair discovery restricts premise-to-future gaps to 30 seconds and scores audio influence from 0 to 2 across Speech, Sound, and Music.Each pair includes a premise event, target event, and rationale explaining the logical bridge.
  • 3.3 Audio-Visual QA Construction: FutureOmni uses visual-only, audio-only, and other distractors to test whether models integrate both modalities rather than relying on one.Candidate QAs undergo automated logical validation followed by human verification.
  • 3.4 Dataset Statistics: FutureOmni contains 919 videos and 1,034 future-forecasting QAs, with an average duration of 163.5 seconds and patterns spanning thematic montage, causal, and routine sequences.The benchmark covers eight primary domains and is entirely dedicated to future forecasting, unlike comparison datasets focused largely on retrospective perception or captioning.

4 Experiments

FutureOmni evaluates 20 MLLMs across modalities, domains, and durations, revealing persistent weaknesses in audio-visual future forecasting. Performance depends on cross-modal integration, with visual perception, joint reasoning, speech, and short contexts presenting major challenges.

  • Overall Evaluation: 20 MLLMs are evaluated across open-source omni-modal, open-source video, and proprietary categories on FutureOmni.The evaluation uses official implementations and includes both omni-modal and video-centric systems.
  • Overall Evaluation: Approximately 61% average accuracy for proprietary Omni-LLMs exceeds the strongest open-source Omni-LLM’s 53%.The comparison indicates a persistent performance gap between proprietary and open-source omni-modal systems.
  • Overall Evaluation: Video-only models reach at most 49.70% accuracy, underperforming Omni-LLMs because they cannot leverage audio cues.The results highlight the importance of audio-visual integration for future event prediction.
  • Fine-grained Results: Qwen3-Omni scores 62.65% on Game, whereas AVicuna reaches only 20.83% on Documentary, showing substantial domain variation.Game and Dailylife are generally easier, while Documentary and Emergency are more difficult.
  • Fine-grained Results: Models perform worst on the shortest videos, with Qwen3-Omni at 34.90% and Gemini 3 Flash at 40.78%, while performance peaks at [2,4) minutes.The results indicate that short clips often provide insufficient historical context for forecasting.
  • Error Analysis: 51.6% of Gemini 3 Flash’s 318 analyzed failures are video perception errors, while 30.8% are audio-video joint reasoning failures and 2.5% reflect lack of knowledge.The analysis attributes the performance gap primarily to dynamic perception and causal reasoning limitations rather than factual knowledge.

5 Omnimodal Future Forecasting: From Prediction to Generalization

The paper addresses weak omni-modal forecasting with rationale-enhanced instruction tuning through FutureOmni-7K and the OFF strategy. Training improves forecasting, especially for speech, and transfers to omni-modal and video-only benchmarks.

  • Method: FutureOmni-7K incorporates rationales explaining why future events follow from audio-visual premises, while OFF trains models to internalize this forecasting logic.The method targets reasoning over causal and temporal relationships rather than outcome prediction alone.
  • Results: All tuned models improve on FutureOmni, with video-SALMONN 2 gaining +3.87% overall and Qwen2.5-Omni improving by nearly 10% in Speech.The largest category-specific gains occur in the challenging Speech setting.
  • Evaluation: OFF is evaluated on FutureOmni, four omni-modal benchmarks, and two video-only benchmarks to test forecasting and generalization.The evaluation includes WorldSense, DailyOmni, JointAVBench, OmniVideoBench, Video-MME, and MLVU.
  • Mechanism: Keyframe Attention Score Difference measures attention shifts assigned to ground-truth video and audio keyframes across transformer layers.The visualization compares internal attention distributions before and after training.

6 Conclusion

FutureOmni establishes a benchmark for omni-modal future prediction and uses a human-in-the-loop pipeline to study its limitations. The conclusion emphasizes difficult speech-dense scenarios and rationale-enhanced training for broader task performance.

  • Conclusion: FutureOmni is presented as the first comprehensive benchmark dedicated to evaluating omni-modal future prediction in MLLMs.The benchmark is designed around synergistic audio-visual reasoning.
  • Conclusion: The paper concludes that current MLLMs struggle with omni-modal future prediction, particularly in speech-dense scenarios.The conclusion links the benchmark findings to a rationale-enhanced instruction-tuning dataset and OFF strategy.
  • Conclusion: The benchmark’s supporting materials include sections on audio and duration results, transfer, experiment details, attention visualization, error analysis, statistics, and prompting.These sections document the benchmark’s analyses and experimental procedures.

B Why Future Forecasting Transfers

The paper argues that future forecasting transfers because OFF trains models to identify causal triggers and reason over temporal progression. OFF consistently outperforms generic caption-based supervision, especially on anticipation and temporal prediction tasks.

  • Why Transfer Occurs: OFF improves performance across fine-grained temporal reasoning benchmarks, including WorldSense, JointAVBench, and FutureOmni.The proposed explanation is that causal-trigger identification and temporal progression encourage structured timeline understanding.
  • Comparison with Caption SFT: OFF consistently outperforms comparable-scale generic video-caption SFT across Qwen2.5-Omni, video-SALMONN 2, and Ola.The advantage is especially pronounced on tasks requiring anticipation and temporal prediction.
  • Comparison with Caption SFT: The comparison attributes OFF’s gains to the future-forecasting objective rather than merely adding more training data.The paper contrasts forecasting supervision with generic descriptive SFT, which does not effectively cultivate causal and temporal capabilities.

C.1 Training and Inference Details with FutureOmni-7K

FutureOmni spans eight major category groups and 21 fine-grained subcategories, covering diverse video domains and reasoning demands. The taxonomy includes entertainment, education, real-world documentation, and safety-critical analysis.

  • FutureOmni contains eight major category groups and 21 fine-grained subcategories spanning diverse video domains and reasoning demands.
  • Cartoon focuses on animation, while Surveillance covers police footage involving complex and rapidly evolving visual events.
  • Education includes Instrument, Knowledge, Science, and Preschool categories emphasizing instructional clarity, factual reasoning, and multimodal alignment.
  • Emergency covers Rescue and Disaster scenarios requiring accurate temporal reasoning and event understanding.
  • Dailylife includes Commercials, Travel Vlogs, and Lifestyle Vlogs with informal narration and varied filming styles.
  • Movie covers TV scene clips, TV trailers, film analysis, and comedy skits, emphasizing narrative coherence and character interactions.
  • The taxonomy enables comprehensive evaluation across entertainment, education, real-world documentation, and safety-critical analysis.

C.5 UGCVideoCaptioner Experiment

The UGCVideoCaptioner experiment compares captions generated from audio-visual input with captions generated from visual input alone. Caption semantic differences are used to select videos where audio contributes substantially to the description.

  • Two captions are generated for each video: one conditioned on audio and visual inputs, and one conditioned on visual input only.
  • Each caption is encoded with Sentence-BERT using the all-MiniLM-L6-v2 model.
  • Caption similarity is computed as cosine similarity between normalized embeddings in GPU batches.
  • Videos are ranked by caption similarity, and the bottom 50% are retained because their captions differ most.
  • The multimodal caption prompt requests the primary setting, key entities and actions, audio cues, on-screen text, and overall video purpose.

D Prompt

The D Prompt specifies how to identify challenging cross-modality causal relationships and construct future-event questions with temporally valid answers and deceptive distractors. It emphasizes plot relevance, causal restriction, and multimodal necessity.

  • D Prompt: The task requires identifying premise and conclusion events, their modalities, timestamps, and causal direction in a fixed output format.
  • D Prompt: Cross-modality causal relationships connect video and audio events when one modality makes a subsequent event in the other modality predictable.
  • D Prompt: The prompt prioritizes plot-relevant causal pairs that drive narrative development over simple everyday occurrences.
  • D Prompt: Causal examples include a red button press followed by mechanical whirring and a musical crescendo followed by a character jumping from a building.
  • D Prompt: Selected relationships should require both modalities and high-level reasoning rather than simple perception or descriptive association.
  • D Prompt: The constraints require effects to follow causes, prohibit invented timestamps, require causes to start after 01:00, and cap the video duration.
  • D Prompt: Future-event questions use a causal premise, one precise caption-supported answer, and four contextually plausible but logically inferior distractors.
  • D Prompt: A worked example resolves suspenseful misdirection by selecting the functional event: the character pulls jumper cables from the trunk.
Loading 2601.13836v2…