PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-CoE: Reinforcing Video Event Prediction via Chain of Events

Qile Su, Jing Tang, Rui Chen, Lei Sun, Xiangxiang Chu

arXiv:2603.14935v1cs.CV

TL;DR

Video event prediction remains underexplored because MLLMs struggle to model fine-grained temporal structure and logically infer unseen future events from video evidence. The paper evaluates these failures, introduces Chain of Events with two-stage training, and reports state-of-the-art performance across VEP benchmarks. Its main limitation is the continuing challenge posed by inadequate logical reasoning and visual-information utilization in MLLMs.

  • Problem

    Video event prediction is underexplored, requiring fine-grained temporal modeling and logical reasoning about future events that current MLLMs lack.

  • Method

    Chain of Events constructs temporal event chains and uses CoE-SFT followed by CoE-GRPO to connect video evidence with future-event reasoning.

  • Results

    The method achieves state-of-the-art performance across established VEP benchmarks and outperforms baseline MLLMs.

  • Takeaways & Limitations

    Constructing temporal event chains provides an effective paradigm for improving logical reasoning and visual grounding in video event prediction.

  • Takeaways & Limitations

    VEP remains constrained by MLLMs’ lack of logical reasoning for future events and insufficient utilization of visual information.

Abstract

from arXiv · show

Despite advances in the application of MLLMs for various video tasks, video event prediction (VEP) remains relatively underexplored. VEP requires the model to perform fine-grained temporal modeling of videos and establish logical relationships between videos and future events, which current MLLMs still struggle with. In this work, we first present a comprehensive evaluation of current leading MLLMs on the VEP task, revealing the reasons behind their inaccurate predictions, including lack of logical reasoning ability for future events prediction and insufficient utilization of visual information. To address these challenges, we propose \textbf{C}hain \textbf{o}f \textbf{E}vents (\textbf{CoE}) paradigm, which constructs temporal event chains to implicitly enforce MLLM focusing on the visual content and the logical connections between videos and future events, incentivizing model's reasoning capability with multiple training protocols. Experimental results on public benchmarks demonstrate that our method outperforms both leading open-source and commercial MLLMs, establishing a new state-of-the-art on the VEP task. Codes and models will be released soon.

1. Introduction

Video event prediction remains underexplored despite MLLMs’ progress on other video tasks, because it requires fine-grained temporal modeling and logical inference about unseen future events. The paper identifies weak video grounding and logical reasoning, then proposes Chain of Events with two-stage training to address them.

  • VEP requires predicting unseen future events from observed videos, a capability largely underexplored in current MLLM research.
  • Current MLLMs perform worse on VEP than standard vision tasks, partly because insufficient pretraining leaves them without needed inductive biases and reasoning skills.
  • Models often rely on textual answer options instead of linking observed video evidence to plausible future events, weakening logical reasoning.
  • Attention analyses show substantially lower attention to visual tokens than textual tokens, while fine-grained temporal modeling is essential for forecasting.
  • Chain of Events constructs fine-grained temporal event chains and jointly reasons over video and chains to connect observed events with future events.
  • CoE-SFT establishes logical connections during supervised training, while CoE-GRPO strengthens temporal localization and video understanding with modest training costs.
  • Experiments report state-of-the-art performance across VEP benchmarks against open-source and commercial MLLMs.

2. Related Works

VEP differs from conventional video reasoning because it requires forecasting unseen future content from visible evidence. Related MLLM work has advanced visual reasoning and reinforcement learning, but targeted VEP methods remain limited.

  • VEP predicts the next possible event from an input video rather than reasoning only about currently visible content.
  • Accurate forecasting depends on fine-grained temporal modeling of historical events and stronger video understanding and logical reasoning.
  • Recent MLLMs such as Qwen2.5-VL, GLM-4.1V, and Kimi-VL demonstrate visual reasoning capability, while other work applies GRPO-based reinforcement learning.
  • VEP remains largely underexplored, with few targeted methods specifically designed to improve event prediction capabilities.

3. Evaluation and Analysis of MLLMs on VEP

The evaluation finds that current MLLMs have substantial room for improvement on VEP. Their reasoning commonly selects answer options from coarse video summaries, while attention patterns show inadequate use of visual information.

  • Qwen3-VL performs best across most metrics among evaluated MLLMs, yet its average accuracy is only 66.9%.
  • Existing MLLMs often summarize videos coarsely, analyze answer options, and select the most relevant option without logically connecting video content to future events.
  • Coarse summaries can overlook details relevant to future events and neglect temporal dynamics underlying event evolution.
  • The analysis concludes that VEP improvement requires addressing both weak future-event reasoning and insufficient visual-information utilization.

4. Method

CoE models video event prediction by constructing fine-grained temporal event chains and incorporating them into logical reasoning. Its two-stage training combines supervised reasoning demonstrations with reinforcement learning that rewards event-chain structure, temporal alignment, and prediction accuracy.

  • 4.1. Chain of Events (CoE) Paradigm: CoE represents historical video content as a temporally ordered chain of fine-grained events before predicting the future event.Each event contains timestamps and a textual description, and the model reasons jointly over the video, question, and event chain.
  • 4.1. Chain of Events (CoE) Paradigm: The paradigm addresses VEP limitations through logical connections between observed video and future events plus fine-grained temporal modeling.These mechanisms target weak visual grounding and inadequate reasoning in vanilla MLLMs.
  • 4.2. CoE-SFT: CoE-SFT uses Qwen2.5-VL-72B to generate reasoning traces connecting videos, questions, and correct future events without analyzing alternative options.The resulting small-scale dataset undergoes manual quality checking with a pass rate above 90%.
  • 4.4. Evaluation: The method uses an open-set judge evaluation in which a judge assesses reasoning validity and answer correctness, with final win rate as the metric.This evaluation is intended to assess logical reasoning ability and real-world applicability.
  • 4.3. CoE with Group Relative Policy Optimization: The CoE reward combines event-chain structure and length control, video-text temporal alignment, and accuracy supervision.The alignment reward crops timestamp-defined clips and compares their visual features with event-description features; the final reward is a weighted sum of components.
  • 4.3. CoE with Group Relative Policy Optimization: CoE-GRPO trains the model to construct event chains through an improved GRPO framework designed for temporal localization and visual-information utilization.Event tags mark temporal boundaries and descriptions, while chronological chains provide visual grounding for subsequent reasoning without additional cold-start annotations.

5. Experiments

Experiments evaluate CoE on FutureBench and AVEP, showing improved benchmark performance, visual-token attention, and reasoning quality. Ablations examine attention strategies, rollout group size, event-chain length, similarity rewards, and training dynamics.

  • Benchmarks and Metrics: FutureBench measures overall event prediction accuracy across 1-HOP, 2-HOP, 3-HOP, and interpolation types, while AVEP evaluates verbs and event participants.FutureBench reports AVG as the overall average; AVEP reports participant precision, recall, and F1-Score alongside verb accuracy.
  • Main Results: CoE-GRPO consistently surpasses baseline MLLMs on both benchmarks, while CoE-SFT outperforms vanilla SFT and CoE-GRPO outperforms vanilla GRPO.The results are attributed to logical connections between video and future events and to modeling historical event chains.
  • Main Results: CoE substantially increases attention to visual tokens, whereas vanilla SFT reduces such attention.Attention variation is reported using Winning Rate and Improvement Rate relative to the base model.
  • Ablation Study: Prompt-guided and Constant-Bias strategies both degrade performance despite testing multiple prompt formulations and bias magnitudes.Prompt-guided methods ask the model to produce a detailed video description, while Constant-Bias adds a fixed value to visual-token attention weights at inference.
  • Ablation Study: Performance improves as CoE-GRPO group size increases, but excessive rollouts raise training costs, motivating a recommended group size of 4.The group size G controls the number of rollouts during CoE-GRPO.
  • Ablation Study: Event-chain length has a non-monotonic effect: short chains miss visual detail, whereas long chains add redundancy and complicate contextual reasoning.Similarity-model variants perform comparably, VideoCLIP-XL performs best, and removing the similarity reward decreases all metrics.
  • Training Dynamics: During CoE-GRPO training, the accuracy reward generally rises while the CoE reward first increases rapidly and then steadily approaches the target event-chain length.The curves indicate improving event prediction and progressive learning of event-chain construction.

6. Conclusion

The work evaluates MLLMs on video event prediction, identifies key causes of inaccurate predictions, and introduces CoE to improve temporal and logical reasoning. CoE achieves state-of-the-art performance across existing benchmarks.

  • The study presents the first evaluation of various MLLMs on video event prediction and establishes comprehensive baselines.
  • The analysis attributes inaccurate VEP predictions to weak logical reasoning about future events and insufficient use of visual information.
  • CoE enables MLLMs to construct temporal event chains and logically reason over observed videos to predict future events.
  • CoE achieves state-of-the-art performance across existing video event prediction benchmarks.
  • The authors position this work as a foundation for future research on MLLMs for video event prediction.

A. Additional Results

Additional results compare CoE with reinforcement-learning and traditional methods, analyze visual-token attention, and describe similarity-reward computation for event chains.

  • The experiments include comparisons with RL-based methods and traditional state-of-the-art approaches.
  • Table 6 reports additional results, with an asterisk denoting traditional methods.
  • Attention weights are computed for visual and text tokens by averaging attention scores across model layers and heads during test-set inference.
  • For an event chain of length n, a similarity model embeds video and text features for the chain’s clips and descriptions.
  • The similarity reward averages the similarities between video and text features of the events, using cosine similarity as the typical computation.
  • Similarity rewards can use video-text alignment on cropped event segments sampled at 8 frames or image-text alignment over extracted frames.

D. Video Event Prediction Benchmarks

The paper describes benchmarks that evaluate video event prediction across different temporal reasoning patterns and fine-grained event arguments.

  • Futurebench evaluates MLLM video event prediction using SFT and GRPO training datasets collected across varied viewpoints, durations, and video types.
  • Futurebench includes 1-Hop, 2-Hop, and 3-Hop tasks requiring increasingly long sequences of consecutive future events.
  • Its Internp. task requires predicting multiple non-consecutive future events from partially observed scenes with intermediate anchors.
  • AVEP decomposes video events into arguments and evaluates future-event prediction at the argument level.
  • The supplied figure passage identifies Figure 6 as the data scaling curve of CoE.

E. Training Data Scaling of CoE

Training data is important for CoE: benchmark-provided training data improves VEP performance, while scaling experiments show that performance rises as the model learns the CoE reasoning pattern.

  • Training with benchmark-provided data produces strong improvements on VEP, highlighting CoE’s data efficiency.
  • Applying CoE without CoE-specific training causes a performance drop because the model cannot effectively use visual information to construct logical links to future events.
  • With a small amount of training data, performance improves steadily as the model gradually acquires the CoE reasoning pattern.

F. Details of Judge Model Evaluation

The judge-model evaluation tests open-set future-event prediction without answer options and finds CoE outputs more visually grounded and logically consistent. Attention analyses and examples further show improved visual utilization, temporal modeling, and rare timestamp failures.

  • Judge Model Evaluation: The judge-model evaluation removes answer options and assesses whether predictions are logically grounded in the video.The judge model receives the video, question, and reference answer, then evaluates the output’s logic and visual grounding.
  • Judge Model Evaluation: CoE-SFT produces reasoning that is more visually grounded, clear, and concise than competing methods in the illustrated evaluation.Other methods either fail to focus on visual content or produce logically inconsistent predictions.
  • Qualitative Examples: CoE examples use adaptive event-segment granularity while keeping generated event-chain length relatively stable across video durations.The event descriptions are generally consistent with their corresponding video content.
  • Qualitative Examples: CoE-GRPO retains the logical reasoning capabilities learned during CoE-SFT and supports connections between video details and future events.The reasoning process demonstrates fine-grained temporal modeling and logical inference from video content.
  • Failure Cases: Timestamp-generation failures occur in approximately three of one thousand samples and have minimal impact on event descriptions and prediction results.These bad cases concern incorrect timestamp generation according to the instructions.
  • Attention Analysis: CoE-SFT and CoE-GRPO increase attention to visual tokens, whereas vanilla GRPO, vanilla SFT, and untrained chain-of-thought prompting do not adequately resolve visual underuse.The latter approaches either limit improvement efficiency or fail to significantly increase attention to visual content.

J. Limitations and Future Works

The paper identifies temporal localization, richer historical-event structures, and broader evaluation as directions for future work. Supplementary materials also document examples, prompts, and model-comparison visualizations used throughout the study.

  • Limitations: Temporal localization accuracy remains a limitation because the CoE paradigm imposes requirements on the model’s temporal localization capability.The authors expect improving this capability could produce additional performance gains.
  • Future Directions: The study explores only relatively basic historical-event chains, leaving relation-aware event chains and event graphs as promising alternatives.These structures are proposed as more complex ways to model historical events.
  • Future Directions: Resource constraints prevented experiments across all available MLLMs and tasks, although the authors believe CoE should apply broadly because it imposes no base-model-specific requirements.The broader applicability claim is presented as an expectation rather than an experimentally exhaustive finding.
  • Prompt Templates: The prompt format requires chronological seen events, enclosed reasoning, and a final answer containing the option letter.Separate templates support training, inference, CoE-SFT data generation, and judge-model evaluation.
Loading 2603.14935v1…