PaperlayerView exact revision on arXiv

Source-linked AI summary

VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

Qiutong Chen, Yuchan Guo, Zhenlong Yuan, Haobo Yang, Fangfang Lin, Xinyi Long, Yin Wang, Zijian Song, Rui Lan, Shi Qiu, Boyuan Pan, Yang Luo, Yuyin Zhou

arXiv:2610.06293v1cs.CV

TL;DR

Video event prediction requires inferring future events from incomplete visual evidence, but MLLMs can lack causal-transition reasoning and rely on language priors. VepAgent combines structured causal reasoning, diagnostic tools, and reinforcement learning; it reports higher average accuracy than prior best MLLM baselines on both benchmarks.

  • Problem

    Video Event Prediction requires inferring plausible future events from partially observed visual evidence, while MLLMs can lack causal-transition reasoning and rely on language priors.

    Source: §1 · p. 1

  • Method

    VepAgent combines futurebench-4K supervised reasoning data, diagnostic visual tools, and reinforcement learning with accuracy, causal-gap, and anti-prior rewards.

    Source: §1 · pp. 2–3, §3.4 · p. 6

  • Results

    VepAgent (4B) improves average accuracy over prior strongest reported MLLM baselines on both benchmarks, including a 14.58-point gain on FutureBench (66.86% to 81.44%).

    Source: §5 · p. 9

  • Takeaways & Limitations

    The framework recasts VEP as active causal analysis, visual verification, and temporal grounding rather than passive future guessing.

    Source: §1 · p. 2

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in video understanding, yet their reliance on retrospective summarization and text-centric priors often limits their ability to bridge unobserved causal transitions when applied to Video Event Prediction (VEP). To address this, we propose VepAgent, an agentic framework that integrates causal-transition reasoning with tool-augmented reinforcement learning (RL) for robust VEP. Unlike prior methods that passively project future trajectories from historical dependencies, our approach explicitly models the logical progression from terminal observed states to future events. Specifically, we first construct futurebench-4K, a high-quality chain-of-thought dataset for supervised fine-tuning (SFT) that effectively bridges the causal-logic gap by structuring the deduction of unobserved intermediate states. Subsequently, we develop a diagnostic tool library integrating state tracking, frame retrieval, and region magnification, enabling the agent to dynamically augment reasoning with external tools to recover missing spatio-temporal evidence and resolve visual ambiguities during inference. Moreover, we propose a composite reward mechanism that jointly optimizes prediction accuracy, causal coherence, and reliable prior, compelling the agent to rely on genuine visual grounding rather than superficial textual similarities. Extensive evaluations on FutureBench and NEPBench datasets demonstrate that our method achieves state-of-the-art performance, significantly outperforming larger MLLMs and validating the empirical effectiveness of our agentic, future-oriented reasoning paradigm.

1 INTRODUCTION

Video Event Prediction requires reasoning from incomplete visual evidence to unobserved futures, a challenge that retrospective summaries and language priors do not resolve. VepAgent combines causal-transition reasoning, supervised data, diagnostic tools, and reinforcement learning to ground future-event predictions.

1 INTRODUCTION

  • VEP requires bridging incomplete visual evidence to plausible futures, unlike tasks focused on summarizing observed content.The paper describes this forward-looking capability as foundational for proactive robotics, autonomous driving, and intelligent surveillance.Source: §1 · p. 1
  • MLLMs face causal-logic gaps, language-prior shortcuts, and omitted visual evidence when predicting future events.The cited challenges include unobserved intermediate transitions, textually similar answer choices, and missing cues from occlusion, motion, or frame sampling.Source: §1 · p. 2
  • VepAgent combines causal-transition reasoning, anti-prior reinforcement, and tool-augmented perception to ground predictions in video states.The tools track state transitions, retrieve frames, magnify local regions, and extract fine-grained visual facts; rewards jointly target accuracy, causal bridging, and language-shortcut avoidance.Source: §1 · pp. 2–3
  • VepAgent unifies causal-transition reasoning, dynamic visual evidence acquisition, and tool-augmented reinforcement learning for video event prediction.The framework uses futurebench-4K to structure reasoning from observed states and ongoing actions toward future events.Source: §1 · pp. 2–3
  • VepAgent (4B) improves average accuracy over prior best MLLM baselines on FutureBench and NEPBench, with a 14.58-point gain on FutureBench (66.86% to 81.44%).The reported average-accuracy gains are 14.58 and 22.70 percentage points, respectively.Source: §1 · p. 3

2 RELATED WORK

Prior VEP research spans script-based event induction, action anticipation, structured event benchmarks, and MLLM future-event reasoning. VepAgent builds on work in multimodal reasoning and tool use by targeting causal transitions and evidence recovery for VEP.

2 RELATED WORK

  • Earlier VEP methods used script induction, action anticipation, structured event datasets, graph attention, and benchmarks for future-event reasoning.The passage traces work from VidSitu and EPIC-Kitchens to VidEvent, EventFormer, VLEP, and FutureBench.Source: §2 · p. 3
  • RL-based post-training has been applied to MLLM reasoning across tasks, while language-prior shortcuts remain a challenge in some methods.The cited applications include mathematical VQA, image segmentation, and video understanding.Source: §2 · p. 3
  • Prior tool-augmented systems introduced visual evidence, tool-use training, and RL-based acquisition; VepAgent adapts these directions to VEP-specific causal transitions.Its pipeline constructs causal-transition CoT data, initializes behaviors with SFT, optimizes with GRPO, and uses diagnostic tools for prediction.Source: §2 · p. 3, Figure 2 · p. 4

3 METHODOLOGY

VepAgent frames video event prediction as bridging an unobserved transition from the terminal observed state and ongoing action to a plausible future event. Its methodology combines structured training data, a two-stage tool-augmented workflow, supervised fine-tuning, and reinforcement learning with a composite reward.

3.1 PROBLEM FORMULATION & WORKFLOW DESIGN

  • The agent infers the terminal state and unfinished action, selects diagnostic tools for ambiguous transitions, then uses their evidence to predict a plausible future event.The tools track state changes, retrieve dense frames, magnify regions, and extract visible details; they operate only on the observed video.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 TRAINING DATA CONSTRUCTION

  • futurebench-4K structures causal-transition reasoning chains, generated through multiple rollouts, expert validation, and filtering for factual or logical errors and language-prior shortcuts.The resulting dataset contains approximately 4,000 chains; it and the RL trajectories use only VEP training splits.Source: §3.2 · p. 5

3.3 AGENTIC SUPERVISED FINE-TUNING

  • SFT precedes RL after direct RL trials showed declining tool invocation, and jointly optimizes the structured reasoning tokens and final answer.Each training instance includes input modality, task instructions, structured reasoning, and target output.Source: §3.3 · pp. 5–6

3.4 AGENTIC REINFORCEMENT LEARNING

  • GRPO refines the policy through groupwise rollout comparisons, using normalized trajectory rewards to calculate the advantage estimator.The optimization objective includes a KL term balancing exploration and stability.Source: §3.4 · p. 6
  • The format-gated composite reward combines binary prediction accuracy, causal-gap consistency, and an anti-prior term that favors correct options less guessable from text.The causal reward checks the predicted terminal state and transition against reference reasoning, with greater emphasis on the transition and credit conditioned on prediction correctness.Source: §3.4 · pp. 6–7

4 EXPERIMENT

Experiments compare VepAgent with reported baselines on FutureBench and NEPBench, then examine training stages, diagnostic tools, reward components, and GRPO generation counts.

4.1 COMPARISON WITH REPORTED BASELINES

  • VepAgent (4B) raises FutureBench AVG accuracy 14.58 percentage points over Qwen3-VL-30B-A3B, from 66.86% to 81.44%.The comparison uses the same 32-frame budget.Source: §4.1 · p. 8
  • VepAgent raises NEPBench Overall accuracy 22.70 percentage points over Qwen2.5-VL-72B, from 47.50% to 70.20%.It also leads on the reported Charades, VidSitu, and YouCook2 subsets.Source: §4.1 · p. 8
  • In the cooking case, VepAgent reasons from the unfinished process to predict a future event, while the baseline repeats an already observed event.StateTransitionTracker reconstructs the progression from meat addition through mixing to active cooking.Source: §4.1 · p. 8, Figure 4 · p. 8

4.2 ABLATION STUDIES

  • AVG accuracy rises from 59.09% zero-shot to 71.59% with SFT, 75.09% with SFT+RL, and 78.79% for the full model.The authors attribute the progression to SFT providing a structured reasoning foundation before RL optimization.Source: §4.2 · p. 9
  • Removing STT, FRE, RFM, or VDE lowers AVG accuracy from 78.79% to 75.66%, 76.99%, 78.31%, or 77.46%, respectively.The corresponding absolute drops are 3.13, 1.80, 0.48, and 1.33 percentage points; removing STT causes the largest drop.Source: §4.2 · p. 9
  • The full reward reaches 78.79% AVG accuracy, versus 74.62% with accuracy-only reward; causal-gap and anti-prior additions reach 76.42% and 76.33%.The causal-gap and anti-prior additions are reported separately; the full reward is highest among these settings.Source: §4.2 · p. 9
  • Using eight generations yields 81.44% AVG accuracy, up from 78.60% with two, while two generations produce the highest 1-Hop score at 84.39%.Eight generations also yield the highest 3-Hop score, 80.10%, and are adopted for the final model.Source: §4.2 · p. 9

5 CONCLUSION

The conclusion presents VepAgent as a causal-transition and tool-augmented VEP framework, reports gains on FutureBench and NEPBench, and limits evaluation to those benchmarks.

5 CONCLUSION

  • VepAgent combines causal-transition reasoning, futurebench-4K CoT data for SFT, diagnostic tools, and a composite reward for accuracy, causal coherence, and anti-prior robustness.The tools integrate state tracking, frame retrieval, and region magnification.Source: §5 · p. 9
  • On FutureBench, VepAgent (4B) raises average accuracy 14.58 percentage points over prior strongest reported MLLM baselines, from 66.86% to 81.44%.Source: §5 · p. 9
  • On NEPBench, VepAgent (4B) raises average accuracy 22.70 percentage points over prior strongest reported MLLM baselines, from 47.50% to 70.20%.Source: §5 · p. 9
  • Evaluation covers FutureBench and NEPBench; broader video-understanding settings remain untested.The authors identify expanding the tools and applying causal-transition reasoning to those settings as future work.Source: §5 · p. 9
  • Generative AI assisted with CoT data synthesis, external textual descriptions during inference, code modification, language editing, and manuscript assessment and revision.The authors state that AI was not involved in core research ideas, methodology, or experimental design.Source: AI USE STATEMENT · p. 10

A.1 MORE EXPERIMENTAL DETAILS

Appendix A.1 specifies the evaluation datasets and accuracy metrics, training-data split safeguards, and implementation settings for SFT and RL.

A.1 MORE EXPERIMENTAL DETAILS

  • Experiments use FutureBench and NEPBench; all SFT and RL trajectories come exclusively from training splits to avoid data leakage.NEPBench includes Charades, VidSitu, and YouCook2; FutureBench covers 1-Hop, 2-Hop, 3-Hop, and Interpolation reasoning depths.Source: §A.1 · p. 17
  • The reported metric is prediction accuracy: FutureBench includes 1-Hop, 2-Hop, 3-Hop, Interp., and AVG, while NEPBench reports overall and subset accuracy.Source: §A.1 · p. 17
  • SFT uses approximately 4K CoT trajectories for one epoch; RL uses approximately 5K validated trajectories for 250 GRPO steps, with G = 8.SFT uses batch size 16 and learning rate 5 × 10−6; RL uses learning rate 1 × 10−6, and experiments run on eight RTX A5000 GPUs.Source: §A.1 · pp. 17–18

A.2 REWARD IMPLEMENTATION DETAILS

Appendix A.2 details how the causal-gap and anti-prior rewards are computed and how reward ablations handle removed terms.

A.2 REWARD IMPLEMENTATION DETAILS

  • The causal-gap score compares generated terminal-state and causal-transition descriptions with their corresponding references using MiniLM cosine similarity.The terminal-state description is s1; the causal-transition description concatenates reasoning steps 3 and 4 as s34.Source: §A.2 · p. 18
  • When one reasoning block is missing, its available block's score is weight-renormalized; when both are missing, the full thinking content is used.Source: §A.2 · p. 18
  • The causal reward is conditioned on prediction correctness to limit excessive process reward for incorrect predictions.Source: §A.2 · p. 18
  • The anti-prior reward ranks candidate options by query-text similarity and uses the correct option's rank to compute the reward.Query and options are encoded with a MiniLM sentence encoder and compared by cosine similarity.Source: §A.2 · p. 18
  • Reward ablations retain the format reward as a hard structural gate and do not renormalize the remaining term weights.Source: §A.2 · p. 18

A.3 MORE ABLATION STUDIES

Dynamic tool selection achieves the strongest average result while using fewer calls than random selection; temporal and spatial tools together outperform either dimension alone.

A.3 MORE ABLATION STUDIES

  • 78.79% AVG and 83.82% 1-Hop: Dynamic Selection leads the tool-selection strategies while averaging 1.13 calls.Random Selection reaches 76.99% AVG with 1.85 calls; static All Tools reaches 74.91% AVG versus 75.09% for No Tools.Source: Table 4 · p. 18, §A.3 · p. 18
  • 83.82% / 76.12% / 78.79% on 1-Hop / 3-Hop / AVG: combining temporal and spatial tools yields the best overall results.Temporal-only tools reach 77.84% AVG, while spatial-only tools reach 75.38% AVG and 71.64% 3-Hop accuracy.Source: §A.3 · pp. 18–19, Table 5 · p. 19
  • Dense late-window retrieval reveals that fork-pressing has ended while cooking continues, supporting prediction of the subsequent cooking transition.The baseline instead repeats the earlier fork-pressing action.Source: Figure 5 · p. 19

A.4 MORE CASE STUDIES

The reviewed literature spans VEP, multimodal reasoning, tool use, and adjacent areas that offer perspectives on incomplete observations and evidence recovery. The case studies illustrate how VepAgent uses diagnostic evidence to distinguish completed actions from ongoing processes.

A.4 MORE CASE STUDIES

  • Diagnostic evidence helps VepAgent avoid repeating observed events in two cases: dense temporal retrieval reveals ongoing cooking, while fine-grained facts verify that onion addition is complete.The examples address missing temporal evidence and ambiguity about an action’s completion state.Source: §A.4 · pp. 19–20, Figure 6 · p. 20

B.1 VIDEO EVENT PREDICTION AND TEMPORAL EVENT REASONING

  • Prior VEP work covers event prediction, action anticipation, structured event benchmarks, and temporal reasoning, while VepAgent targets the transition from terminal states and unfinished actions to future events.The related-work review names VLEP, VidEvent, EventFormer, FutureBench, and Video-CoE among these directions.Source: §B.1 · pp. 20–21

B.2 MULTIMODAL LLM REASONING AND REINFORCEMENT LEARNING

  • Multimodal reinforcement-learning research includes reasoning-oriented post-training and video-specific applications; VepAgent pairs reinforcement learning with causal-gap and anti-prior objectives.The review notes that stronger reasoning need not entail stronger visual-evidence use.Source: §B.2 · p. 21

B.3 TOOL-AUGMENTED AGENTIC SYSTEMS

  • Tool-augmented systems acquire visual evidence through interleaved operations, dynamic processing, or reinforcement learning, whereas VepAgent’s diagnostic tools target VEP-specific ambiguities.The reviewed examples include TACO, PyVision, and VITAL.Source: §B.3 · p. 22

B.4 VISION-LANGUAGE-ACTION MODELS AND EMBODIED REASONING

  • Embodied VLA research predicts actions and supports decision-making, while VepAgent predicts future semantic events by bridging causal transitions and selectively acquiring evidence.The review includes work on causal modeling and reasoning in autonomous driving.Source: §B.4 · pp. 22–23

B.5 SPATIO-TEMPORAL PERCEPTION AND GEOMETRIC EVIDENCE

  • Geometric-perception work addresses incomplete or ambiguous visual evidence, and VepAgent instead retrieves temporal evidence and magnifies relevant regions without reconstructing full 3D geometry.Source: §B.5 · p. 23

B.6 HUMAN-CENTRIC INTERACTION AND ACTION UNDERSTANDING

  • Human-centric interaction research grounds people, objects, and spatial relations; VepAgent focuses on reasoning from potentially incomplete interactions toward subsequent events.Source: §B.6 · p. 23

B.7 CROSS-MODAL RETRIEVAL, REPRESENTATION LEARNING, AND MULTIMODAL FUSION

  • Cross-modal retrieval and representation research addresses visual-textual alignment and heterogeneous evidence, while VepAgent uses alignment to support causal reasoning toward future events.Source: §B.7 · p. 24

B.9 SUMMARY

The literature spans directly related VEP, reinforcement-learning, and tool-agent work, as well as adjacent areas that inform reasoning under incomplete observations. VepAgent centers on connecting terminal video states to future events through causal reasoning, diagnostic tools, and anti-prior reinforcement.

B.9 SUMMARY

  • Prior work motivates structured temporal reasoning, visually grounded policy optimization, and active evidence acquisition, but does not explicitly integrate them around the transition from terminal video states to future events.Source: §B.9 · p. 24
  • Adjacent research offers perspectives on state verification, spatial reasoning, evidence recovery, multimodal alignment, and robust inference under incomplete observations.Source: §B.9 · p. 25
  • VepAgent combines causal-transition reasoning, diagnostic tool augmentation, and anti-prior reinforcement to verify unfinished actions, recover evidence when needed, and reason toward subsequent events.Source: §B.9 · p. 25

C TOOL LIBRARY DETAILS

The diagnostic library addresses temporal and spatial ambiguities through four tools, which the agent selects to augment causal-transition reasoning. The prompt pipeline uses retrieved evidence to identify the terminal state and unfinished action before predicting a future event.

C TOOL LIBRARY DETAILS

  • The library groups four diagnostic tools into chronological state tracking, temporal retrieval, spatial magnification, and fine-grained fact extraction.Tools are selected once and run only on observed video; STT and VDE use a hosted VLM, while FRE and RFM use visual-processing operators.Source: C TOOL LIBRARY DETAILS · p. 25

C.1 STATE TRANSITION TRACKER

  • The State Transition Tracker structures the observed event chain chronologically to resolve action-order or state-change ambiguity and identify the final observed state.Source: §C.1 · p. 25

C.2 FOCUSED EVIDENCE RETRIEVER

  • The Focused Evidence Retriever densely samples a specified temporal window when sparse frames may have missed a transient action, returning a contact sheet of augmented evidence.Source: §C.2 · p. 25

C.3 REGION FOCUS MAGNIFIER

  • The Region Focus Magnifier crops and enlarges key interaction regions when the global view cannot verify subtle changes such as object release.Source: §C.3 · p. 26

C.4 VISUAL DETAIL EXTRACTOR

  • The Visual Detail Extractor turns visible fine-grained facts from final observed frames into text without predicting the future, grounding causal hypotheses in visual evidence.Source: §C.4 · p. 26

D PROMPTS

  • The prompts use two rounds: first estimate the terminal state and ongoing action and select tools if needed, then integrate observations and tool outputs to predict the future event.The second round structures reasoning around state, unfinished action, plausible evolution, and prediction.Source: D PROMPTS · p. 26
Loading 2610.06293v1…