Source-linked AI summary
Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO
Junhao Cheng, Liang Hou, Xin Tao, Jing Liao
TL;DR
Next-event prediction has generally returned text, although video can demonstrate physical-world information that text alone struggles to convey. VANS coordinates a vision-language model and video diffusion model with Joint-GRPO. It is reported to achieve state-of-the-art performance in event prediction accuracy and video generation quality on established benchmarks.
Problem
Next-Event Prediction has typically produced textual answers, while video generation can convey dynamic physical-world information that text alone struggles to express.
Source: §1 · p. 2Method
VANS uses Joint-GRPO to coordinate a vision-language model and video diffusion model under a joint reward, alongside the VANS-Data-100K dataset.
Source: §1 · pp. 2–3Results
Takeaways & Limitations
Video-Next-Event Prediction advances next-event reasoning from textual description to dynamic video demonstration.
Source: §6 · p. 9The VLM and VDM are optimized in isolation: the VLM receives no feedback on whether its descriptions yield visually plausible videos, while the VDM must coordinate caption and input-visual signals.
Source: §4 · p. 4
Abstract
from arXiv · showhide
While language models have become impactful in many real-world applications, video generation remains largely confined to entertainment. Motivated by video's inherent capacity to demonstrate physical-world information that is difficult to convey through language alone (e.g., imagine teaching someone to tie a tie using only text), we identify an underutilized opportunity to extend video as a new answer modality for Next-Event Prediction (NEP), formalized as Video-Next-Event Prediction (VNEP). While the established NEP task takes a video with a procedural or predictive question as input to predict the next event in text, VNEP requires dynamic video responses. This shift from telling to showing unlocks more intuitive and customized answers for procedural learning and creative exploration. However, this task remains challenging for existing models, as it demands an understanding of multimodal input, instruction-conditioned reasoning, and the generation of video with visual and semantic consistency. To address this, we introduce VANS, a model that leverages reinforcement learning to align a Vision-Language Model (VLM) with a Video Diffusion Model (VDM) for VNEP. The core of VANS is our proposed Joint-GRPO that orchestrates the VLM and VDM to function as a unit. Driven by a shared reward on their respective output, it optimizes the VLM to produce captions that are both accurate and friendly to visualize, while guiding the VDM to generate videos that are faithful to these captions and the input visual context. To enable this learning, we craft VANS-Data-100K, a dedicated dataset for the VNEP task. Experiments on procedural and predictive benchmarks demonstrate that VANS achieves state-of-the-art performance in both video event prediction and visualization. Codes are released in https://github.com/KlingTeam/VANS.
1. Introduction
VNEP turns next-event prediction from a textual answer into a dynamic video demonstration. VANS coordinates a VLM and VDM through Joint-GRPO and is reported to perform favorably on prediction and generated-video quality.
1. Introduction
- VNEP changes next-event prediction from a textual answer to a dynamic video response that can demonstrate actions and adapt to the user's current state.The example is a Windsor-knot demonstration tailored to the current tie configuration.Source: §1 · p. 2, Figure 2 · p. 2
- VNEP requires event-conditioned reasoning over a video and question, followed by video generation that is visually coherent and semantically faithful to the inferred event.The paper contrasts this with frame continuation based on spatiotemporal patterns.Source: §1 · p. 2
- Joint-GRPO uses a shared reward to coordinate a VLM and VDM, training captions to be accurate and easy to visualize while guiding video generation to follow the captions and input context.The VANS approach uses a two-stage optimization.Source: §1 · pp. 2–3
- VANS-Data-100K provides 100K video-question-answer triplets for supervised fine-tuning, with 1K manually selected samples for RL post-training.The triplets support the VNEP task.Source: §1 · p. 3
- VANS performs favorably on procedural and predictive VNEP benchmarks for both event prediction accuracy and generated-video quality.The paper identifies this as state-of-the-art performance.Source: §1 · p. 3
2. Related Work
Prior Next-Event Prediction work generally produces textual answers, while this paper introduces video responses. It also distinguishes Joint-GRPO as a way to coordinate reasoning and visualization models together.
Next-Event Prediction
- Existing Next-Event Prediction datasets and methods focus on textual answers, leaving the answer modality exclusively textual.Prior work includes benchmarks such as VLEP, MVP, and V1-33K, alongside methods using event understanding, temporal modeling, commonsense reasoning, fine-tuning, and RL.Source: §2 · p. 3
Group Relative Policy Optimization
- Joint-GRPO differs from prior GRPO applications by coordinating a VLM for reasoning and a VDM for visualization simultaneously.The cited prior uses of GRPO optimize a single model.Source: §2 · p. 3
3. VANS-Data-100K
VANS-Data-100K addresses existing datasets' video-quality and instructional-question gaps with procedural and predictive VNEP samples. Its curation combines source-specific segmentation, clip selection, and QA generation.
Dataset overview
- VANS-Data-100K contains 30K procedural and 70K predictive samples, each pairing an input video and question with text and video answers.Existing Next-Event Prediction datasets are described as having suboptimal video quality and insufficiently diverse instructional questions for direct VNEP use.Source: §3 · pp. 3–4
Raw Data Collection
- The collection draws procedural videos from COIN and YouCook2, and predictive videos from general-scene datasets and shortfilms.The sources are chosen to cover step-by-step tasks and narrative or causal dynamics, respectively.Source: §3 · p. 4
Shot Split
- Curation segments videos into coherent clips, using ground-truth timestamps for procedural data and shot-boundary detection for predictive data, while filtering segments under three seconds.The stated purpose of the length filter is to ensure action completeness.Source: §3 · p. 4
Clip Selection and QA Pair Generation
- Gemini-2.5-Flash selects optimal 3–5 second clips and generates QA pairs that target procedural next steps or predictive what-if scenarios.QA generation also supplies chain-of-thought reasoning and a ground-truth answer, with a self-check for logical soundness and information leakage.Source: §3 · p. 4
4. VANS
VANS uses a VLM to reason from the input video and question, then guides a VDM to generate a visually consistent event video. Joint-GRPO co-steers both models through a structured two-stage optimization process.
4. VANS
- VANS feeds the VLM high-level ViT features and the question, then conditions the VDM on the predicted-event caption and low-level VAE tokens from sampled input frames.The VAE tokens are concatenated into the VDM conditioning latent space to support fine-grained visual correspondence while generating novel scenes.Source: §4 · p. 4
- Optimizing the VLM and VDM in isolation leaves the VLM without feedback on video plausibility and leaves the VDM coordinating caption and visual-context signals.The paper identifies this disconnect as a semantic-to-visual gap that Joint-GRPO is intended to address.Source: §4 · p. 4
4.1. Preliminary of GRPO
- GRPO samples trajectories for an input context, scores them with rewards, and computes normalized advantages relative to the group average before policy optimization.Clipping and a KL-divergence term are described as stabilizing training by preventing drastic policy updates.Source: §4.1 · pp. 4–5
4.2. Joint-GRPO
- Joint-GRPO co-steers the VLM and VDM in two stages, aligning VLM reasoning with visual generation and then adapting the VDM to follow the improved VLM.The VLM is optimized while the VDM is frozen in Stage 1; Stage 2 optimizes the VDM using the VLM as a frozen anchor.Source: §4.2 · p. 5
- The VLM’s joint reward combines instruction-format compliance, caption similarity, and generated-video coherence to encourage captions that are accurate and visually actionable.Caption similarity uses ROUGE-L, while video coherence uses CLIP Similarity.Source: §4.2 · p. 5
- The VDM generates a novel scene guided by the anchor caption while preserving relevant input-video elements, with rewards for semantic alignment and video fidelity.The video-fidelity reward targets visual quality and coherence with the input; the semantic reward measures consistency with the anchor caption using CLIPScore.Source: §4.2 · p. 6
5. Experiments
The experiments evaluate VANS on procedural and predictive VNEP benchmarks against adapted baselines. Results report gains in prediction and video quality, while qualitative comparisons and ablations examine alignment and training design.
5.1. Settings
- The evaluation uses 400 procedural and 400 predictive samples with no video or question overlap with training, and compares adapted extension, cascaded, and unified-model baselines.Text metrics are BLEU@1/2/3/4 and ROUGE-L; video metrics are FVD, CLIP-V, and CLIP-T.Source: §5.1 · p. 6
5.2. Main Results
- ROUGE-L 0.3631 and CLIP-V 0.8021 for procedural VNEP, above Gemini-FilmWeaver (0.2802; 0.7102) and Omni-Video (0.1075; 0.6293).Joint-GRPO also raises procedural ROUGE-L from 0.2812 to 0.3631 and CLIP-V from 0.7655 to 0.8021 versus SFT.Source: §5.2 · p. 7
- Joint-GRPO corrects SFT’s semantic hallucination and action misalignment in qualitative cases, producing captions and videos aligned with the intended event.Examples include identifying document relationships while maintaining character appearance, and visualizing “sprinkle cheese” as the ground-truth “shower” action.Source: Figure 6 · p. 7, §5.2 · p. 8
5.3. Ablation Study
- Joint-GRPO outperforms isolated VLM or VDM optimization and simply cascading separately optimized models, supporting co-adaptation for coherent caption-video generation.The reported ablations also find Stage 1 alone can yield semantic deviations, while all-in-one training is unstable because reward attribution between captioning and video generation is ambiguous.Source: §5.3 · pp. 8–9
- Removing Stage 1 text fidelity reduces caption accuracy, removing video fidelity harms consistency or coherence, and removing Stage 2 semantic alignment leads to static-frame reward hacking.The ablations support the complete staged design with balanced reward components.Source: §5.3 · p. 9
6. Conclusion
The paper introduces VNEP, VANS with two-stage Joint-GRPO, and VANS-Data-100K. On established benchmarks, VANS achieves state-of-the-art performance in event prediction accuracy and video generation quality.
6. Conclusion
- VANS achieves state-of-the-art performance in both event prediction accuracy and video generation quality on established benchmarks.The paper presents VNEP as next-event reasoning through dynamic video demonstrations, supported by VANS and the 100K-triplet VANS-Data-100K dataset.Source: §6 · p. 9
A. Details of VANS-Data-100K
VANS-Data-100K combines procedural and predictive samples for VNEP. The accompanying reward design combines complementary signals in two training stages to balance semantic and visual qualities.
A. Details of VANS-Data-100K
- VANS-Data-100K contains 100K samples: 30K procedural and 70K predictive, sourced from public video datasets and the Internet.Input videos average 9.43 seconds and target demonstrations average 3.76 seconds.Source: A. Details of VANS-Data-100K · p. 10
B.1. Reward Design
- In Stage 1, text fidelity alone can overlook visual inconsistency, while video fidelity alone can miss an incorrect action; combining them ranks the ground truth highest.The example shows why caption semantics and visual consistency are both assessed.Source: §B.1 · p. 10
- In Stage 2, video fidelity alone can reward a semantically wrong action, while semantic consistency alone can overlook poor visual consistency; their joint reward prioritizes both.The design uses normalized component rewards summed with equal weighting coefficients of 1.Source: §B.1 · p. 10
B.2. Training Process
Training curves show improving rewards across Joint-GRPO’s two stages, with stabilization at different points. The appendix also specifies the model initialization, optimization schedule, and Stage 2 caption filter.
B.2. Training Process
- Stage 1 text and video fidelity rewards improve, its total reward stabilizes after about 600 steps, and thinking length increases.The format reward quickly saturates as the VLM adopts the instruction template.Source: §B.2 · p. 10
- Stage 2 video fidelity and semantic alignment rewards improve and converge after about 1,000 steps, while its total reward stabilizes.The passage attributes this to VDM adaptation that preserves visual consistency while rendering Stage 1 captions semantically faithfully.Source: §B.2 · p. 10
C.1. Training of VANS
- VANS uses Qwen2.5-VL-3B and Wan-2.1-1.3B; Stage 1 trains for 800 steps, and Stage 2 filters captions below ROUGE-L 0.6 before 1,000-step VDM adaptation.Both stages use a 5 × 10^-5 learning rate; the VLM uses LoRA and the VDM uses an ODE-to-SDE conversion for GRPO training.Source: §C.1 · p. 12
C.2. Evaluation
The evaluation uses a shared input and output protocol, compares fine-tuned baselines, and reports VANS inference times alongside unified-model runtimes.
C.2. Evaluation
- Most methods receive the input video, question, and system prompt; generated answers are standardized to 352 × 640 resolution and 33 frames.Video-GPT receives only the input video and uses its native continuation capability; input frame-rate adaptation varies by model architecture.Source: §C.2 · p. 12
- Table 4 compares fine-tuned baselines and identifies architectural design, rather than data advantage, as the primary source of improvement.Source: Table 4 · p. 12
D.1. Inference Time
- Approximately 4 seconds for caption generation and 35 seconds for video generation are reported for VANS, versus 50 seconds for Omni-Video and 60 seconds for VideoGPT.VANS timing uses the official VAN library.Source: §D.1 · p. 12
D.2. Comparison with Fine-tuned Baseline
Joint-GRPO gives VANS its reported strongest fine-tuned performance, while the paper also reports multi-future prediction and reasoning image-to-video generalization.
D.2. Comparison with Fine-tuned Baseline
- Fine-tuned Qwen* reaches ROUGE-L 0.2812, compared with 0.2802 for zero-shot Gemini-2.5-Flash.The comparison is presented as evidence that VANS-Data-100K provides a foundation for reasoning capability.Source: §D.2 · p. 12
- With the same text input, VANS (SFT) improves over Qwen*-Wan*: FVD falls from 140.07 to 85.34 and CLIP-V rises from 0.6795 to 0.7655.The authors suggest the VAE reference feature aids visual consistency.Source: §D.2 · p. 12
- Joint-GRPO raises ROUGE-L from 0.2812 to 0.3631 and CLIP-T from 0.3202 to 0.3824 over VANS (SFT), while reducing FVD to 78.32.The authors identify Joint-GRPO as the decisive enhancement for aligning textual and visual outputs.Source: §D.2 · pp. 12–13
D.3. Generalization
- Figure 10 shows VANS generating a darkening banana after the instruction to leave it for a week, while other baselines struggle with the transformation.The paper attributes image-to-video generalization to training that includes Koala-36M I2V data.Source: Figure 10 · p. 13, §D.3 · p. 13
- Figure 11 compares distinct plausible futures from one scene: VANS predicts coughing for an everyday scenario and smoke exhaling for a cinematic one.The model conditions video answers on different hypothetical questions rather than only deterministic continuation.Source: Figure 11 · p. 13, §D.3 · p. 13
D.4. Human Evaluation
Human evaluators rated Joint-GRPO highest across the reported criteria; the paper also directs readers to supplementary video results.
D.4. Human Evaluation
- Thirty evaluators rated 20 randomly selected examples—10 procedural and 10 predictive—on three dimensions using the human-evaluation protocol.Evaluators averaged 25 years of age, all held at least a bachelor’s degree, and included 20 postgraduate or PhD students and 10 professionals.Source: §D.4 · pp. 13–14
- Joint-GRPO receives the highest human-evaluation scores across semantic correctness, visual consistency, and overall satisfaction.Source: Table 5 · p. 14, §D.4 · p. 14
D.5. Video Results
- Video results corresponding to the paper’s figures, with additional examples, are provided in supplementary.html.Source: §D.5 · p. 14