PaperlayerView exact revision on arXiv

Source-linked AI summary

From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models

Meng Luo, Yicheng Liu, Jiahao Wang, Yuanxing Zhang, Xin Tao, Pengfei Wan, Kun Gai, Hao Fei

arXiv:2609.11242v1cs.CVcs.AI

TL;DR

Video generators can render fluent videos but still struggle to execute rules, satisfy constraints, and realize long-horizon goals. The paper introduces VWG-Bench for fine-grained diagnosis and Vid-PRE for constraint-aware prompt rewriting, finding improved reasoning across generators without architectural modification.

  • Problem

    Whether video generation models genuinely reason over symbolic rules, physical dynamics, and intentional goals remains unresolved, while existing benchmarks often conflate visual quality with cognitive correctness.

  • Method

    The paper combines a 9-dimension, 38-task VWG-Bench with three-level VLM judging and a model-agnostic Vid-PRE rewriter trained using SFT, GRPO, and text-based rewards.

  • Results

    Models consistently score better on visual rendering than on logic-heavy and rule-constrained reasoning, while Vid-PRE substantially improves reasoning accuracy across multiple generators.

  • Takeaways & Limitations

    Separating rendering quality from cognitive correctness provides a rigorous diagnostic lens, while prompt-level reasoning enhancement offers a scalable path toward improved think-with-video capabilities.

Abstract

from arXiv · show

Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--executing symbolic rules, respecting physical laws, and pursuing intentional goals--remains an open question. Existing benchmarks only partially address this, often conflating visual quality with cognitive correctness. We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks. To enable precise diagnosis, we design a three-level VLM-as-Judge protocol that independently assesses video-level fluency, task-level rule adherence, and sample-level goal realization. Evaluations of leading models reveal a striking gap: while models achieve strong rendering scores, they consistently fail on logic-heavy and rule-constrained tasks. To address this, we propose Vid-PRE (Video Prompt Reasoner and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM. Trained via reinforcement learning with purely text-based rewards, Vid-PRE produces concise, constraint-aware prompts without the instability of video-level reward signals. Experiments show that Vid-PRE yields substantial reasoning improvements across multiple generators without architectural modifications. Together, VWG-Bench and Vid-PRE offer a rigorous diagnostic lens and a scalable path toward true think-with-video capabilities. All data and code are publicly available at https://huggingface.co/datasets/KlingTeam/VWG-Bench.

1 Introduction

The paper introduces VWG-Bench to diagnose think-with-video reasoning across broad task dimensions and proposes Vid-PRE to improve rule-aware generation. Its evaluation separates visual fluency from task-rule adherence and goal realization, exposing reasoning failures that prompt rewriting substantially addresses.

  • Benchmark: VWG-Bench covers 9 reasoning dimensions and 38 fine-grained tasks through an automated pipeline for scalable scenario generation.The dimensions span symbolic, graphical, embodied, interactive, counterfactual, spatial, temporal, physical, and social reasoning.
  • Contributions: Together, VWG-Bench and Vid-PRE provide a diagnostic benchmark and a model-agnostic enhancement strategy for think-with-video generation.The approach separates reasoning from rendering while retaining compatibility with different downstream video generators.
  • Evaluation: Its three-level VLM-as-Judge protocol separately scores video fluency, task-level rule adherence, and sample-level goal realization on a 1–5 scale.This decomposition distinguishes rendering quality from cognitive correctness and supports interpretable failure diagnosis.
  • Findings: Leading models show strong video-level rendering but severe degradation on compositional, rule-constrained, and logic-heavy tasks.The reported gap indicates that plausible visual generation does not reliably imply correct reasoning through task rules and success criteria.
  • Vid-PRE: Vid-PRE rewrites inputs into concise, constraint-aware prompts by offloading rule and success-criterion reasoning to a dedicated VLM.It is trained with SFT and GRPO using text-based rewards and improves reasoning across multiple generators without architectural changes.

2 VWG-Bench: Video World Generalist Benchmark

VWG-Bench evaluates video-generation reasoning across nine dimensions and 38 task categories using a structured three-level protocol that separates visual quality, rule adherence, and goal realization.

  • 2 VWG-Bench: Video World Generalist Benchmark: Nine reasoning dimensions and 38 task categories span symbolic logic, visual patterns, embodied interaction, digital interfaces, counterfactuals, spatial geometry, temporal evolution, world simulation, and social understanding.The benchmark samples 10 high-quality instances per category to reduce evaluation and annotation costs.
  • 2 VWG-Bench: Video World Generalist Benchmark: Its three-level protocol separately checks video validity, task-specific rules and constraints, and final objective completion.Video-level assessment filters rendering flaws; task-level assessment checks progress and implicit rules; sample-level assessment checks intermediate milestones and terminal goals.
  • 2 VWG-Bench: Video World Generalist Benchmark: The benchmark combines open-source datasets, programmatic generation, and text-to-image models to construct diverse prompt-image pairs.
  • 2 VWG-Bench: Video World Generalist Benchmark: Structured rule dictionaries specify expected foreground changes, background invariance, and implicit physical constraints, while VLMs generate progress and last-frame goals.
  • 2 VWG-Bench: Video World Generalist Benchmark: The VLM-as-Judge framework uses structured annotations and 1–5 integer scores to prioritize logical coherence and physical adherence over visual fidelity.

3 Vid-PRE: A Universal Prompt Reasoner and Enhancer

Vid-PRE rewrites image-to-video instructions by generating a reasoning chain before a concise executable prompt, then aligns this behavior with supervised data and text-based reinforcement learning.

  • 3 Vid-PRE: A Universal Prompt Reasoner and Enhancer: Vid-PRE assigns spatio-temporal reasoning to a dedicated VLM while downstream video generators focus on rendering explicit executable instructions.
  • 3.1 Problem Formulation: The model learns P(CoT, Popt | I, Puser), inserting a chain-of-thought representation between visual context, user intent, and the optimized prompt.
  • 3.2 Stage 1: Supervised Fine-Tuning with Reasoning Data: Supervised fine-tuning uses approximately 20K curated reasoning tuples generated through initialization, CoT and prompt generation, rejection sampling, and difficulty patching.
  • 3.2 Stage 1: Supervised Fine-Tuning with Reasoning Data: Rejection sampling checks visual consistency against the input image and user instruction, including hallucinated objects, spatial relations, and directional misunderstandings.
  • 3.2 Stage 1: Supervised Fine-Tuning with Reasoning Data: The curated pairs are mixed with general SFT data and used to fine-tune Qwen3-VL-8B with standard negative log-likelihood optimization.
  • 3.3 Stage 2: Policy Alignment via GRPO: GRPO further trains the policy on 1.5K difficult prompts from 10 high-error tasks after supervised fine-tuning leaves complex long-horizon reasoning suboptimal.
  • 3.3 Stage 2: Policy Alignment via GRPO: Text-based rewards evaluate intent preservation, visual-spatial consistency, and prompt quality without rendering videos during reinforcement learning.The design improves stability and efficiency while remaining independent of any specific video generator.
  • 3.3 Stage 2: Policy Alignment via GRPO: GRPO uses group-normalized advantages and KL regularization against the SFT-initialized reference policy to favor higher-reward rewrites.Vid-PRE is intended to produce logically precise and physically grounded reasoning chains for downstream generation.

4 Experiments

Experiments establish a rendering–reasoning gap in current video generators and show that Vid-PRE improves reasoning across benchmarks without modifying generator architectures.

  • 4.2 Main Results: Wan2.2 scores 2.32 in video quality but 1.56 in implicit rule following, while Sora2 falls from 3.45 to 2.98 between video quality and last-frame goal accuracy.
  • 4.2 Main Results: Vid-PRE raises Wan2.2 from 1.72 to 2.58 on MME-CoF, from 26.65 to 44.48 on V-ReasonBench, and from 2.13 to 2.51 on VWG-Bench.
  • 4.2 Main Results: Vid-PRE outperforms native Wan prompt extension, raising Wan2.5 from 2.84 to 3.07 and Wan2.6-Flash from 2.82 to 3.09 on MME-CoF.
  • 4.3 Analysis and Discussion: Human and automatic VWG-Bench scores agree with Spearman’s ρ of 0.69–0.80 and mean absolute error below 0.55 across criteria.
  • 4.3 Analysis and Discussion: SFT provides a structural foundation, while GRPO further improves constraint satisfaction by penalizing spatial hallucinations and logical misalignments.
  • 4.3 Analysis and Discussion: Vid-PRE converts abstract logical problems into explicit, step-by-step visual instructions, improving physical constraint compliance and reducing visual hallucinations.

5 Related Work

Related work spans image-to-video generation, prompt rewriting, perceptual benchmarks, and emerging evaluations of video reasoning, motivating more systematic cognitive assessment.

  • Prompt Rewriting and Optimization: Qualitative comparisons report that generic constraints fail to prevent logical errors and visual hallucinations, whereas Vid-PRE improves reasoning accuracy and video quality.
  • Image-to-Video Generation: Modern image-to-video systems achieve high-resolution synthesis and temporal consistency, but their structured, rule-bound reasoning remains underexplored.
  • Prompt Rewriting and Optimization: Prompt rewriting addresses the distributional gap between concise user inputs and detailed training captions, while reward-guided methods target hallucination and intent drift.
  • Benchmarks for I2V Models: Perceptual benchmarks such as VBench/VBench++ and WorldScore measure visual fidelity and scene dynamics without probing higher-order logical reasoning.
  • Benchmarks for I2V Models: Reasoning benchmarks include broad frameworks such as MME-CoF, VideoThinkBench, and TiViBench alongside specialized pathfinding and rule-adherence evaluations.

6 Conclusion

The paper introduces VWG-Bench to diagnose think-with-video reasoning and Vid-PRE to improve it through constraint-aware prompt enhancement.

  • 6 Conclusion: VWG-Bench reveals that current models render well but struggle with logic-heavy, rule-constrained, and long-horizon tasks.
  • 6 Conclusion: Vid-PRE offloads spatiotemporal reasoning to a dedicated VLM and translates raw instructions into executable prompts.
  • 6 Conclusion: Training with SFT and GRPO using purely text-based rewards yields consistent reasoning improvements across multiple state-of-the-art generators.

Supplementary Material for “From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models”

The supplementary material is attributed to Meng Luo et al.

  • The supplementary material lists Meng Luo et al. as authors.

A.1 Physics Generalization and Per-Dimension Analysis

The evaluation reports physics-focused and dimension-level results, compares Seedance 2.0 with Veo3.1, and analyzes which reward components drive Vid-PRE’s gains and latency.

  • Benchmark Analysis: Table 1 reports physics-focused results, while Table 2 provides the complete dimension-level breakdown for VWG-Bench.
  • Commercial Model Comparison: Table 3 compares Seedance 2.0 with Veo3.1 on VWG-Bench.
  • Reward Sensitivity: Solution correctness is the main source of improvement in GRPO, while intent preservation and image consistency provide complementary gains.Format compliance remains active in every ablation to ensure outputs are parsable.
  • Efficiency: Vid-PRE prompt rewriting adds approximately 3–5 seconds per sample, keeping overhead below 8% of end-to-end latency.The evaluated I2V pipelines typically require more than 60 seconds for video generation.

Task 1: Arithmetic

The arithmetic task requires models to compute expression results and write them in the corresponding positions, while the associated visual prompt asks for the value of x in the final equation.

  • Task Definition: Arithmetic reasoning requires computing a set of expression results and writing them in their corresponding positions.
  • Example Prompt: The example video presents equations over time and asks the model to determine the value of x in the last equation.
  • Related Ordering Task: A separate visual example requires bubbles to leave the frame in ascending order according to their numbers.

Task 3: Coding

The coding task requires computing an answer from a coding problem and input, then writing it in the designated position. Its example prompt asks the model to write the correct answer after “Output.”

  • The task maps a coding problem and input to a computed output written in the corresponding position.
  • Together, the task definition and prompt emphasize both solving the input and placing the answer correctly.
  • The example prompt instructs the model to write the correct answer after “Output.”

Task 4: Sudoku

The section defines Sudoku as filling an incomplete puzzle correctly, while presenting a broader set of symbolic, embodied, interface, physical, and spatial video reasoning tasks. Their prompts require models to infer rules, execute actions, or transform scenes under explicit constraints.

  • Task 4: Sudoku: Sudoku requires filling an incomplete puzzle correctly, illustrated by a prompt to fill the displayed Sudoku.
  • Symbolic tasks include coding, Tic-Tac-Toe, object categorization, color matching, sequence completion, shape placement, transformations, and visual analogies.
  • Embodied tasks require controlling a robotic arm or hand to perform actions, follow candidate trajectories, infer implicit goals, use visual cues, or decompose complex instructions.
  • GUI tasks require operating an interface or clicking a specified location while presenting plausible subsequent interface changes.
  • Physical and causal tasks test reversed laws, intervention-driven chain reactions, fictional state transitions, anomalous forces, and reversed-time evolution.
  • Spatial and viewpoint tasks require maze navigation, real-world object reasoning, fixed-angle image rotation with bounding boxes, and specified 3D transformations.
Loading 2609.11242v1…