From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models

Benchmark

• VWG-Bench covers 9 reasoning dimensions and 38 fine-grained tasks through an automated pipeline for scalable scenario generation.The dimensions span symbolic, graphical, embodied, interactive, counterfactual, spatial, temporal, physical, and social reasoning.Source: §1 · pp. 2, 4

Contributions

• Together, VWG-Bench and Vid-PRE provide a diagnostic benchmark and a model-agnostic enhancement strategy for think-with-video generation.The approach separates reasoning from rendering while retaining compatibility with different downstream video generators.Source: §1 · pp. 2–4

Evaluation

• Its three-level VLM-as-Judge protocol separately scores video fluency, task-level rule adherence, and sample-level goal realization on a 1–5 scale.This decomposition distinguishes rendering quality from cognitive correctness and supports interpretable failure diagnosis.Source: §1 · pp. 2–4

Findings

• Leading models show strong video-level rendering but severe degradation on compositional, rule-constrained, and logic-heavy tasks.The reported gap indicates that plausible visual generation does not reliably imply correct reasoning through task rules and success criteria.Source: §1 · p. 3

Vid-PRE

• Vid-PRE rewrites inputs into concise, constraint-aware prompts by offloading rule and success-criterion reasoning to a dedicated VLM.It is trained with SFT and GRPO using text-based rewards and improves reasoning across multiple generators without architectural changes.Source: §1 · pp. 3–4

2 VWG-Bench: Video World Generalist Benchmark

• Nine reasoning dimensions and 38 task categories span symbolic logic, visual patterns, embodied interaction, digital interfaces, counterfactuals, spatial geometry, temporal evolution, world simulation, and social understanding.The benchmark samples 10 high-quality instances per category to reduce evaluation and annotation costs.Source: §2.1 · p. 4, §2.2 · p. 5
• Its three-level protocol separately checks video validity, task-specific rules and constraints, and final objective completion.Video-level assessment filters rendering flaws; task-level assessment checks progress and implicit rules; sample-level assessment checks intermediate milestones and terminal goals.Source: §2.1 · p. 4, §2.4 · pp. 6–7
• The benchmark combines open-source datasets, programmatic generation, and text-to-image models to construct diverse prompt-image pairs.Source: §2.3 · p. 5
• Structured rule dictionaries specify expected foreground changes, background invariance, and implicit physical constraints, while VLMs generate progress and last-frame goals.Source: §2.3 · pp. 5–6
• The VLM-as-Judge framework uses structured annotations and 1–5 integer scores to prioritize logical coherence and physical adherence over visual fidelity.Source: §2.4 · p. 6

3 Vid-PRE: A Universal Prompt Reasoner and Enhancer

• Vid-PRE assigns spatio-temporal reasoning to a dedicated VLM while downstream video generators focus on rendering explicit executable instructions.Source: §3 · p. 7

3.1 Problem Formulation

• The model learns P(CoT, Popt | I, Puser), inserting a chain-of-thought representation between visual context, user intent, and the optimized prompt.Source: §3.1 · pp. 7–8

3.2 Stage 1: Supervised Fine-Tuning with Reasoning Data

• Supervised fine-tuning uses approximately 20K curated reasoning tuples generated through initialization, CoT and prompt generation, rejection sampling, and difficulty patching.Source: §3.2 · pp. 8–9
• Rejection sampling checks visual consistency against the input image and user instruction, including hallucinated objects, spatial relations, and directional misunderstandings.Source: §3.2 · p. 8
• The curated pairs are mixed with general SFT data and used to fine-tune Qwen3-VL-8B with standard negative log-likelihood optimization.Source: §3.2 · p. 9

3.3 Stage 2: Policy Alignment via GRPO

• GRPO further trains the policy on 1.5K difficult prompts from 10 high-error tasks after supervised fine-tuning leaves complex long-horizon reasoning suboptimal.Source: §3.3 · p. 9
• Text-based rewards evaluate intent preservation, visual-spatial consistency, and prompt quality without rendering videos during reinforcement learning.The design improves stability and efficiency while remaining independent of any specific video generator.Source: §3.3 · p. 9
• GRPO uses group-normalized advantages and KL regularization against the SFT-initialized reference policy to favor higher-reward rewrites.Vid-PRE is intended to produce logically precise and physically grounded reasoning chains for downstream generation.Source: §3.3 · pp. 9–10

4.2 Main Results

• Wan2.2 scores 2.32 in video quality but 1.56 in implicit rule following, while Sora2 falls from 3.45 to 2.98 between video quality and last-frame goal accuracy.Source: §4.2 · p. 11
• Vid-PRE raises Wan2.2 from 1.72 to 2.58 on MME-CoF, from 26.65 to 44.48 on V-ReasonBench, and from 2.13 to 2.51 on VWG-Bench.Source: §4.2 · p. 11, Table 3 · p. 12
• Vid-PRE outperforms native Wan prompt extension, raising Wan2.5 from 2.84 to 3.07 and Wan2.6-Flash from 2.82 to 3.09 on MME-CoF.Source: §4.2 · p. 11, Table 3 · p. 12

4.3 Analysis and Discussion

• Human and automatic VWG-Bench scores agree with Spearman’s ρ of 0.69–0.80 and mean absolute error below 0.55 across criteria.Source: §4.3 · p. 12, Table 4 · p. 13
• SFT provides a structural foundation, while GRPO further improves constraint satisfaction by penalizing spatial hallucinations and logical misalignments.Source: §4.3 · pp. 12–13, Table 5 · p. 13
• Vid-PRE converts abstract logical problems into explicit, step-by-step visual instructions, improving physical constraint compliance and reducing visual hallucinations.Source: §4.3 · p. 13

Prompt Rewriting and Optimization

• Qualitative comparisons report that generic constraints fail to prevent logical errors and visual hallucinations, whereas Vid-PRE improves reasoning accuracy and video quality.Source: Figure 3 · p. 14

Image-to-Video Generation

• Modern image-to-video systems achieve high-resolution synthesis and temporal consistency, but their structured, rule-bound reasoning remains underexplored.Source: §5 · pp. 13–14

Prompt Rewriting and Optimization

• Prompt rewriting addresses the distributional gap between concise user inputs and detailed training captions, while reward-guided methods target hallucination and intent drift.Source: §5 · p. 14

Benchmarks for I2V Models

• Perceptual benchmarks such as VBench/VBench++ and WorldScore measure visual fidelity and scene dynamics without probing higher-order logical reasoning.Source: §5 · pp. 14–15
• Reasoning benchmarks include broad frameworks such as MME-CoF, VideoThinkBench, and TiViBench alongside specialized pathfinding and rule-adherence evaluations.Source: §5 · p. 15

6 Conclusion

• VWG-Bench reveals that current models render well but struggle with logic-heavy, rule-constrained, and long-horizon tasks.Source: §6 · p. 15
• Vid-PRE offloads spatiotemporal reasoning to a dedicated VLM and translates raw instructions into executable prompts.Source: §6 · p. 15
• Training with SFT and GRPO using purely text-based rewards yields consistent reasoning improvements across multiple state-of-the-art generators.Source: §6 · p. 15

• The supplementary material lists Meng Luo et al. as authors.Source: Supplementary Material for “From… · p. 19

Benchmark Analysis

• Table 1 reports physics-focused results, while Table 2 provides the complete dimension-level breakdown for VWG-Bench.Source: Table 1 · p. 19, Table 2 · p. 19, §A.1 · p. 19

Commercial Model Comparison

• Table 3 compares Seedance 2.0 with Veo3.1 on VWG-Bench.Source: §A.2 · p. 19, Table 3 · p. 20

Reward Sensitivity

• Solution correctness is the main source of improvement in GRPO, while intent preservation and image consistency provide complementary gains.Format compliance remains active in every ablation to ensure outputs are parsable.Source: §A.2 · p. 20, Table 4 · p. 20

Efficiency

• Vid-PRE prompt rewriting adds approximately 3–5 seconds per sample, keeping overhead below 8% of end-to-end latency.The evaluated I2V pipelines typically require more than 60 seconds for video generation.Source: §A.3 · p. 20

Task Definition

• Arithmetic reasoning requires computing a set of expression results and writing them in their corresponding positions.Source: Task 1: Arithmetic · p. 20

Example Prompt

• The example video presents equations over time and asks the model to determine the value of x in the last equation.Source: Figure 1 · p. 21

Related Ordering Task

• A separate visual example requires bubbles to leave the frame in ascending order according to their numbers.Source: Figure 2 · p. 21

• The task maps a coding problem and input to a computed output written in the corresponding position.Source: Task 3: Coding · p. 22
• Together, the task definition and prompt emphasize both solving the input and placing the answer correctly.Source: Task 3: Coding · p. 22, Figure 3 · p. 22
• The example prompt instructs the model to write the correct answer after “Output.”Source: Figure 3 · p. 22

Task 4: Sudoku

• Sudoku requires filling an incomplete puzzle correctly, illustrated by a prompt to fill the displayed Sudoku.Source: Task 4: Sudoku · p. 22, Figure 4 · p. 22

• Symbolic tasks include coding, Tic-Tac-Toe, object categorization, color matching, sequence completion, shape placement, transformations, and visual analogies.Source: Task 5: Tic-Tac-Toe · p. 23, Task 6: Box Matching · p. 23, Task 7: Color Connecting · p. 24
• Embodied tasks require controlling a robotic arm or hand to perform actions, follow candidate trajectories, infer implicit goals, use visual cues, or decompose complex instructions.Source: Task 11: Embodied Action Execution · p. 26, Task 12: Embodied Trajectory Reasoning · p. 26, Task 13: Implicit Goal Reasoning · p. 27
• GUI tasks require operating an interface or clicking a specified location while presenting plausible subsequent interface changes.Source: Task 16: GUI Instruction Execution · p. 28, Task 17: GUI Click Reasoning · p. 29
• Physical and causal tasks test reversed laws, intervention-driven chain reactions, fictional state transitions, anomalous forces, and reversed-time evolution.Source: Task 18: Physics Inversion · p. 29, Task 19: Causal Intervention · p. 30, Task 20: Fictional Rule Execution · p. 30
• Spatial and viewpoint tasks require maze navigation, real-world object reasoning, fixed-angle image rotation with bounding boxes, and specified 3D transformations.Source: Task 23: Maze · p. 32, Task 24: Real World Spatial Reasoning · p. 32, Task 25: Rotation Reasoning · p. 33