PaperlayerView exact revision on arXiv

Source-linked AI summary

Planning with Sketch-Guided Verification for Physics-Aware Video Generation

Yidong Huang, Zun Wang, Han Lin, Dong-Ki Kim, Shayegan Omidshafiei, Jaehong Yoon, Yue Zhang, Mohit Bansal

arXiv:2511.17450v1cs.CVcs.AIcs.CL

TL;DR

Single-shot video plans can be inaccurate, while iterative refinement entails repeated diffusion calls and substantial computational overhead. SketchVerify verifies sampled motion plans on lightweight sketches before synthesis; across two benchmarks, it improves motion quality and physical coherence while cutting planning cost by nearly an order of magnitude versus iterative refinement.

  • Problem

    Single-shot plans can be inaccurate and produce inconsistent or implausible motion, while iterative refinement incurs substantial overhead from repeated diffusion calls.

    Source: §1 · p. 1

  • Method

    SketchVerify samples motion-plan candidates and uses a multimodal verifier to assess lightweight video sketches before passing verified trajectories to a diffusion model for synthesis.

    Source: §1 · p. 1, §3.4 · p. 5

  • Results

    On WorldModelBench and PhyWorldBench, SketchVerify improves instruction following, physical-law adherence, and motion realism over state-of-the-art baselines while reducing planning cost by nearly an order of magnitude versus iterative refinement.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Increasing the number of sampled trajectories yields steady performance gains, supporting broader candidate exploration at test time.

    Source: §1 · p. 2

    The verifier focuses on coarse object motion; 2D bounding boxes can struggle with fine-grained 3D interactions, and external MLLMs may misjudge candidates.

    Source: §A.5 · p. 13

Abstract

from arXiv · show

Recent video generation approaches increasingly rely on planning intermediate control signals such as object trajectories to improve temporal coherence and motion fidelity. However, these methods mostly employ single-shot plans that are typically limited to simple motions, or iterative refinement which requires multiple calls to the video generator, incuring high computational cost. To overcome these limitations, we propose SketchVerify, a training-free, sketch-verification-based planning framework that improves motion planning quality with more dynamically coherent trajectories (i.e., physically plausible and instruction-consistent motions) prior to full video generation by introducing a test-time sampling and verification loop. Given a prompt and a reference image, our method predicts multiple candidate motion plans and ranks them using a vision-language verifier that jointly evaluates semantic alignment with the instruction and physical plausibility. To efficiently score candidate motion plans, we render each trajectory as a lightweight video sketch by compositing objects over a static background, which bypasses the need for expensive, repeated diffusion-based synthesis while achieving comparable performance. We iteratively refine the motion plan until a satisfactory one is identified, which is then passed to the trajectory-conditioned generator for final synthesis. Experiments on WorldModelBench and PhyWorldBench demonstrate that our method significantly improves motion quality, physical realism, and long-term consistency compared to competitive baselines while being substantially more efficient. Our ablation study further shows that scaling up the number of trajectory candidates consistently enhances overall performance.

1. Introduction

SketchVerify addresses the trade-off between uncorrected single-shot motion plans and costly iterative video generation by refining candidate plans through verification on lightweight sketches before synthesis. On WorldModelBench and PhyWorldBench, it improves instruction following, physical coherence, and commonsense consistency while substantially reducing planning cost.

1. Introduction

  • Single-shot planning can propagate inaccuracies into implausible motion, whereas iterative refinement improves realism through feedback but requires repeated diffusion calls and substantial computation.Prior methods also use intermediate object layouts or trajectory plans to guide video generation.Source: §1 · p. 1
  • SketchVerify iteratively verifies motion plans on lightweight video sketches rather than repeatedly synthesizing full videos, enabling test-time search in about five minutes.The paper reports this as over an order of magnitude faster than baselines requiring full generation.Source: §1 · p. 1
  • For each sub-instruction, the method samples bounding-box trajectory candidates, composites the segmented object over a static background, and scores sketches for semantic alignment and physical plausibility.The verifier considers Newtonian consistency, non-penetration, gravity-coherent vertical motion, and shape stability.Source: §1 · p. 2
  • Verified trajectories for the sub-instructions are merged and passed to a trajectory-conditioned diffusion model for final synthesis, without additional training.Planning and verification occur at test time.Source: §1 · p. 2
  • SketchVerify consistently outperforms state-of-the-art open-source I2V models across instruction following, physical-law adherence, and commonsense consistency, while reducing planning cost by nearly an order of magnitude versus iterative refinement.Ablations also report gains from multimodal verification, sketch-based scoring, and sampling more trajectories.Source: §1 · p. 2

2. Related Works

Prior work uses language-model-generated plans or repeated full-video refinement, while SketchVerify shifts refinement to lightweight sketches and plan-level physical reasoning.

MLLM Planning for Video Generation

  • Recent LLM/MLLM video-generation methods expand sparse text prompts into video plans—bounding-box trajectories [23], scene-level keyframes [15], or motion-aware sketches [23]—for layout-guided diffusion synthesis [11] [24] [51].Source: §2 · p. 2

Iterative Refinement for Visual Generation

  • RPG, PhyT2V, VideoRepair, and VISTA [29] refine visual generations by repeatedly evaluating full videos, with the whole process usually taking over 30 minutes.Source: §2 · p. 3

Physics-Aware Video Generation

  • Physics-aware approaches use simulators [35] [42], post-training or reward optimization [21] [26] [41], and physical priors [6] [10] [39] [47]; SketchVerify instead refines plans through multimodal sketch scoring.Source: §2 · p. 3

3. Method – SketchVerify

SketchVerify plans object motion at test time, verifying candidate trajectories with lightweight sketches before final video synthesis. It decomposes prompts into sub-instructions, extracts moving objects and a static background, then passes a verified plan to a trajectory-conditioned generator.

3.1. Overview

  • The overview separates prompt and scene parsing, test-time trajectory search, and trajectory-conditioned video generation into three stages.Source: §3.1 · p. 3

3.2. High-Level Planning and Object Parsing

  • An MLLM converts the prompt into actionable sub-instructions, identifies moving objects, and supports mask extraction and background inpainting to create a static canvas.Source: §3.2 · p. 3

3.3. Test-Time Planning

  • SketchVerify samples object-level trajectory candidates, renders them as lightweight sketches, and ranks them for instruction alignment and physical plausibility before video generation.Source: §3.3 · pp. 3–5
  • If every candidate falls below a quality threshold, the method discards the batch, resamples using failure feedback, and carries the selected sketch’s last frame into the next sub-instruction.Source: §3.3 · p. 5

3.4. Trajectory-Conditioned Video Generation

  • The verified bounding-box paths are converted into sparse points, concatenated and interpolated, then injected as trajectory latents into a pretrained diffusion generator.Source: §3.4 · p. 5

4. Experimental Results

The experiments use two benchmarks and assess instruction following, physical coherence, commonsense consistency, and efficiency. Implementation details specify the candidate-generation, sketch-scoring, and video-generation setup.

4.1. Benchmarks and Evaluation Metrics

  • WorldModelBench covers seven domains, while PhyWorldBench tests fine-grained physical realism across 350 prompts.For PhyWorldBench, the first frame is generated with FLUX before image-to-video generation.Source: §4.1 · p. 5
  • Evaluation covers instruction following, five physical-law dimensions, frame and temporal commonsense, and efficiency, using benchmark-provided MLLM scoring.The physical-law dimensions are Newtonian motion, deformation, fluid, penetration, and gravity.Source: §4.1 · p. 5

4.2. Implementation Details

  • GPT-4.1 generates five candidate trajectories, and Gemini 2.5 scores object-composited sketches for semantic alignment and physical laws.GroundedSAM segments foreground objects; ATI-14B generates 81-frame, 480p videos.Source: §4.2 · p. 5
  • Table 1 compares instruction following, physics, frame and temporal commonsense, and an overall sum score on WorldModelBench.Boldface marks the best result in each column.Source: Table 1 · p. 6

5. Quantitative Results

SketchVerify improves video-generation results across WorldModelBench and PhyWorldBench while reducing planning cost; qualitative comparisons and ablations show gains from multimodal verification and test-time trajectory selection.

5. Quantitative Results

  • SketchVerify leads major WorldModelBench dimensions and PhyWorldBench overall (19.84) and physical standard (23.52), while planning takes 4.7 minutes versus 12.5 minutes for PhyT2V.On WorldModelBench, it improves instruction-following accuracy by 10.6% and overall physics coherence by 6% versus Wan-2.1; on PhyWorldBench, Cosmos leads Obj+Evt (48.29 vs. 43.11).Source: §5 · p. 6, Table 2 · p. 6

5.1. Qualitative Results

  • Across Human, Natural, Video Game, and Robotics scenes, SketchVerify produces more plausible motion, including gravity-consistent jumps, downhill snow flow, accurate contact, and stable grasps.The comparisons use the same prompt per model group and uniformly sampled frames from each 81-frame video.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2. Ablation Study

  • Adding visual input lets the verifier assess geometry and motion interactions, reject implausible trajectories such as gravity violations, and improve instruction following and physical plausibility.Language-only verification offers limited spatial cues for assessing object geometry, depth, and trajectory smoothness.Source: §5.2 · pp. 7–8, Figure 5 · p. 8
  • Stronger verifier models improve motion selection: Gemini-2.5 yields higher overall quality than Qwen-VL-3B in the verifier comparison.The table caption attributes stronger semantic and physical reasoning to larger multimodal verifiers.Source: §5.2 · p. 8, Table 4 · p. 8
  • Stronger planner choices improve semantic and physical reasoning, while weaker open-source VLMs struggle with instruction following in multi-step, numerically precise planning.GPT-4.1 provides more reliable motion-quality assessments than smaller planner choices in the reported comparison.Source: §5.2 · p. 8, Table 5 · p. 8
  • Increasing candidate count improves verifier-guided selection; K = 5 achieves the strongest instruction-following and physics-coherence results, while K = 0 matches VideoMSG.At K = 0, instruction adherence is 1.46 and physical consistency is 4.55; performance continues to rise with larger K.Source: §5.2 · p. 9, Figure 6 · p. 9
  • Sketch-based verification matches or surpasses full-video verification while achieving a nearly 10× speedup.Sketches isolate motion from appearance, avoiding diffusion artifacts that can mislead the verifier; the table reports planning and verification time before final video generation.Source: §5.2 · p. 9, Table 6 · p. 9

6. Conclusion

SketchVerify separates motion planning from video synthesis and uses sketch-based candidate verification to improve physically grounded video generation. Experiments on WorldModelBench and PhyWorldBench report improvements over SoTA baselines and nearly an order-of-magnitude lower planning cost than iterative refinement methods.

6. Conclusion

  • SketchVerify combines test-time trajectory sampling, multimodal verification, and sketch-based proxy rendering, improving instruction following, physical-law adherence, and motion realism on WorldModelBench and PhyWorldBench over SoTA baselines.The framework decouples motion planning from synthesis and reduces planning cost by nearly an order of magnitude compared with iterative refinement methods.Source: §6 · p. 9

A.1.1. SketchVerify Pipeline Specification

The implementation decomposes an action into ordered sub-instructions, plans object trajectories, and verifies lightweight sketches before generating the final video. It specifies the planning, perception, rendering, scoring, and generation components used in this pipeline.

High-Level Planning

  • GPT-4.1 decomposes the prompt into 1–4 ordered motion sub-instructions, each with moving objects and a temporal budget, parsed through a strict JSON schema.The phase lengths sum to 41 frames; malformed or underspecified plans are rejected and resampled.Source: §A.1.1 · p. 12

Object Detection and Masks

  • Grounding DINO detects moving objects and SAM-HQ segments them; the pipeline retains each object's highest-scoring mask and derives a normalized bounding box.The object-detection confidence threshold is 0.3; FLUX.1-dev with an Omnieraser LoRA inpaints the masked objects to create a static background.Source: §A.1.1 · p. 12

Test-Time Search

  • For each sub-instruction, GPT-4.1 samples five candidate trajectories of per-frame bounding boxes, with malformed outputs resampled and candidates separated by a minimum normalized-coordinate ℓ2 distance of 0.05.The planner uses temperature 1.0.Source: §A.1.1 · p. 12

Sketch Rendering and Multimodal Verification

  • Each candidate is rendered as a lightweight sketch by compositing first-frame object sprites onto the static background, then Gemini 2.5-Flash scores semantic alignment and physical plausibility.Semantic alignment uses first/last-frame comparison, physics plausibility uses the full sketch, and the default semantic and physics weights are both 0.5.Source: §A.1.1 · p. 12

Scoring and Video Generation

  • The verifier rates plans from 0.0–0.3 for poor alignment or broken physical laws to 1.0 for perfect alignment and physical-law coherence, and the selected trajectory conditions ATI-14B video generation.ATI-14B generates 81-frame 480p videos using 40 steps and cfg = 5.0.Source: §A.1.1 · p. 12

A.1.2. Baseline and Hardware Specifications

The evaluation compares SketchVerify against open-source video-generation baselines using official implementations and standardized video settings. Experiments use NVIDIA A100 80G and RTX A6000 GPUs.

A.1.2. Baseline and Hardware Specifications

  • The baselines are Wan2.1-14B-480p-I2V, CogVideoX-5B, Cosmos-Predict2-2B, HunyuanVideo-I2V, OpenSora-I2V, and Step-Video-I2V, sampled at 480p for 81 frames with 50 diffusion steps.PhyT2V and VideoMSG use Wan2.1 as their backbone for fair comparison; experiments run on NVIDIA A100 80G and RTX A6000 GPUs.Source: §A.1.2 · p. 12

A.1.3. Per-step Runtime

The appendix reports per-step runtime, benchmark and metric definitions, qualitative comparisons, prompt-template coverage, and remaining limitations. It describes the evaluation settings and notes constraints in physics verification, MLLM judgments, motion representation, and video generation.

A.1.3. Per-step Runtime

  • 72.5 s is the average test-time planning duration per sub-instruction, including 20.3 s for trajectory sampling and 52.2 s for multimodal verification.High-level planning averages 14.16 s, while object detection, segmentation, and background inpainting take 108 s; timings use one NVIDIA A100 GPU with GPT-4.1 and Gemini-2.5 standard-speed APIs.Source: §A.1.3 · p. 12

A.2.1. Benchmark Details

  • WorldModelBench covers seven domains and 56 subdomains across 350 image/text-conditioned tasks, while PhyWorldBench has 350 text prompts testing fine-grained physical realism.WorldModelBench supports I2V and T2V, with this evaluation using I2V; PhyWorldBench is T2V, so the study generates a first frame with FLUX before I2V generation.Source: §A.2.1 · pp. 12–13

A.2.2. Metric Details

  • WorldModelBench scores instruction following, six physics dimensions, and visual realism and motion continuity; PhyWorldBench reports Obj+Evt, Phys. Std, and All pass rates.WorldModelBench uses an MLLM scorer trained on 67K human annotations; PhyWorldBench samples eight frames per video for proprietary MLLM evaluation, with All requiring both Obj+Evt and Phys. Std to pass.Source: §A.2.2 · p. 13

A.4. Prompt Template for SketchVerify Pipeline

  • The prompt templates cover high-level planning, object proposals, sub-instruction trajectory generation, semantic-alignment verification, and physical-plausibility checking.The described pipeline also includes a global movement-planning prompt and instructs the model to complete the task within the specified total frames.Source: §A.4 · p. 13, §A.5 · p. 16

A.5. Limitations

  • The verifier assesses coarse object motion and high-level physical plausibility, while fine-grained dynamics such as friction and collision response require modeling beyond the current verifier-based design.The passage also notes possible incorrect MLLM judgments, limits of 2D bounding boxes for fine-grained 3D interactions, and dependence of final-video realism on the generator.Source: §A.5 · p. 13

A.3. Extra Qualitative Results

  • On WorldModelBench, SketchVerify produces smoother, more semantically aligned motions than baseline models; on PhyWorldBench, it more reliably maintains physical plausibility.The reported PhyWorldBench examples avoid floating against gravity and self-motion that violates Newton's first law.Source: §A.3 · p. 13, Figure 7 · p. 14, Figure 8 · p. 15
Loading 2511.17450v1…