PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning

Chengzu Li, Zanyi Wang, Jiaang Li, Yi Xu, Han Zhou, Huanyu Zhang, Ruichuan An, Dengyang Jiang, Zhaochong An, Ivan Vulić, Serge Belongie, Anna Korhonen

arXiv:2601.21037v1cs.LGcs.AIcs.CLcs.CV

TL;DR

Text-based vision-language models remain limited in fine-grained spatial reasoning and continuous action planning. This paper uses video generation as visual reasoning, evaluating discrete maze navigation and continuous tangram manipulation. It reports robust zero-shot generalization, visual context as control, and frame-budget scaling for complex sequential planning, while identifying geometric consistency as a limitation in high-change manipulation.

  • Problem

    MLLMs often reason through text, which is limited for fine-grained geometric contexts and physical dynamics; prior visual planning studies also underexplored OOD generalization and high-change continuous manipulation.

  • Method

    The paper evaluates video generation models in MAZENAVIGATION and TANGRAMPUZZLE, covering low-change sequential planning and high-change continuous manipulation.

  • Results

    The experiments show strong generalization without fine-tuning, visual context control for unseen patterns, and improved sequential planning as the generated frame budget increases.

  • Takeaways & Limitations

    Generated video frames can serve as intermediate visual reasoning steps, making video generation a scalable paradigm for spatial planning beyond media synthesis.

  • Takeaways & Limitations

    In TANGRAMPUZZLE, longer generation does not consistently improve translation performance because tangram pieces can undergo geometric deformation over long, high-change windows.

Abstract

from arXiv · show

Vision-Language Models have excelled at textual reasoning, but they often struggle with fine-grained spatial understanding and continuous action planning, failing to simulate the dynamics required for complex visual reasoning. In this work, we formulate visual reasoning by means of video generation models, positing that generated frames can act as intermediate reasoning steps between initial states and solutions. We evaluate their capacity in two distinct regimes: Maze Navigation for sequential discrete planning with low visual change and Tangram Puzzle for continuous manipulation with high visual change. Our experiments reveal three critical insights: (1) Robust Zero-Shot Generalization: In both tasks, the model demonstrates strong performance on unseen data distributions without specific finetuning. (2) Visual Context: The model effectively uses visual context as explicit control, such as agent icons and tangram shapes, enabling it to maintain high visual consistency and adapt its planning capability robustly to unseen patterns. (3) Visual Test-Time Scaling: We observe a test-time scaling law in sequential planning; increasing the generated video length (visual inference budget) empowers better zero-shot generalization to spatially and temporally complex paths. These findings suggest that video generation is not merely a media tool, but a scalable, generalizable paradigm for visual reasoning.

1. Introduction

The paper investigates video generation as a visual reasoning medium for spatial planning, addressing limitations of text-based reasoning in fine-grained geometry and dynamics. It evaluates novel regimes and finds robust generalization, visual context control, and frame-budget scaling for complex sequential planning.

  • MLLMs often struggle with fine-grained geometric contexts and physical dynamics when reasoning through textual descriptions.
  • The paper evaluates video reasoning across MAZENAVIGATION for low-change sequential planning and TANGRAMPUZZLE for high-change continuous manipulation.TANGRAMPUZZLE extends evaluation beyond discrete maze actions to continuous geometric manipulation.
  • The experiments report that video-based reasoning consistently outperforms text-based baselines, particularly on TANGRAMPUZZLE.
  • Video generation models use visual context, including agent icons and puzzle shapes, as control to maintain consistency and adapt to unseen patterns without fine-tuning.The reported contexts support zero-shot generalization across spatially and visually out-of-distribution settings.
  • Increasing generated frames from 81 to 121 acts as a larger visual inference budget and enables stronger performance on spatially and temporally complex paths.

2. Related Work

Related work extends textual reasoning with multimodal traces and repurposes video generation for visual reasoning. This paper identifies visual test-time scaling as a direct analogue of longer reasoning traces for planning.

  • Multimodal planning research uses symbolic programs, coordinate representations, external generative tools, and native multimodal traces to ground reasoning.
  • Video generation models were traditionally optimized for media creation, visual fidelity, temporal coherence, and consistency as world models.
  • Prior video-based reasoning efforts included qualitative evaluations of proprietary and open-source models, while quantitative investigation and reproducibility remained limited.
  • The paper shows that increasing the video frame budget is empirically similar to lengthening reasoning traces in language models, enabling planning with OOD complexity.

3. Video Generation for Visual Planning

The paper formulates video generation as visual planning: generated frames transition an initial state toward a goal while preserving temporal and physical constraints. It evaluates contrasting discrete-navigation and continuous-manipulation regimes to test both planning and geometric consistency.

  • Visual planning generates a video sequence from an initial state toward a goal while maintaining temporal consistency and latent physical constraints.
  • Prior grid-world navigation tests sequential logic but largely uses discrete actions, static environments, and local visual changes.
  • Spectrum of Visual Planning Regimes: The evaluation contrasts MAZENAVIGATION’s structured, low-change planning with TANGRAMPUZZLE’s continuous manipulation under high visual change.
  • MAZENAVIGATION: MAZENAVIGATION tests long-term map consistency and precise collision-free pathfinding as an agent moves through a maze.
  • TANGRAMPUZZLE: TANGRAMPUZZLE requires seven geometric pieces to fill a target silhouette while the scene changes continuously through translation and rotation.

4. Experiments

The experiments evaluate video generation for maze navigation and tangram manipulation using hierarchical metrics, multiple model baselines, and both in-distribution and out-of-distribution settings. Results show reliable sequential planning, strong zero-shot generalization, and substantial dependence on preserved visual context for geometric manipulation.

  • Experimental setup: The evaluation covers maze sizes from 3 × 3 to 6 × 6, varied agent icons, and tangram puzzles with geometric constraints.Maze paths are generated with heuristic search, while tangram solutions are assessed on the final frame.
  • Experimental setup: The study compares the Wan 2.2 TI2V 5B video model with proprietary and open-source MLLM baselines, VPRL, and Qwen-Image-Edit.The video model is fine-tuned with LoRA, while Qwen3-VL-8B receives full-parameter fine-tuning for textual reasoning baselines.
  • In-distribution results: 98.0% and 96.0% Exact Match on standard 4 × 4 and 5 × 5 mazes demonstrate strong in-distribution sequential planning.The generated trajectories model valid spatiotemporal transitions, obey environmental constraints, and reach the goal.
  • In-distribution results: 0.8% accuracy in Fade-In tangrams shows that removing initial geometric priors severely impairs solution quality.The model also achieves near-zero training accuracy in this setting, where the initial tangram shapes are absent.
  • In-distribution results: 68.0% performance with Translation versus 22.4% under Rotation indicates that preserving piece shapes and orientations improves geometric consistency.Rotation introduces pixel-level warping, whereas Translation preserves visual context while isolating translation planning.
  • Out-of-distribution results: OOD performance remains strong: EM reaches 92.00% on unseen 7 × 7 mazes, while Translation accuracy reaches 60.8% on unseen tangram silhouettes versus 68.0% on seen silhouettes.Unseen icons cause only marginal changes on reported maze sizes, and the model preserves their identity and texture during generation.

5. Discussion and Analysis

The analysis shows that visual context provides explicit control for robust out-of-distribution planning, while additional video frames improve maze navigation but expose task-dependent fidelity limits.

  • Visual Context as Control: Visual anchoring outperforms textual descriptions for controlling unseen agent appearances in MAZENAVIGATION.Text prompts often revert to the default training-distribution agent, whereas visual context directly specifies the desired appearance.
  • Visual Context as Control: Visual context acts as geometric control in TANGRAMPUZZLE by preserving shape and orientation while the model solves for location.The translation setting provides a reference map, allowing geometric transformations without inventing new shapes.
  • Visual Test-Time Scaling: Increasing total video frames from 61 to 121 steadily improves out-of-distribution maze performance across spatial and temporal challenges.Scaling to 141 frames reduces temporal-OOD performance, although it remains above the 81-frame training baseline.
  • Visual Test-Time Scaling: Allocating more frames per maze step improves performance at κ = 7, 9, and 11 relative to κ = 5, but temporal-OOD performance peaks at κ = 9.The decline at κ = 11 indicates that finer temporal allocation does not improve every setting.
  • Visual Test-Time Scaling: Higher frame budgets produce qualitative self-correction, including backtracking after initially steering toward a wrong direction.These trajectories suggest intermediate frames support active rollout correction rather than simple path retrieval.
  • Fidelity-Reasoning Trade-off: Scaling benefits MAZENAVIGATION but not TANGRAMPUZZLE, where longer generation preserves performance without significant gains and can tax geometric fidelity.Shape-consistency and task success correlate at ρ ≥0.6, highlighting geometry maintenance as a Tangram bottleneck.
  • Generalization: The model adapts zero-shot to irregular mazes by generating diagonal trajectories while respecting collision constraints, although success is lower than on standard mazes.This behavior indicates adaptation beyond the grid-based training layouts.
  • Model Comparison: Image Editing achieves higher TANGRAMPUZZLE performance, whereas Video Generation provides intermediate reasoning traces that improve interpretability and verifiability.The comparison is affected by model-size and optimization-density differences.

6. Conclusion

The paper presents video generation as a visual reasoning medium for both discrete maze planning and continuous geometric manipulation. It reports strong generalization and a frame-based reasoning budget that improves long-horizon maze planning, while geometric consistency remains a bottleneck.

  • Conclusion: Video generation models support visual reasoning across discrete MAZENAVIGATION and continuous, high-fidelity TANGRAMPUZZLE challenges.The paper frames generated video as an expressive representation for spatial planning beyond text-based reasoning.
  • Conclusion: The models generalize across task-level logic rather than relying only on superficial visual patterns.The conclusion identifies this generalization alongside geometric-consistency limitations.
  • Conclusion: Increasing generated frames acts as a temporal reasoning budget that produces emergent competence on out-of-distribution, long-horizon sequential problems.The paper relates this behavior to System-2-style reasoning and calls for further research.

A.1. Dataset

The evaluation uses maze environments spanning grid size, path length, appearance, and distribution shift, alongside a seven-piece Tangram dataset with multiple geometric-action variants.

  • MAZENAVIGATION: MAZENAVIGATION covers 3 × 3 to 8 × 8 grids with optimal planning lengths from 2–18 steps.Training uses randomly initialized reachable states and built-in search solutions.
  • MAZENAVIGATION: The maze training set contains 1,000 unique instances across 3 × 3 to 6 × 6 grids, with 40 distinct visual icons.Evaluation includes structural and temporal difficulty tiers.
  • MAZENAVIGATION: Spatial OOD uses 7 × 7 and 8 × 8 mazes while retaining training-level path lengths.This tests adaptation to unseen spatial scales.
  • MAZENAVIGATION: Temporal OOD increases path lengths to 13–18 steps within 5 × 5 and 6 × 6 grids.The setting targets long-range dependency modeling and compositional reasoning.
  • MAZENAVIGATION: Combined temporal-and-spatial OOD uses 13–18-step paths in 7 × 7 and 8 × 8 grids.This jointly increases maze scale and planning complexity.
  • MAZENAVIGATION: The visually OOD set uses unseen icons with identical instance-wise maze layouts across icon sets.This controls for layout while testing generalization to unseen appearances.
  • TANGRAMPUZZLE: TANGRAMPUZZLE contains 692 silhouettes, each paired with seven disjoint, color-coded geometric pieces and a unique ground-truth layout.The dataset is derived from black-background Kilogram silhouettes with grid structures removed.
  • TANGRAMPUZZLE: Rotation requires sequential SO(2) rotations followed by translations, whereas Translation requires only target-centroid localization.Fade-In likewise limits the objective to localizing target centroids.

A.2. Models and Hyper-Parameters

The study compares text, image, and video planning modalities, using fine-tuned video and image models alongside language-based and proprietary baselines across the two tasks.

  • Modalities: Supervised fine-tuning compares text, image, and video modalities for planning.The modality comparison is conducted across the study’s planning tasks.
  • Text Planning: Text planning generates discrete maze actions or Tangram coordinates and rotation degrees in JSON format conditioned on visual state and prompts.The formulation treats planning as conditional sequence generation grounded in visual context.
  • Baselines: The experiments include GPT-5.1, GPT-5.2, Qwen 2.5 VL 7B, Nano-Banana, and Qwen Image Edit 2509.Qwen 2.5 VL 7B is fine-tuned on both tasks, while proprietary evaluations are task-specific.
  • Image Planning: Image planning uses discrete per-step images for MAZENAVIGATION and direct image-editing prediction for TANGRAMPUZZLE.The maze image baseline replicates the Visual Planning framework, while Tangram uses image-editing models.
  • Video Planning: Video planning fine-tunes Wan 2.2 TI2V 5B with LoRA for 20 epochs on both tasks.MAZENAVIGATION uses 81-frame, 480 × 832 sequences; Tangram uses 201 frames for Rotation and randomized 61–81 frames for Translation.

A.3. Evaluator

The paper evaluates maze navigation with motion-centric, speed-invariant extraction and tangram solving with deterministic pixel-level geometric checks. These evaluators measure path fidelity, goal completion, boundary adherence, and indicative temporal piece integrity.

  • MAZENAVIGATION: Motion-centric extraction isolates the maze agent from a static environment, supporting evaluation across varying agent designs without task-specific detectors.The pipeline models the static maze as a reference and extracts the agent from its dynamics.
  • MAZENAVIGATION: Speed-invariant alignment resamples trajectories by cumulative geometric distance, separating spatial path fidelity from differences in travel speed.This evaluates where the agent went rather than how quickly it matched the reference trajectory.
  • TANGRAMPUZZLE: The tangram evaluator uses pixel-level segmentation and geometric primitives instead of neural similarity scores to assess precise geometric constraints.It processes generated videos and extracts metrics from color segmentation and contour information.
  • TANGRAMPUZZLE: Tangram evaluation establishes each piece’s initial area, shape class, and target silhouette from source or target layouts.The evaluator uses seven uniquely colored pieces and derives geometric properties from the input or golden target layouts.
  • TANGRAMPUZZLE: Shape classification approximates segmented contours and distinguishes triangles, squares, and parallelograms using vertices, angles, and aspect ratios.The method uses cv2.approxPolyDP to tolerate artifacts such as anti-aliasing or slight warping.
  • TANGRAMPUZZLE: Goal completion requires every piece to satisfy area, shape-preservation, and silhouette-containment constraints in the final frame.A trial succeeds only when the average piece-wise completion score is 1.0.
  • TANGRAMPUZZLE: Mask IoU measures boundary adherence by comparing the union of generated color masks with the target silhouette, where 1.0 means perfect filling without overflow or gaps.The score is defined as pixel-wise precision relative to the target silhouette mask.
  • TANGRAMPUZZLE: Piece integrity tracks whether visible piece areas remain within 0.6–1.4 times their initial areas across sampled frames, but is reported only indicatively.The metric is excluded from the main result table and compares integrity flags against a reference video or ideal static assumption.

B. Results

Video generation models perform strongly on maze planning and show robust object permanence across seen and unseen icons, while text-based reasoning struggles with precise continuous tangram manipulation. Error analysis identifies long-term visual-semantic consistency as a remaining bottleneck.

  • Quantitative maze results: 99.98% PR: Wan2.2 TI2V 5B reaches near-saturated maze-navigation performance, while GPT-5 series models remain below a 20% success rate.Across 3×3 to 6×6 mazes, all models degrade as grid complexity increases, but Wan2.2 retains near-saturated performance.
  • Quantitative maze results: Maze performance declines systematically as grid size grows from 3×3 to 6×6 for all evaluated models.The degradation appears in both Exact Match and Progress Rate scores.
  • Qualitative maze results: Wan 2.2 TI2V 5B maintains object permanence and high accuracy for seen and unseen icons across in-distribution and out-of-distribution settings.The qualitative results support zero-shot generalization beyond pixel-level memorization.
  • Tangram results: Qwen VL with textual reasoning achieves zero strict goal completion on Tangram Puzzle because it cannot output precise continuous coordinates.The passage attributes the difficulty to describing fine-grained rotation and translation in text.
  • Error analysis: Long-term visual-semantic consistency remains a fundamental challenge: semantic changes cause maze errors and geometric distortions accumulate in tangram videos.The authors call for further research because prior work emphasized primarily local visual changes.

C. Prompt

The prompts define maze navigation as collision-free shortest-path movement and tangram assembly as constrained visual manipulation. Video prompts additionally require smooth, persistent trajectories that preserve maze and piece geometry.

  • Maze task: Maze shortest-path planning moves an agent from its start to a red goal using four cardinal one-cell actions without crossing black walls.Diagonal movement is explicitly prohibited.
  • Maze task: The maze text prompt requires the shortest valid action sequence and wraps the final answer between <ANSWER> and </ANSWER>.The requested output is an ordered sequence such as right up up.
  • Maze test-time scaling: Fine-grained maze scaling varies the unseen icon and increases the inference-frame budget, including a control-variable scaling factor κ.Additional frames are also used to visualize trajectories resembling self-correction.
  • Maze task: The maze video prompt asks for a static top-down animation in which a custom character slides smoothly along white paths to the red destination without entering black walls.The prompt preserves the full-maze camera view and collision-free movement constraints.
  • Tangram task: Tangram prompts require seven colored pieces to fill a black silhouette without changing piece color or shape, overlapping pieces, or extending beyond boundaries.The pieces include two large triangles, one medium triangle, two small triangles, one square, and one parallelogram.
  • Tangram task: The tangram video prompt specifies step-by-step assembly with pieces appearing or moving individually and remaining locked after placement.The task includes fading pieces in and preserving their unique colors and orientations.
  • Maze generalization: Zero-shot maze inference on irregular mazes preserves background and path constraints while sometimes producing diagonal motion.These irregular mazes use different backgrounds and are absent from training data.
  • Tangram scaling: Increasing Tangram inference frames from 61–81 during training to 81–121 at inference does not severely harm performance, unlike the maze scaling trend.The passage suggests geometric consistency constraints may hinder a comparable visual test-time scaling effect.
Loading 2601.21037v1…