PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Planning: Let's Think Only with Images

Yi Xu, Chengzu Li, Han Zhou, Xingchen Wan, Caiqi Zhang, Anna Korhonen, Ivan Vulić

arXiv:2505.11409v3cs.LGcs.AIcs.CLcs.CV

TL;DR

Text-centered reasoning can be limiting for vision-first spatial and dynamic tasks because visual information is converted into language before planning. The paper proposes Visual Planning, which generates visual state sequences, and VPRL, a GRPO-based reinforcement-learning framework for large vision models. Across visual navigation tasks, the approach improves planning performance and reduces invalid-action failures, while remaining scoped as an early step toward broader multimodal reasoning.

  • Problem

    Text-mediated reasoning may inadequately represent spatial, geometric, and physical relationships, motivating planning directly in visual representations.

  • Method

    Visual Planning generates autoregressive sequences of visual states, while VPRL trains large vision models with reinforcement learning and GRPO.

  • Results

    Across three visual navigation tasks, VPRL achieved 27% EM improvement over language-based planning and reduced invalid-failure ratios by at least 24% versus VPFT.

  • Takeaways & Limitations

    Purely visual representations are presented as a viable alternative to text-based planning for spatially grounded and dynamic tasks.

  • Takeaways & Limitations

    The work is an early step toward systems that integrate verbal and non-verbal reasoning, with future research needed on richer multimodal thinking.

Abstract

from arXiv · show

Recent advancements in Large Language Models (LLMs) and their multimodal extensions (MLLMs) have substantially enhanced machine reasoning across diverse tasks. However, these models predominantly rely on pure text as the medium for both expressing and structuring reasoning, even when visual information is present. In this work, we argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial and geometrical information. Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations for these "vision-first" tasks, as a supplementary channel to language-based reasoning. In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions. We introduce a novel reinforcement learning framework, Visual Planning via Reinforcement Learning (VPRL), empowered by GRPO for post-training large vision models, leading to substantial improvements in planning in a selection of representative visual navigation tasks, FrozenLake, Maze, and MiniBehavior. Our visual planning paradigm outperforms all other planning variants that conduct reasoning in the text-only space. Our results establish Visual Planning as a viable and promising supplement to language-based reasoning, opening new avenues for tasks that benefit from intuitive, image-based inference.

1 INTRODUCTION

Visual Planning addresses the limits of text-mediated reasoning for vision-first spatial tasks by structuring inference as sequences of images. The paper introduces VPRL and reports improved visual planning performance over textual reasoning and supervised baselines.

  • Motivation: Text-based reasoning can create a modality gap by converting visual content into language before inference.This conversion may hinder representation of visual features and state transitions in spatial, geometric, and physical tasks.
  • Motivation: Visual Planning asks whether models can plan directly through images without mediating reasoning through text.The paradigm targets tasks where verbal descriptions may inadequately capture complex spatial relationships.
  • Paradigm: Visual Planning generates step-by-step image sequences that encode planning or inference without language mediation.The approach produces visualizations of intermediate reasoning states rather than textual rationales and answers.
  • Method: VPRL applies reinforcement learning to sequential image generation for visual planning.The framework is introduced as a two-stage training approach for achieving visual planning through image sequences.
  • Results: 27% higher average exact-match rate was achieved than supervised textual reasoning across MAZE, FROZENLAKE, and MINIBEHAVIOR.These grid-based navigation tasks require reaching targets without violating environmental constraints.
  • Results: VPRL significantly outperformed traditional textual reasoning and supervised baselines while exhibiting improved generalization.The empirical evaluation focused on visual spatial planning settings.

2 VISUAL PLANNING VIA REINFORCEMENT LEARNING

The visual planning framework represents plans as autoregressive sequences of visual states and trains a vision model in two stages. VPRL combines visual fine-tuning, exploration, GRPO, and rewards for valid, goal-progressing transitions.

  • 2.1 THE VISUAL PLANNING PARADIGM: Visual planning generates intermediate images autoregressively, conditioning each visual state on the initial state and preceding states.The resulting image sequence forms a visual planning trajectory from the input image.
  • 2.1 THE VISUAL PLANNING PARADIGM: Unlike text-based planning, actions are implicitly represented by transitions between visual states rather than explicitly predicted.This keeps planning within the visual modality as a holistic process.
  • Stage 1: Policy Initialization: Stage 1 initializes the policy on random-walk trajectories to produce valid visual-state sequences and preserve exploration.The stage also serves as a warm-up that improves visual coherence and generation quality.
  • Stage 2: Reinforcement Learning for Visual Planning: Stage 2 uses reinforcement learning to simulate potential next states and optimize planning according to feedback on simulated actions.The initialized exploration capability supports coverage of possible transitions and helps prevent collapse to suboptimal behaviors.
  • Stage 2: Reinforcement Learning for Visual Planning: GRPO scores groups of candidate visual responses using relative advantages instead of a learned value-function critic.This provides computationally efficient and interpretable training signals through within-group comparisons.
  • Reward Design: The reward design evaluates transition validity and progress toward the goal using a dynamics interpreter and progress estimator.Candidates are partitioned into optimal, non-optimal valid, and invalid actions before reward assignment.
  • Reward Design: The experiments assign αopt = 1, αnopt = 0, and αinv = −5, rewarding progress and heavily penalizing invalid transitions.This reward configuration distinguishes goal-directed actions from valid but non-progressing actions and invalid behavior.

3 EXPERIMENTS AND RESULTS

The experiments evaluate visual planning on grid-based navigation tasks using visual-only and language-based systems. VPRL achieves the strongest overall performance, remains robust as task complexity increases, and addresses weaknesses observed in textual planning.

  • Tasks and Models: The evaluation covers FROZENLAKE, MAZE, and MINIBEHAVIOR, where agents navigate visually specified environments while respecting constraints.Synthetic datasets vary in environmental and pattern complexity.
  • Tasks and Models: The study compares visual planners trained with VPFT or VPRL against open-source and proprietary language-based planning systems.The visual planners use an LVM-7B trained only on image sequences and videos, while language baselines include Qwen and Gemini models.
  • Evaluation Metrics: Exact match evaluates whether the predicted trajectory is a shortest optimal solution, accepting any valid optimal trajectory rather than one reference.State equality is based on environment transitions and underlying configurations, not pixel-wise image identity.
  • Evaluation Metrics: Progress Rate measures the longest prefix of consecutive valid moves that aligns with at least one optimal trajectory, providing a softer signal than exact match.The metric uses the same transition-level state equality as exact match.
  • Results: 91.6% EM and 93.2% PR are achieved by VPRL on FROZENLAKE, while it outperforms VPFT by over 20% on average across the evaluated tasks.VPRL's advantage is attributed to outcome-based reinforcement learning that supports exploration and learning from action consequences.
  • Results: 82.4% EM is retained by VPRL on 6 × 6 FROZENLAKE grids, compared with 97.6% on 3 × 3 grids, indicating a flatter degradation curve as difficulty increases.Gemini 2.5 Pro drops from 98.0% to 38.8% over the same grid-size range.

4 DISCUSSIONS AND ANALYSIS

The analysis finds that visual planning handles visual grounding and increasing navigation difficulty more robustly than text-based planning, while VPRL improves exploration and reduces invalid-action failures.

  • Error Analysis and Case Study: 25.7% of coordinate-based descriptions and 22.3% of ASCII representations mismatched ground-truth layouts, exposing a modality gap in textual planning.Visual planning can take obstacle-bypassing detours, whereas VPFT may get stuck.
  • Exploration: VPRL Stage 1 maintains higher exploration entropy and a lower invalid-action ratio than VPFT.This supports random policy initialization before GRPO training.
  • Invalid Action Failure: VPRL reduces the invalid-failure ratio by at least 24% across all three tasks, compared with VPFT’s 61%–78% range.The ratio counts failed trajectories containing at least one invalid action.
  • Scaling Complexity: As FROZENLAKE grids grow, language-based methods decline sharply, whereas visual planning shows a more gradual performance drop.The figure compares performance across increasing grid sizes and difficulty levels.

5 RELATED WORK

Related work covers multimodal reasoning with symbolic grounding or tool-generated visualizations, reinforcement learning for vision, and action-conditional models that predict future observations.

  • Exploration Comparison: VPRL’s exploration comparison evaluates VPFT against VPRL Stage 1 using entropy and invalid-action behavior.VPRL Stage 1 balances high entropy with a low invalid-action ratio.
  • MLLM Reasoning: MLLM reasoning methods ground visual inputs into symbolic structures or generate visualizations through tools.Examples include graphs, bounding boxes, cropping, and zooming.
  • Reinforcement Learning for Visual Reasoning: Reinforcement learning has been applied to vision tasks including perception optimization and visual question answering.GRPO is part of this broader expansion of RL for visual reasoning.
  • Action-conditional Generative Models: Action-conditional generative models predict future observations from actions and support world-model simulation in model-based reinforcement learning.These models learn transition dynamics for simulating outcomes without direct environment interaction.

6 CONCLUSION

The paper presents visual planning as a visual-only alternative for visually oriented reasoning and reports gains from VPRL across three navigation tasks. It also documents licensing and risk checks.

  • Conclusion: Visual Planning enables reasoning through visual state transitions without textual mediation.The conclusion positions it as an alternative to language-based planning for spatially grounded and dynamic tasks.
  • Conclusion: 27% EM improvements over language-based planning were achieved across three visual navigation tasks.The reported gains accompany stronger generalization on out-of-distribution scenarios.
  • Ethics: The study verified software and dataset licenses and reported no additional anticipated potential risks.These checks are stated as part of the paper’s ethical guidelines.

D LIMITATIONS AND FUTURE WORK

The limitations concern the restricted 7B LVM setting, inference overhead from image generation, and a rule-based dynamics interpreter that may not scale to complex visual structures. Future work considers broader models, compact representations, and more robust transition interpretation.

  • Model Scope: Using only a 7B LVM excludes native multimodal models and constrains the evaluated model size.The authors argue that visual planning could extend to multimodal generators supporting image generation.
  • Inference Cost: Image generation adds computational overhead during inference compared with textual responses.The authors suggest compact image representations as a mitigation.
  • Dynamics Interpretation: The rule-based dynamics interpreter is effective in the controlled setup but remains unexplored for broader settings with more complex visual structures.Future alternatives include dynamic models, holistic neural judges, and trajectory-level success feedback.
  • Broader Impact: The broader-impact discussion connects visual-only planning to robotics, navigation, and assistive technologies where perception and decision-making are tightly coupled.It also advocates richer multimodal systems combining verbal and non-verbal reasoning.

E.1 DATASET

The study evaluates visual and textual planners on grid-based navigation tasks, using structured datasets, visual-state transitions, and rule-based rewards that measure progress and validity.

  • Task and dataset setup: FROZENLAKE and MAZE use four navigation actions, while MINIBEHAVIOR adds pick and drop operations.The tasks require reaching targets without violating environment constraints.
  • Task and dataset setup: FROZENLAKE and MAZE contain 3×3 through 6×6 environments, with 1,000 training and 250 testing samples per grid size.Each environment has a unique layout and a reachable goal from the initial state.
  • Training data construction: Visual planning forms training pairs by mapping each state to its subsequent visual state, while textual SFT targets verbalized action sequences.VPFT uses optimal trajectories, and VPFT* mirrors VPRL’s two-stage structure for an ablation.
  • State-action interpretation: VPRL parses visual transitions with grid coordinates, grayscale pixel comparisons, and task-specific rules to identify actions and invalid movements.The interpreter detects navigation transitions, disappearances, and MINIBEHAVIOR pick and drop operations.
  • Reward implementation: Valid transitions receive reward 1 only when they increase heuristic progress, while non-progressing valid transitions receive 0 and invalid actions receive −5.Breadth First Search supplies the progress map used by the heuristic estimator.
  • Generalization: The authors describe the pixel-level analysis and three-way reward structure as extensible to other structured visual planning tasks.The proposed categories are valid and helpful, valid but non-progressing, and invalid actions.

E.4 TRAINING DETAILS

The training setup compares visual and textual planning across supervision and optimization variants, including VPFT, textual SFT, textual RL, and VPRL-related procedures.

  • Visual planning baselines: VPFT trains the visual model to predict the next state along optimal, minimal-step trajectories.Its objective uses the prefix of an optimal trajectory to supervise the next visual state.
  • Textual planning baselines: Textual SFT predicts a verbalized action sequence from the prompt and initial visual state using cross-entropy loss.The input concatenates prompt tokens with visual tokens, while targets contain action tokens.
  • Textual planning baselines: Textual variants prepend coordinate or ASCII descriptions of the environment before the action sequence.The ASCII representation uses symbols including S, G, H, and F for environment elements.
  • Reinforcement learning variants: Textual RL uses GRPO with the VPRL progress reward plus a format reward enforcing <think> and <answer> tags.The comparison tests whether reinforcement learning improves language-based planning with multimodal input.
  • Optimization details: Post-training applies LoRA to attention and feed-forward layers, freezes the image tokenizer and detokenizer, and uses AdamW.SFT runs for up to 30 epochs, while VPRL uses 10 epochs each for exploration and GRPO optimization.

F.1 VPRL TRAINING

The experiments compare visual and textual planning under multiple representations, rewards, and difficulty levels, showing stronger visual robustness while identifying limitations in textual RL reward design.

  • Training analysis: Reward curves with standard deviation are reported for FROZENLAKE, MAZE, and MINIBEHAVIOR, with Gaussian smoothing applied for visualization.The shaded regions represent standard deviation across groups.
  • Textual baselines: Coordinate and ASCII SFT variants provide slight EM gains but lower PR than direct textual SFT, without consistent significant improvement.The comparison covers Qwen-2.5-VL-Instruct-3B and 7B across FROZENLAKE difficulty levels.
  • Textual baselines: 54.4% EM is achieved by the textual RL baseline using the VPRL progress reward on FROZENLAKE.The authors attribute its limited performance to insufficiently discriminative rewards.
  • Textual baseline analysis: Textual models generate erroneous environment descriptions even on simple 3 × 3 inputs, including misclassified cells and incorrect goal descriptions.These errors can lead to incorrect action sequences and are described as a modality gap in grounding visual information into text.
  • Difficulty and robustness: Visual planners maintain higher accuracy and flatter performance curves across MAZE and MINIBEHAVIOR grid sizes.The reported pattern indicates greater robustness as environment complexity increases.
  • Difficulty and robustness: MINIBEHAVIOR visual-planner accuracy increases with grid size, which the authors hypothesize reflects fixed table-and-printer layout components.This fixed layout complexity may allow knowledge from smaller grids to generalize to larger grids.
  • Out-of-distribution evaluation: VPRL achieves visual generalization on unseen larger-grid configurations and consistently outperforms VPFT in Exact Match and Progress Rate.The authors interpret this pattern as evidence that VPRL captures planning strategies beyond memorizing training patterns.

F.6 VISUAL PLANNING RESULTS

VPRL’s second training stage converts visually plausible generations into goal-directed plans, while qualitative analyses show robustness to occlusion alongside residual planning errors.

  • VPRL training stages: Stage 1 produces plausible images but near-random task performance, whereas Stage 2 enables goal-directed planning and outperforms VPFT across all benchmarks.The two stages are compared across FROZENLAKE, MAZE, and MINIBEHAVIOR.
  • Robustness analysis: Under masked inputs, VPRL preserves coherent planning traces aligned with the visible environment structure.Black or gray patches simulate partial occlusion of the input environment.
  • Generated visual traces: VPRL trajectories begin with the initial state and continue through generated intermediate visual states forming the predicted plan.The qualitative examples cover FROZENLAKE, MAZE, and MINIBEHAVIOR.
  • Generated visual traces: The visual traces include optimal, non-optimal, and invalid cases, distinguishing shortest valid paths from inefficient or constraint-violating trajectories.Non-optimal examples may contain loops, while invalid examples prevent task completion.
  • Limitations: VPRL still exhibits occasional planning errors, and reinforcement learning does not fully eliminate failure cases.The reported categories include redundant or non-optimal actions and invalid environment-violating transitions.
  • Visual analysis: The qualitative comparison includes original images, model predictions, and reconstructions obtained by encoding and decoding the original inputs.

F.7 IMAGE QUALITY ANALYSIS

The visual planner’s intermediate images contain artifacts attributed to the image tokenizer, but these images remain sufficient for effective planning and robust to visual noise.

  • Tokenizer reconstruction introduces artifacts that make decoded images differ from the original ground-truth images.Re-encoding and decoding ground-truth images produces similar artifacts to the model’s predictions.
  • The model’s intermediate images are comparable in quality to images reconstructed by the tokenizer.
  • The dynamics interpreter penalizes images it cannot parse, implicitly constraining generated transitions to preserve environmental semantics.
  • High-quality environment-rendered intermediate images produce similar performance to self-generated images across all grid sizes.This robustness study replaces each self-generated intermediate image with the corresponding ground-truth rendering.

F.8 COMPUTATIONAL COST ANALYSIS

The computational analysis compares visual planning with textual reasoning across FrozenLake, Maze, and MiniBehavior, including trained textual planner variants on FrozenLake. Visual planning costs more tokens because it repeatedly generates images, but the reported overhead remains affordable relative to textual chain-of-thought.

  • The analysis measures average inference token cost across FrozenLake, Maze, and MiniBehavior, plus trained textual planner variants on FrozenLake.Table 12 reports cross-task costs, while Table 13 focuses on trained textual planners for FrozenLake.
  • Visual planning introduces computational overhead because it repeatedly generates images during inference.
  • 3 times the token cost of Qwen 2.5-VL-Instruct-7B with CoT and around 6 times that of Gemini 2.0 Flash with CoT are the visual planner’s average cross-task costs.The comparison averages token usage across the three evaluated tasks.
  • FrozenLake: FrozenLake requires shortest valid four-direction paths that avoid impassable ice holes.
  • Maze: Maze requires shortest valid cardinal-direction paths from the start to the goal without crossing black walls.
  • MiniBehavior: MiniBehavior requires picking up a printer and placing it on a table under movement, adjacency, and obstacle constraints.
Loading 2505.11409v3…