Source-linked AI summary
Probing Visual Planning in Image Editing Models
Zhimu Zhou, Yanpeng Zhao, Qiuyu Liao, Bo Zhao, Xiaojian Ma
TL;DR
Visual planning is often handled verbally or through inefficient step-wise image generation, leaving intrinsic planning ability in editing models insufficiently understood. The paper introduces EAR and the AMAZE benchmark to probe this ability through single-step image editing on abstract Maze and Queen tasks. Models improve substantially with fine-tuning and generalize across scales and geometries, but the best model still trails human solvers in efficiency.
Problem
Editing-based models lack a comprehensive evaluation of intrinsic visual planning, while existing step-wise visual approaches are computationally inefficient.
Method
EAR reformulates visual planning as a single-step image-editing task, evaluated on procedurally generated abstract Maze and Queen puzzles in AMAZE.
Results
Fine-tuning improves visual planning and generalization across scales and geometries, but editing models initially struggle zero-shot and the best model remains less efficient than human solvers.
Takeaways & Limitations
AMAZE exposes progress in neural visual planning while showing that current editing architectures remain limited relative to human spatial reasoning efficiency.
Takeaways & Limitations
Constraint violations increase for out-of-domain geometries, while incomplete solutions become more prevalent at larger scales requiring long-range dependencies.
Abstract
from arXiv · showhide
Visual planning represents a crucial facet of human intelligence, especially in tasks that require complex spatial reasoning and navigation. Yet, in machine learning, this inherently visual problem is often tackled through a verbal-centric lens. While recent research demonstrates the promise of fully visual approaches, they suffer from significant computational inefficiency due to the step-by-step planning-by-generation paradigm. In this work, we present EAR, an editing-as-reasoning paradigm that reformulates visual planning as a single-step image transformation. To isolate intrinsic reasoning from visual recognition, we employ abstract puzzles as probing tasks and introduce AMAZE, a procedurally generated dataset that features the classical Maze and Queen problems, covering distinct, complementary forms of visual planning. The abstract nature of AMAZE also facilitates automatic evaluation of autoregressive and diffusion-based models in terms of both pixel-wise fidelity and logical validity. We assess leading proprietary and open-source editing models. The results show that they all struggle in the zero-shot setting, finetuning on basic scales enables remarkable generalization to larger in-domain scales and out-of-domain scales and geometries. However, our best model that runs on high-end hardware fails to match the zero-shot efficiency of human solvers, highlighting a persistent gap in neural visual reasoning.
1 INTRODUCTION
The paper introduces EAR, which reframes visual planning as a single image edit, and AMAZE, a controlled benchmark for probing intrinsic visual reasoning. Evaluations examine generalization, scaling, and efficiency, finding that models improve with fine-tuning but remain less efficient than humans.
- Motivation and approach: EAR reformulates visual planning as a fully visual, single-step image-editing task rather than explicit step-wise generation.The atomic edit is intended to probe intrinsic spatial and visual reasoning while avoiding the computational inefficiency of step-wise planning.
- Benchmark and evaluation: AMAZE uses procedurally generated Maze and Queen puzzles to test sequential planning under local constraints and combinatorial planning under global constraints.Its abstract tasks reduce visual-recognition confounds while varying scales and Maze geometries to probe generalization and geometric invariance.
- Experimental scope: The experiments assess proprietary and open-source autoregressive and diffusion-based editing models across generalizability, scaling effect, and human-comparison dimensions.These dimensions cover transfer to unseen scales and geometries, relationships between performance and training resources, and efficiency relative to human solvers.
- Findings: Both proprietary and open-source models initially struggle with zero-shot planning, while fine-tuning improves performance and generalization to larger scales.Fine-tuning Bagel on basic 3 × 3 mazes improves PASS@1 from 0 to 11.54%, exceeding the best proprietary model by 6.14 percentage points.
- Findings: Diffusion-based models surpass autoregressive models on both Maze and Queen after fine-tuning, although the best model remains less efficient than near-instantaneous zero-shot human solvers.The reported human comparison identifies a persistent efficiency gap despite the gains from fine-tuning.
2 THE AMAZE BENCHMARK
AMAZE is an abstract, procedurally generated benchmark designed to isolate visual planning and evaluate both logical correctness and pixel-level fidelity. Its automatic metrics are validated against human judgments, with 98% agreement reported.
- Benchmark design: AMAZE comprises procedurally generated Maze and Queen puzzles spanning locally constrained sequential planning and globally constrained combinatorial planning.Maze scales from 3 × 3 to 16 × 16 and varies circular, hexagonal, square, and triangular geometries; Queen scales from 4 × 4 to 10 × 10.
- Benchmark design: The benchmark minimizes visual-recognition complexity so evaluations can focus on intrinsic visual planning rather than multimodal understanding.Its abstract structures support controlled analysis of planning ability across task scales and geometries.
- Automatic metrics: Logical validity measures whether generated solutions match the goal at the cell level using COVERAGE, VIOLATION, and PASS.PASS = max(0, COVERAGE − VIOLATION), and PASS = 1 indicates an exact match to the solution structure.
- Automatic metrics: Pixel-wise fidelity is measured with mean squared error separately inside and outside the solution area.The resulting metrics are MSE-IN for solution cells and MSE-OUT for non-solution cells.
- Metric validation: 98% agreement between PASS rate and binary human judgments supports the reliability of the logical-validity metric.Three annotators evaluated 50 randomly sampled images per task for each model.
3 EXPERIMENT
The experiment benchmarks proprietary and open-source editing models on abstract visual-planning tasks, comparing paradigms, prompting, and qualitative denoising behavior. Results expose weak zero-shot planning, stronger fine-tuned diffusion performance, mixed CoT effects, and distinct Maze-versus-Queen planning dynamics.
- Experimental Setup: The benchmark covers proprietary and open-source editing models, including diffusion-based and autoregressive systems, evaluated with concise shared prompts and multiple visual and logical measures.Models are evaluated five times using PASS@5, MSE-IN, MSE-OUT, COVERAGE, and VIOLATION, with PASS@1 reported from the first generation.
- Main Results: NanoBanana-Pro reaches the best proprietary PASS@1 of 5.4% on Maze and 30.35% on Queen, while other proprietary models nearly fail on Queen.Proprietary models also show constraint failures: GPT-Image-1 has a 62.88% violation rate, whereas NanoBanana-Pro over-generates paths with 47.76% violation.
- Main Results: After fine-tuning, Bagel improves Maze PASS@1 from 0 to 11.54%, while Janus-Pro reaches 1.43%; on Queen, Bagel reaches 14.57% versus 12.57%.The authors qualify this comparison because differences in pretraining transparency prevent a definitive conclusion, but the findings suggest an advantage for diffusion-based modeling.
- Main Results: CoT provides negligible zero-shot benefits for Bagel and Janus-Pro but yields marginal improvements after fine-tuning.The results suggest that models first need to internalize the task logic before effectively using intermediate reasoning steps.
- Main Results: On Maze, fine-tuned Bagel forms a faint global path early and progressively corrects subpaths into a valid solution through later denoising steps.The observed trajectory is coarse-to-fine, with the overall structure emerging at t = 1, 2, and 4 and later refinement at t = 8 and t = 10.
- Main Results: On Queen, Bagel first establishes a coarse global placement configuration and then makes fine adjustments, making global coordination especially challenging.The contrast indicates that sequential Maze planning permits local refinement, whereas combinatorial Queen planning requires substantial global updates.
3.3 GENERALIZABILITY
Fine-tuned editing models generalize across maze geometries and scales, but the difficulty of transfer depends on task structure and training scale. Scaling data and compute yields task-dependent gains, while failures remain constraint violations or incomplete solutions.
- Evaluation setup: Fine-tuned Bagel was evaluated on Maze scales from 3 × 3 to 16 × 16 and Queen scales from 4 × 4 to 10 × 10.Maze evaluation also varied geometry types, with 50 samples per scale for each geometry; Queen used 50 samples per scale.
- Cross-geometry generalization: Increasing Maze training from 3 × 3 to 8 × 8 produced a substantial leap in generalization across all test domains.Larger-scale training shifts performance beyond in-domain geometric patterns toward out-of-domain visual planning capabilities.
- Cross-scale generalization: Queen training at 4 × 4 achieved perfect in-domain performance but no larger-scale generalization, whereas 7 × 7 training yielded better non-trivial cross-scale transfer.Unlike Maze, Queen requires more complex training scales for non-trivial generalization because it involves combinatorial planning under global constraints.
- Scaling effect: 65.2% to 68.1%: Maze performance on one geometry improved when data increased from 800 to 1600 samples, then plateaued.Across tasks, data-scaling gains became marginal after N > 1600 under a fixed 1000-step compute budget.
- Scaling effect: +10.3%: Increasing data from 800 to 1600 samples produced a larger initial gain on 7-Queens than on Maze.The paper attributes this pattern to the diversity of solution patterns in combinatorial tasks.
- Failure analysis: Model failures fall into constraint violations and incomplete solutions, including invalid trajectories, erroneous Queen placements, and partial outputs.Incomplete solutions are particularly pronounced at larger scales or on out-of-domain geometries.
3.6 HUMAN STUDIES
The human study compares Bagel with three age groups across controlled Maze and Queen difficulties and matched single-attempt conditions. Human success rises with additional time, whereas model performance remains relatively flat.
- Study design: Four participants from each of the 6-, 12-, and 18-year-old groups completed Maze and Queen trials across easy, moderate, and hard scales.Each age group completed 12 trials per task, covering Maze scales 8 × 8, 16 × 16, and 24 × 24 and Queen scales 4, 7, and 10.
- Study design: Participants had unlimited reasoning time but completed each drawing in one continuous attempt without erasing, backtracking, or restarting.The study recorded reasoning and drawing latencies and allocated the model an equivalent time budget.
- Results: Human success rates correlate more positively with permitted time than model success rates.Humans improve with additional time, particularly on harder tasks, while model performance remains relatively flat.
- Results: 7.5 seconds: one model drawing took Bagel about 7.5 seconds on average across 20 runs.This timing provides the model-side inference duration used in the comparison context.
- Results: Bagel’s visual-planning ability resembles the 18-year-old group on Maze and the 6-year-old group on Queen, but its performance trend does not generally match either age group.Pearson correlations were estimated between model and each human group for each task.
4 RELATED WORK
Prior work approaches spatial reasoning through textual or multimodal verbal reasoning, while image-editing models use autoregressive or diffusion-based generation. Existing editing evaluations often emphasize semantic fidelity over logical correctness.
- Spatial reasoning: Spatial-reasoning methods commonly translate visual problems into text or use vision-language models with text-based chain-of-thought.Other work explores direct-generation and fully visual approaches, but intrinsic visual planning in editing models remains incompletely understood.
- Image editing models: Image-editing models generally fall into autoregressive token-based systems and diffusion-based systems that iteratively refine the image manifold.These paradigms represent the two dominant generative approaches considered in the paper’s model evaluation.
- Evaluation: Image-editing evaluations use visual question-answering checks, vision-language judges, and image-text alignment scores.These approaches often prioritize semantic fidelity or consistency rather than logical correctness.
5 CONCLUSION
The paper introduces EAR and AMAZE to study intrinsic visual planning in image-editing models using abstract Maze and Queen tasks. Fine-tuning substantially improves performance, but the best model remains less efficient than human solvers.
- Conclusion: EAR reformulates visual planning as a single-step image-editing task, while AMAZE provides abstract Maze and Queen benchmarks for complementary planning paradigms.AMAZE removes perceptual complexity and supports automatic evaluation of intrinsic visual planning.
- Conclusion: Existing editing models remain limited in abstract visual planning despite remarkable improvements from supervised fine-tuning on simple tasks.The conclusion frames these findings as empirical evidence about current editing-model capabilities.
- Conclusion: The best fine-tuned model still falls short of humans’ instantaneous, nearly zero-shot visual planning.This marks a remaining gap between neural visual reasoning and human solver efficiency.
A COMPLETE PROMPTS FOR MAZE AND QUEEN TASKS
The appendix specifies prompts for Maze and Queen visual-planning tasks, with and without chain-of-thought, and describes a two-stage formulation for models lacking joint text-and-image generation.
- A.1 PROMPTS WITHOUT CHAIN-OF-THOUGHT: Maze without CoT asks the model to generate a valid path from entrance to exit while obeying geometric constraints.
- A.1 PROMPTS WITHOUT CHAIN-OF-THOUGHT: The Maze prompt instructs the model to add a blue path connecting the red start circle to the red-X endpoint while preserving original maze elements.
- A.1 PROMPTS WITHOUT CHAIN-OF-THOUGHT: Queen without CoT asks the model to place queens without conflicts across rows, columns, and colored regions.
- A.2 PROMPTS WITH CHAIN-OF-THOUGHT (COT): CoT-augmented prompts encourage intermediate reasoning before the model produces the final output.
- A.2 PROMPTS WITH CHAIN-OF-THOUGHT (COT): Maze CoT prompts require the planning process to be enclosed within <think> and </think> tags before output.
- A.2 PROMPTS WITH CHAIN-OF-THOUGHT (COT): Queen CoT uses the same intermediate-reasoning convention while retaining row, column, region, and 8-neighborhood constraints.
- A.2 PROMPTS WITH CHAIN-OF-THOUGHT (COT): Models without native joint text-and-image generation use two inference stages: text generation followed by image generation for both tasks.
- A.2 PROMPTS WITH CHAIN-OF-THOUGHT (COT): For image generation, the model outputs only the final image, with an ellipsis standing for text-generation reasoning.
B SCALING UP TRAINING DATA ON CROSS-DOMAIN PERFORMANCE
Cross-domain transfer depends strongly on training geometry, while increasing training data can reduce transferability through geometry-specific overfitting.
- Training geometry critically determines cross-domain transferability across geometry types and scales.The study trains on single geometries and evaluates all geometry types from 3 × 3 to 16 × 16.
- Models trained on 7 mazes transfer robustly across tested shapes, whereas models trained on circle mazes transfer more weakly.The reported explanation contrasts translation-invariant navigation strategies with arbitrary action spaces.
- Larger training datasets can decrease cross-domain performance by encouraging specialization to the source geometry.
C EXTENDED ANALYSIS OF DATA–COMPUTE SCALING
Data scaling and compute interact strongly in visual planning: useful scaling requires sufficient optimization capacity, and the relationship differs between Maze and Queen tasks.
- Training-data effects on performance depend strongly on the available compute budget.The analysis uses the same training and evaluation settings as the preceding scaling study.
- For Maze, more training steps consistently improve performance, while data scaling helps at N ≤3200 but often diminishes beyond that.
- For Queen, higher-step models benefit more consistently from larger datasets, whereas low-step models show unstable scaling.
- Effective scaling requires a balanced regime in which both training data and optimization steps are sufficiently large.
- The reported bottleneck is jointly constrained by optimization capacity and the ability to absorb the training distribution.
D ADDITIONAL ERROR CASES FOR MAZE TASK
Maze failures vary with geometry and scale: out-of-domain action spaces increase constraint violations, while larger instances increase incomplete solutions.
- Constraint violations are more frequent when test geometries differ from the training distribution.
- Incomplete solutions are more prevalent in larger mazes requiring long-range dependencies to connect distant regions.