PaperlayerView exact revision on arXiv

Source-linked AI summary

Probing Visual Planning in Image Editing Models

Zhimu Zhou, Yanpeng Zhao, Qiuyu Liao, Bo Zhao, Xiaojian Ma

arXiv:2604.22868v1cs.CVcs.AI

TL;DR

Visual planning is central to spatial reasoning, but editing models’ intrinsic capabilities remain poorly understood. This paper introduces EAR, which casts planning as a single image edit, and AMAZE, a benchmark of abstract Maze and Queen puzzles. Models struggle zero-shot, and despite fine-tuning, the best model remains short of humans’ nearly zero-shot reasoning.

  • Problem

    Although visual planning is central to spatial reasoning, editing-based models’ intrinsic visual-planning capabilities remain insufficiently understood.

    Source: §1 · pp. 1–2

  • Method

    EAR casts visual planning as a single image edit, while AMAZE uses abstract Maze and Queen puzzles to assess intrinsic planning with automatic metrics.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Across the benchmark, editing models struggle zero-shot, fine-tuning simple tasks improves performance, and the best fine-tuned model falls short of humans’ nearly zero-shot reasoning.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The human comparison indicates a persistent gap in neural visual reasoning on abstract planning tasks.

    Source: §5 · p. 10

    Models exhibit task-constraint violations, including invalid maze trajectories and queen placements that break global constraints.

    Source: §3.5 · p. 8

Abstract

from arXiv · show

Visual planning represents a crucial facet of human intelligence, especially in tasks that require complex spatial reasoning and navigation. Yet, in machine learning, this inherently visual problem is often tackled through a verbal-centric lens. While recent research demonstrates the promise of fully visual approaches, they suffer from significant computational inefficiency due to the step-by-step planning-by-generation paradigm. In this work, we present EAR, an editing-as-reasoning paradigm that reformulates visual planning as a single-step image transformation. To isolate intrinsic reasoning from visual recognition, we employ abstract puzzles as probing tasks and introduce AMAZE, a procedurally generated dataset that features the classical Maze and Queen problems, covering distinct, complementary forms of visual planning. The abstract nature of AMAZE also facilitates automatic evaluation of autoregressive and diffusion-based models in terms of both pixel-wise fidelity and logical validity. We assess leading proprietary and open-source editing models. The results show that they all struggle in the zero-shot setting, finetuning on basic scales enables remarkable generalization to larger in-domain scales and out-of-domain scales and geometries. However, our best model that runs on high-end hardware fails to match the zero-shot efficiency of human solvers, highlighting a persistent gap in neural visual reasoning.

1 INTRODUCTION

The paper introduces EAR, which recasts visual planning as a single image edit, and AMAZE, an abstract benchmark for probing editing models’ planning capabilities. Experiments find that fine-tuning improves performance and generalization, but a gap from human efficiency remains.

Contributions

  • EAR reformulates visual planning as an atomic image edit that produces a complete solution in one step, enabling probing of editing models’ intrinsic planning capabilities.The approach avoids explicit step-wise modeling and leverages models’ internalized spatial and visual priors.Source: §1 · pp. 2–3
  • AMAZE provides procedurally generated Maze and Queen puzzles for sequential planning under local constraints and combinatorial planning under global constraints.It separates pixel-wise fidelity from logical validity, varies Queen scales and Maze geometries, and supports probing generalization across scales and geometries.Source: §1 · pp. 2–3

Experiments

  • The experiments assess proprietary and open-source autoregressive and diffusion editing models for generalizability, fine-tuning scale effects, and planning efficiency relative to human solvers.The abstract puzzles isolate intrinsic visual reasoning from complex visual recognition.Source: §1 · pp. 2–3

Results

  • Fine-tuning Bagel on 3 × 3 mazes raises PASS@1 from 0 to 11.54%, 6.14% above the best proprietary model; fine-tuned models generalize to larger scales.Proprietary models are relatively stronger on Maze in the zero-shot setting, while both proprietary and open-source editing models initially struggle.Source: §1 · p. 2
  • After fine-tuning, diffusion-based models surpass autoregressive models on Maze and Queen, yet the best model on one NVIDIA RTX 5090 still trails near-instantaneous, zero-shot human solvers.The authors describe this as an efficiency gap between neural visual planning and human solvers.Source: §1 · pp. 2–3

2 THE AMAZE BENCHMARK

AMAZE uses classical Maze and Queen puzzles to probe visual planning while limiting recognition complexity and enabling automatic assessment of logical correctness. It combines procedural task generation, rule-based validity metrics, pixel-wise fidelity, and human-judge validation.

2 THE AMAZE BENCHMARK

  • AMAZE pairs Maze’s locally constrained sequential planning with Queen’s globally constrained combinatorial planning, using abstract puzzles to reduce visual-recognition complexity.The tasks also support automatic metrics for quantifying logical correctness.Source: §2 · p. 3

2.1 AUTOMATIC DATA CURATION

  • Procedural generation covers Maze scales 3 × 3–16 × 16 across four geometries and Queen scales 4 × 4–10 × 10, yielding 2,800 Maze and 350 Queen test examples.Maze examples span circular, hexagonal, square, and triangular geometries, with 50 mazes per scale and solution algorithm; Queen includes 50 puzzles per scale.Source: §2.1 · p. 3

2.2 AUTOMATIC EVALUATION METRICS

  • Rule-based evaluation defines logical validity through solution-level COVERAGE and VIOLATION ratios, with PASS = max(0, COVERAGE − VIOLATION); PASS = 1 indicates an exact structural match.Pixel-wise fidelity is measured separately inside the goal solution (MSE-IN) and outside it (MSE-OUT).Source: §2.2 · p. 3

2.3 CONSENSUS WITH HUMAN JUDGES

  • 98% agreement between human judges and PASS rates supports the reliability of AMAZE’s automatic logical-validity metric.The remaining 2% discrepancy primarily involved human perception errors on complex tasks or faint solutions and altered non-solution areas; pixel-wise fidelity helps detect these cases.Source: §2.3 · p. 4

3 EXPERIMENT · 3.1 EXPERIMENTAL SETUP · 3.2 MAIN RESULTS

The experiments find weak zero-shot planning, stronger fine-tuned results for Bagel than Janus-Pro, and distinct denoising-stage planning patterns across Maze and Queen.

3.1 EXPERIMENTAL SETUP

  • The proprietary benchmark included GPT-Image-1, NanoBanana-Pro ¹, and Seedream-4.5 (Seedream et al., 2025).The study benchmarks representative models from diffusion-based and autoregressive paradigms to probe proprietary and open-source capabilities.Source: §3.1 · p. 4
  • The open-source lineup comprised Qwen-Image-Edit (Wu et al., 2025a), Flux-Kontext-Dev (Labs et al., 2025), Bagel (Deng et al., 2025), and autoregressive Janus-Pro-7B (Chen et al., 2025).Qwen-Image-Edit, Flux-Kontext-Dev, and Bagel are diffusion-based.Source: §3.1 · p. 4
  • All models received the same concise prompt to draw the solution; each was evaluated five times using average PASS@5, MSE-IN, MSE-OUT, COVERAGE, and VIOLATION, plus first-round PASS@1.The common prompt was used to minimize variation attributable to prompting.Source: §3.1 · p. 4

3.2 MAIN RESULTS

  • Bagel and Janus-Pro struggle zero-shot; fine-tuning used 3 × 3 mazes across four geometries and 4-Queens puzzles, with 800 samples per geometry and 800 puzzles, for up to 8 epochs with validation-loss early stopping.The training set used the simplest scale and included a separate held-out validation set.Source: §3.2 · p. 5
  • On Maze, proprietary models’ best PASS@1 was 5.4%; on Queen, NanoBanana-Pro’s was 30.35%, while other proprietary models were near zero.On Maze, GPT-Image-1 had a 62.88% violation rate, NanoBanana-Pro 47.76%, and Seedream-4.5 stayed below 20% by under-generating incomplete paths. NanoBanana-Pro also performed best in pixel-wise fidelity on Maze.Source: §3.1 · p. 4, §3.2 · p. 5
  • Fine-tuned Bagel (Deng et al., 2025) reached 11.54% Maze and 14.57% Queen PASS@1, versus Janus-Pro (Chen et al., 2025) at 1.43% and 12.57%, respectively.Both models had zero PASS@1 before fine-tuning; the authors suggest diffusion-based modeling may develop visual reasoning logic more effectively, while noting training-data opacity prevents a definitive conclusion.Source: §3.2 · p. 6
  • Chain-of-Thought prompting gives Bagel and Janus-Pro negligible zero-shot benefit but marginal gains after fine-tuning, when they have internalized the task logic.The results are described as mixed overall.Source: §3.2 · p. 6
  • Fine-tuned Bagel’s Maze paths emerge as faint global trajectories early in denoising, are refined and corrected over time, and become valid later; Queen placements undergo global configuration then fine adjustment.The paper notes that global coordination remains challenging, especially for combinatorial Queen puzzles; Figure 3 shows denoising steps for both tasks.Source: Figure 3 · p. 5, §3.2 · p. 6

3.3 GENERALIZABILITY · 3.4 SCALING EFFECT · 3.5 ERROR ANALYSIS

Larger training scales improve generalization, while added data and compute produce task-dependent gains; model failures arise from constraint violations or incomplete solutions.

3.3 GENERALIZABILITY

  • The generalization study fine-tunes Bagel and tests Maze across 3 × 3–16 × 16 scales and Queen across 4 × 4–10 × 10, with 50 samples per scale.Maze testing spans geometry types; the study uses Bagel because it demonstrates non-trivial visual planning capabilities.Source: §3.3 · p. 6

3.3.1 CROSS-GEOMETRY GENERALIZATION

  • Training on 8 × 8 mazes substantially improves generalization across all tested domains compared with 3 × 3 training.The authors interpret this as a shift from learning in-domain geometric patterns toward out-of-domain visual planning.Source: Figure 4 · p. 6, §3.3.1 · p. 7

3.3.2 CROSS-SCALE GENERALIZATION

  • Queen models trained at 4 × 4 achieve perfect in-domain performance but do not generalize to larger scales; 7 × 7 training improves transfer.Maze models show non-trivial cross-scale generalization from 3 × 3 training, unlike Queen models; larger training scales are described as crucial for scale-invariant reasoning.Source: §3.3.2 · p. 7, Figure 5 · p. 7

3.4 SCALING EFFECT

  • 65.2% to 68.1% on 7 as N rises from 800 to 1600, after which Maze performance plateaus; 7-Queens gains +10.3% over that increase.With 1000 training steps fixed, data scaling yields only marginal gains beyond N > 1600; the authors say it mainly improves robustness to scale variation rather than intrinsic sequential planning.Source: §3.4 · pp. 7–8, Figure 6 · p. 7
  • 6.1% over steps 500–700 and 15.8% over steps 700–1000: performance on 7 improves more in the later interval.Training compute is increased from 500 to 1000 steps with 6400 samples fixed; gains are generally marginal over 500–700 steps, with slight drops on Maze at step 800 and Queen at step 700.Source: §3.4 · p. 8, Figure 7 · p. 8

3.5 ERROR ANALYSIS

  • Model failures fall into two modes: constraint violations and incomplete solutions.The paper presents examples from Maze and Queen tasks.Source: §3.5 · p. 8, Figure 8 · p. 8
  • Constraint violations reflect instruction-following deficits: Maze paths may cross boundaries or connect endpoints directly, while Queen placements can break global constraints.These Maze violations are particularly pronounced in complex geometries such as the circle and 7-shaped geometry.Source: §3.5 · p. 8
  • Incomplete solutions are locally valid but globally unfinished: Maze paths stop before the endpoint, and Queen solutions place only a subset of goals.This conservative-generation pattern is particularly pronounced for larger scales or out-of-domain Maze geometries.Source: §3.5 · pp. 8–9

3.6 HUMAN STUDIES

Humans benefit more from additional time than the model, and the model’s similarity to human solvers differs between Maze and Queen tasks.

Human studies

  • The comparison uses Bagel fine-tuned on 8 × 8 7 mazes and 7-Queens against four volunteers each at ages 6, 12, and 18.Each group completes 12 trials per task; puzzles span Maze scales 8 × 8, 16 × 16, and 24 × 24, and Queen scales 4, 7, and 10.Source: §3.6 · p. 9
  • Humans receive unlimited mental-reasoning time but must draw each solution in one continuous attempt; the model receives an equivalent time budget to generate candidates.Total reasoning and drawing latencies are recorded for both.Source: §3.6 · p. 9
  • Within 225 seconds, 18-year-olds achieve a perfect 7-Queens score, while human success improves more with extra time than the model’s relatively flat performance.The human advantage in using additional time is particularly apparent on harder tasks.Source: §3.6 · p. 9
  • The model correlates best with 18-year-olds on Maze and 6-year-olds on Queen, although its performance trend across difficulty levels does not match one age group consistently.The authors suggest that combinatorial planning under Queen’s global constraint may be harder.Source: §3.6 · p. 10

4 RELATED WORK

Prior visual-planning approaches are text-mediated or costly and restricted, while image-editing evaluations often miss logical correctness; this work positions image editing and rule-based abstract-task evaluation as responses.

Spatial reasoning

Image editing models

  • Image-editing models use autoregressive token-based representations or diffusion denoising, and have evolved from standalone editors to unified multimodal systems; this work repurposes strong editors for spatial reasoning.The cited examples include Brooks et al. (2023) for standalone editing and Team (2024), Chen et al. (2025), and Deng et al. (2025) for unified models.Source: §4 · p. 10

Evaluations of image editing models

  • Image-editing evaluations use visual question answering, vision-language judges, or image-text alignment, but often prioritize semantic fidelity over logical correctness (Tong et al., 2024; Yu et al., 2025).The paper addresses this gap with abstract reasoning tasks that avoid perceptual complexity, reliable rule-based automatic metrics, and evaluation of intrinsic visual planning.Source: §4 · p. 10

5 CONCLUSION

EAR recasts visual planning as single-step image editing, and AMAZE benchmarks this approach with abstract Maze and Queen tasks. Existing editing models remain limited: fine-tuning improves performance, but the best fine-tuned model still falls short of human solvers.

5 CONCLUSION

  • Existing editing models remain limited in abstract visual planning; fine-tuning on simple tasks yields remarkable improvements, but the best fine-tuned model still falls short of human solvers.AMAZE removes perceptual complexity to focus evaluation on intrinsic visual planning.Source: §5 · p. 10
  • EAR reformulates visual planning as a single-step image-editing task.Source: §5 · p. 10
  • AMAZE benchmarks visual planning with abstract Maze and Queen tasks covering two complementary planning paradigms.Its lack of perceptual complexity facilitates focused study of intrinsic visual planning and reliable automatic evaluation.Source: §5 · p. 10

A COMPLETE PROMPTS FOR MAZE AND QUEEN TASKS

The appendix specifies prompts for Maze and Queen tasks both without and with Chain-of-Thought (CoT), including a two-stage option for models that do not support joint text-and-image generation.

A COMPLETE PROMPTS FOR MAZE AND QUEEN TASKS

A.1 PROMPTS WITHOUT CHAIN-OF-THOUGHT

  • Without CoT, the Maze prompt asks for a valid entrance-to-exit path while preserving the original maze elements.It specifies a blue path connecting the solid red start circle to the red X exit.Source: §A.1 · p. 14
  • Without CoT, the Queen prompt requires one queen in each row, column, and colored region, with no queens touching in the 8-neighborhood.Source: §A.1 · p. 14

A.2 PROMPTS WITH CHAIN-OF-THOUGHT (COT)

  • CoT-augmented prompts ask models to reason through the planning process before producing the final output, enclosing that process in <think> and </think> tags.Source: §A.2 · p. 14
  • The Maze CoT prompt retains the path-drawing and preservation instructions, then asks the model to plan before output.Source: §A.2 · p. 14
  • The Queen CoT prompt retains the row, column, region, and 8-neighborhood constraints while adding the request to plan before output.Source: §A.2 · p. 14
  • For models without joint text-and-image generation, inference separates text-CoT generation from image generation, a formulation shared by Maze and Queen.The text stage produces reasoning in <think> tags; the image stage outputs only the final image according to that reasoning.Source: §A.2 · pp. 14–15

B SCALING UP TRAINING DATA ON CROSS-DOMAIN PERFORMANCE

The section examines whether training on one maze geometry transfers across geometries and scales. Training topology affects transferability, while denser training data can increase source-geometry specialization.

B SCALING UP TRAINING DATA ON CROSS-DOMAIN PERFORMANCE

  • Models trained on 8×87 mazes and 8×8 ⃝mazes for 500 steps are evaluated across all geometry types and scales from 3 × 3 to 16 × 16.Source: B SCALING UP TRAINING DATA ON… · p. 15
  • Models trained on 7 mazes transfer robustly across tested shapes, while ⃝maze-trained models transfer less well; all perform best in □mazes.The authors attribute the stronger transfer to stable, translation-invariant navigation strategies learned from 7 mazes, versus arbitrary action spaces in ⃝mazes; □mazes may balance action space and topological constraints.Source: B SCALING UP TRAINING DATA ON… · p. 15, Figure 11 · p. 15
  • As training data becomes denser, models increasingly specialize to the source geometry, reducing generalization to structurally different domains.Source: B SCALING UP TRAINING DATA ON… · p. 15

C EXTENDED ANALYSIS OF DATA–COMPUTE SCALING

Data scaling depends on available compute across both tasks. Effective scaling requires enough training data and optimization steps together.

C Extended Analysis of Data–Compute Scaling

D ADDITIONAL ERROR CASES FOR MAZE TASK

Additional maze examples show that error types vary with task conditions: out-of-domain geometries more often produce constraint violations, while larger instances more often yield incomplete solutions.

D ADDITIONAL ERROR CASES FOR MAZE TASK

  • Out-of-domain geometries more often cause constraint violations, while larger mazes more often produce incomplete solutions, reflecting difficulty maintaining local validity and global consistency.Long-range dependencies are required to connect distant regions in larger instances.Source: D ADDITIONAL ERROR CASES FOR MAZE TASK · p. 16

Related papers

Loading 2604.22868v1…