PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning with Image Generation

Nishad Singhi, Hector Garcia Rodriguez, Aditya Arora, Marcus Rohrbach, Anna Rohrbach

arXiv:2609.16409v1cs.CV

TL;DR

Text-only reasoning and fixed specialist vision tools are limited for visual tasks requiring flexible spatial or physical transformations. REIMAGIN interleaves textual reasoning with natural-language-directed image generation. Across six tasks, it consistently outperforms text-only reasoning and specialist-tool baselines, with gains up to 25% on path tracing.

  • Problem

    Text-centered multimodal reasoning is ill-suited to tasks requiring spatial or physical intuition, while fixed visual tools cannot flexibly generate or transform visual content.

    Source: §1 · p. 2

  • Method

    REIMAGIN interleaves textual chain-of-thought with natural-language calls to an image-generation model that returns visual intermediates for subsequent reasoning.

    Source: §1 · pp. 2–3

  • Results

    Across six visual reasoning tasks, REIMAGIN consistently improves over text-only reasoning and Visual Sketchpad, with gains up to 25% on path tracing and 40% relative on occlusion counting.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Image generation can serve as a general visual-imagination mechanism that reduces the need to engineer a separate tool for each transformation.

    Source: §6 · p. 10

    In an audit of 120 generated images, 75% were faithful; final answers were correct 87% of the time for faithful images versus 43% for unfaithful ones.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Chain-of-thought reasoning has revolutionized natural language processing by enabling large language models (LLMs) to decompose problems into intermediate steps before answering. Yet confining reasoning to the textual domain presents limitations for tasks requiring direct manipulation of visual representations. Recent efforts augment multimodal LLMs with external visual expert tools such as depth estimation or object detection modules, but these remain fundamentally limited by their reliance on narrow, rigid operations that cannot flexibly generate or transform visual content. We propose ReImaGin, which leverages image generation models as a flexible visual reasoning mechanism for multimodal LLMs: unlike fixed-function tools, they accept natural language commands and can perform open-ended visual operations, like removing an occlusion or generating a floorplan from multiple disjoint views of a room. Across six diverse visual reasoning tasks including multi-view spatial reasoning and collision prediction, ReImaGin consistently outperforms both text-only reasoning and specialist vision-tool baselines, with gains of up to 25\%, demonstrating the advantage of flexible, generative visual reasoning.

1 Introduction

Text-centered multimodal reasoning and fixed-function vision tools are limited for tasks requiring flexible visual manipulation. REIMAGIN uses image generation as an open-ended visual reasoning tool and is evaluated across six tasks.

1 Introduction

  • Text-centric multimodal reasoning describes visual inputs in words, making it ill-suited to tasks requiring spatial or physical intuition, such as occlusion removal or novel-view synthesis.The introduction situates this limitation against the success of Chain-of-Thought reasoning in language and multimodal tasks.Source: §1 · p. 2
  • Fixed specialist tools such as cropping, depth estimation, and object detection perform predefined operations, limiting flexible transformations; handcrafted tool-use examples also require manual effort and limit generalization.The cited examples include generating an alternative view of a scene, which fixed modules cannot perform.Source: §1 · p. 2
  • REIMAGIN interleaves textual chain-of-thought with natural-language calls to a free-form image-generation tool, returning visual intermediates for subsequent reasoning.The same tool can support different transformations, such as drawing a trajectory line for collision prediction.Source: §1 · p. 2
  • Across six tasks, REIMAGIN with handcrafted strategies consistently improves over text-only reasoning and Visual Sketchpad, with gains up to 25% on path tracing and 40% relative on occlusion counting.The same image generation model performs transformations including depth-map generation, occlusion removal, and floorplan synthesis without task-specific specialization.Source: §1 · p. 3
  • Prompt optimization can discover visual reasoning strategies that recover most handcrafted-strategy gains and often resemble them, suggesting MLLMs have useful priors about helpful transformations.The iterative process proposes prompts, evaluates them on a small development set, and refines them using observed successes and failures.Source: §1 · pp. 2–3

2 Related Work

Related work includes fixed specialist tools, image-refinement methods, and generative or latent visual reasoning. REIMAGIN differs in being training-free, modular, and evaluated across diverse tasks in interpretable pixel space.

2 Related Work

  • Fixed specialist vision tools provide narrow, predefined transformations, so agents can only apply manipulations implemented in advance.Examples include cropping, detection, segmentation, and depth estimation.Source: §2 · p. 3
  • Image-refinement methods optimize an image as the final object, while nearby generative or latent reasoning methods often require task-specific training, target a narrow domain, or use non-interpretable latent space.These are distinct from using generated images as intermediates for a downstream question.Source: §2 · p. 3
  • REIMAGIN is training-free and modular, invokes one instruction-following image generator for open-ended transformations in interpretable pixel space, and is evaluated across six visual reasoning tasks.Source: §2 · p. 3

3 REIMAGIN - Reasoning with Image Generation

REIMAGIN places image generation inside an iterative multimodal reasoning loop, alongside programmatic image utilities. Its prompt-based strategy search and image sampling address how to choose and obtain useful visual intermediates.

3 REIMAGIN - Reasoning with Image Generation

  • REIMAGIN combines an instruction-following image generator with programmatic image tools, enabling natural-language-directed visual operations within an iterative reasoning loop.The overview depicts an agent selecting actions turn by turn and passing generated images into the following turn.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Agent Architecture

  • At each turn, the MLLM reasons over context, emits an executable Python action, and can call image utilities or generate_image; outputs, including generated images, return to context until termination.Simple deterministic utilities remain available alongside image generation, and tasks can compose both kinds of tools.Source: §3.1 · p. 4
  • For difficult transformations with variable outputs, generate_image samples N candidate images and an MLLM selects the one that most faithfully satisfies the prompt.The agent still receives a single image; extra test-time compute is spent on image selection rather than multiple reasoning traces.Source: §3.1 · pp. 4–5

3.2 Automated Discovery of Visual Reasoning Strategies

  • Automated strategy discovery iteratively evaluates candidate prompts on training and development data, then proposes refinements using successful and failed training trajectories; only the prompt is optimized.The reasoning agent and image generator remain fixed, and the search uses no human-authored task-specific strategy or reasoning trace.Source: §3.2 · p. 5

4 Experimental Setup

The experiments cover six visual reasoning tasks, three MLLM backbones, and a generative image tool, comparing REIMAGIN with a no-tools baseline and Visual Sketchpad. Strategy discovery uses bounded training and development samples.

4 Experimental Setup

  • The six tasks cover relative depth, puzzle-piece completion, counting hidden objects, predicting collisions, reasoning across overlapping indoor views, and tracing dashed number-to-letter paths.The first five use named benchmark tasks; Path Tracing is introduced in this work.Source: §4 · pp. 5–6
  • Experiments use Gemini-3.1-Pro, GPT-5, or Qwen-3.5-27B as MLLM backbones, with Nano-Banana-Pro as the primary image generator and two open-weight generators also tested.Gemini-3.1-Pro selects among generated images for image-generation test-time scaling.Source: §4 · p. 6
  • The baselines are a vanilla MLLM without tools and Visual Sketchpad with specialist vision modules and programmatic tools but no open-ended generative capability.For MMSI, the baselines receive comparable test-time compute through majority vote over 20 sampled answers.Source: §4 · p. 6
  • All tasks use multiple-choice accuracy except Occlusion Counting, which uses sMAPE; reported metrics average three seeds with standard error across runs.Source: §4 · p. 6
  • Strategy discovery uses at most 50 training and 50 development samples, runs four rounds with five prompt proposals per round, and has an approximately $214 one-time cost per task.MMSI uses fewer samples due to dataset-size limitations.Source: §4 · p. 6

5 Results

REIMAGIN generally outperforms text-only and specialist-tool baselines across the main visual reasoning tasks, and its generative tool contributes to gains across all six tasks. The results also examine test-time scaling, alternative generators, and automatically discovered strategies, including their transfer to another MLLM.

5.1 Main Results

  • Across three MLLMs, REIMAGIN beats No Tools and Visual Sketchpad on most tasks; removing its generative tool degrades all six tasks, with depth on Gemini the stated exception.The table covers six tasks; Occ. Count uses lower-is-better sMAPE, while the others use accuracy.Source: §5.1 · p. 6, Table 1 · p. 7
  • For MMSI spatial reasoning, sampling 10 images raises accuracy from 51.0% to 59.0% on Gemini-3.1-Pro and from 42.0% to 54.3% on Qwen-3.5-27B.Scaling affects generated images while leaving the rest of reasoning unchanged; the other five tasks show little sample variance, so scaling is not applied there.Source: Figure 3 · p. 7, §5.1 · p. 7
  • Open-weight generators are competitive on occlusion counting, but FLUX.2 falls below No Tools on depth and Qwen-Image-Edit trails Nano-Banana-Pro on path tracing.The pattern is qualitatively similar when Qwen-3.5-27B replaces Gemini as the MLLM.Source: §5.1 · p. 7, Table 2 · p. 8
  • An audit found 75% of generated images faithful; answers were correct 87% of the time for faithful images versus 43% otherwise, and reasoning costs $0.04 per instance plus $0.12 for image generation.The reported error types are generator failures and MLLM failures to interpret or reason from a faithful visualization.Source: §5.1 · p. 8

5.2 Automated Discovery of Visual Reasoning Strategies

  • Gemini-discovered prompts transfer unchanged to Qwen-3.5-27B, beating No Tools on all five tasks, but can fall short on occlusion counting and depth.They also outperform Visual Sketchpad and No Strategy on collision, MMSI, and path tracing.Source: Table 4 · p. 9, §5.2 · p. 10
  • Automatic Strategy improves over No Strategy on all five Gemini tasks and on most Qwen tasks, while the discovery results also beat No Tools on five Gemini and four Qwen tasks.The Qwen comparison ties on MMSI.Source: Table 3 · p. 8, §5.2 · p. 9
  • Discovered prompts often recover handcrafted operations, but adding line coloring can hurt path tracing, and the current search does not recover the composite MMSI strategy.A panorama-like room view is discovered for MMSI instead.Source: §5.2 · pp. 9–10, Figure 5 · p. 10

6 Conclusion

REIMAGIN improves over text-only reasoning and specialist-tool baselines across six tasks and multiple model backbones. The conclusion also highlights strategy choice as important and reliability as a next step.

6 Conclusion

  • Across six visual reasoning tasks and multiple model backbones, REIMAGIN consistently improves over text-only reasoning and a strong specialist-tool baseline.Source: §6 · p. 10
  • The benefits of image generation are not tied to one task, transformation, or model, while effectiveness also depends on the strategy for visualization.Source: §6 · p. 10
  • Improving reliability remains a next step, with stronger generators, better visualization selection and verification, and more scalable strategy discovery proposed as avenues.Source: §6 · p. 10

A Extended Related Work

  • Fixed specialist tools perform narrow, predefined transformations, whereas REIMAGIN uses natural-language instructions for a wider range of visual operations.Source: A Extended Related Work · p. 15
  • Image-refinement systems optimize a final image, while REIMAGIN uses generated images as intermediate artifacts to answer downstream questions.Source: A Extended Related Work · p. 15
  • Some prior image-based reasoners target narrow domains, while latent-token approaches require specialized training and produce non-interpretable intermediate steps.Li et al. (2025) and Visual Planning target narrow domains; Yang et al. (2026) and Qin et al. (2025) are examples of specialized training approaches.Source: A Extended Related Work · pp. 15–16

B Benchmark Details

The benchmark suite covers six visual reasoning tasks using established datasets and a newly generated path-tracing set, with the passages specifying each evaluation split or sample size.

Depth Perception; Puzzle Completion

  • Depth perception uses the BLINK validation split with 124 instances, and puzzle completion uses the MIRA test split with 26 instances.Source: B Benchmark Details · p. 16

Occlusion Counting; Collision Prediction

  • Occlusion counting samples 100 CAPTURe instances, while collision prediction samples 100 instances from Spatial457.Source: B Benchmark Details · p. 16

Multi-View Spatial Reasoning; Path Tracing

  • Multi-view spatial reasoning uses 50 instances each from the Obj-Obj and Obj-Reg splits of MMSI-Bench, and path tracing uses 100 programmatically generated instances.Path-tracing lines are dashed, with at least one pair passing close without intersecting.Source: B Benchmark Details · p. 16

C Method Overview

REIMAGIN alternates reasoning and tool use, incorporates tool outputs into the context, and stops when it is ready to answer.

Method Overview

D Additional Test-Time Scaling Results

Additional MMSI scaling results report changes for Automatic Strategy and Gemini No Strategy. An illustrative collision-prediction trajectory shows how an intermediate overlay supports identifying the first object hit.

D Additional Test-Time Scaling Results

  • 56.7±0.7 versus 55.7±1.2 for Gemini Automatic Strategy and 44.8±1.6 versus 44.4±0.5 for Qwen/Flash Automatic Strategy under MMSI test-time scaling; Gemini No Strategy changes from 51.2 to 49.3±0.3.Source: D Additional Test-Time Scaling Results · p. 16

E Illustrative Example

  • In the Spatial457 example, the agent requests a forward vector and first-hit marker; the returned arrow-and-circle image is added to context beside the original scene.Source: E Illustrative Example · p. 17
  • Inspecting the overlay and checking the original scene yields a brown minivan as the first object hit; without the visualization, the no-tools baseline frequently confuses the trajectory.Source: E Illustrative Example · p. 17

F Test-Time Scaling via Repeated Sampling

Repeated image generation is used to improve spatial reasoning, where outputs vary; consistent puzzle and occlusion samples leave little room for test-time scaling.

F Test-Time Scaling via Repeated Sampling

  • Test-time scaling is applied only to spatial reasoning; puzzle and occlusion samples are consistent, leaving little room for improvement.The paper reports sample variability for spatial reasoning but not for the puzzle and occlusion tasks.Source: F Test-Time Scaling via Repeated… · p. 17, Figure 9 · p. 20, Figure 10 · p. 20
  • An image generator independently produces N candidates from a prompt and optional reference images, and an MLLM selector jointly chooses the most faithful output.The selector is Gemini-3.1-Pro in the described setup.Source: Figure 7 · p. 18
  • The spatial-reasoning illustration calls the generator 10 times and marks the selection model’s most prompt-faithful image in green.Source: Figure 8 · p. 19

G Computational Cost

REIMAGIN and Visual Sketchpad use similar MLLM reasoning effort, but image generation adds inference cost and latency; strategy-discovery cost is accounted for separately.

G Computational Cost

  • REIMAGIN adds 1,300 generation tokens and $0.12 per instance, raising median wall-clock time from 52 s to 76 s versus Sketchpad.The comparison uses Gemini-3.1-Pro and Nano-Banana-Pro, averaged over task instances.Source: G Computational Cost · p. 21, Table 5 · p. 21
  • Both methods have an MLLM cost of $0.04 per instance, with matched input tokens (19,700 vs. 21,000), output tokens (350 vs. 400), and MLLM calls (1.79 vs. 1.83).They also produce 1.2 vs. 1.3 intermediate images, respectively.Source: G Computational Cost · p. 21, Table 5 · p. 21
  • The inference accounting excludes the one-time cost of automated strategy discovery, which is reported separately in Appendix L.Source: G Computational Cost · p. 21

H Open-Weights Generators with an Open-Weights MLLM

With Qwen-3.5-27B, the open-weights generator comparison broadly tracks the Gemini setting: Nano-Banana-Pro is strongest on most tasks, while open alternatives struggle on spatially precise operations.

H Open-Weights Generators with an Open-Weights MLLM

I Ablating the Generative Tool

Replacing REIMAGIN’s image generator with a no-op lowers performance across all six tasks; the controlled comparison keeps prompts and scaffolding fixed.

I Ablating the Generative Tool

  • The ablated agent still beats No Tools on several tasks because it retains code execution and programmatic image manipulation, but it does not recover REIMAGIN’s full performance.Source: I Ablating the Generative Tool · p. 23
  • Removing the generative tool lowers performance on all six tasks, with the largest decreases on path tracing (89.0 to 77.5) and puzzle completion (42.3 to 36.5).Source: Table 7 · p. 23, I Ablating the Generative Tool · p. 23
  • The ablation replaces generate_image with an unchanged-input no-op while keeping prompts, tool definitions, Python execution, and programmatic image utilities identical.This isolates generative capability as the changing variable.Source: Table 7 · p. 23, I Ablating the Generative Tool · p. 23

J Faithfulness of the Generated Images

This section examines how ReImaGin prompts image generation to transform visual inputs into intermediate reasoning evidence, and how a selector evaluates candidate images with priority on visual accuracy. It also documents strategy limitations and failures, and reports that more faithful generated images are associated with higher answer accuracy and improved spatial reasoning accuracy when selected.

J Faithfulness of the Generated Images

  • 75% of 120 audited generations were faithful; answers were correct in 87% of faithful-image instances versus 43% of unfaithful-image instances.For MMSI, the audited image was the selector-chosen image that the MLLM actually used. The association is correlational; sampling N = 10 images and selecting the most faithful improved accuracy from 51.0% to 59.0% on Gemini-3.1-Pro and from 42.0% to 54.3% on Qwen-3.5-27B, with textual reasoning unchanged.Source: J Faithfulness of the Generated Images · p. 24

K.1 Failures of Image Generator

  • Generator failures included maps that misplaced scene objects and a trajectory line perpendicular to a motorcycle’s heading.The examples show failures to preserve relative positions in a top-down map and to follow the stated direction when drawing a line.Source: §K.1 · p. 25

K.2 Failures of the MLLM

  • Even when visualizations were correct, the MLLM misidentified a sedan’s collision target and selected the wrong missing puzzle piece.The correct answers were the large object and option D, respectively.Source: §K.2 · p. 26

L Prompt Details for Strategy Discovery

Shared prompt with no task-specific strategy

Meta prompt for automated strategy discovery: system prompt

  • The strategy-discovery system prompt asks the optimizer to improve tool-using prompts, use examples, and design visual intermediates suited to the task.It also directs candidates to use necessary task inputs and generalize to unseen examples.Source: Meta prompt for automated strategy… · p. 28

Meta prompt for automated strategy discovery: user prompt

  • The discovery guidance favors diverse, simple visual operations, worked examples, and checking generated images against the original before relying on them.It also encourages transformations beyond annotation and advises against overloaded multi-goal image edits.Source: Meta prompt for automated strategy… · p. 29
  • $214 per task was the approximate strategy-discovery cost in the typical configuration, with candidate evaluation dominating the total.Qwen independently recovered several strategies, but its arrow-based collision strategy performed worse on development data because the generator often reversed the arrow; the selected prompt instead removes an obscuring object.Source: Meta prompt for automated strategy… · p. 31
  • The meta-prompt’s occlusion-counting strategy removes a black box, completes the hidden pattern, and counts from the generated view.The user prompt also allows movement-path overlays instead of always following the example.Source: Meta prompt for automated strategy… · p. 32

Optimized strategy prompt for path following

Optimized strategy prompt for depth perception

Optimized strategy prompt for multi-image spatial reasoning

Qwen-discovered collision-prediction strategy

  • The discovered collision strategy says not to trust blocked views and to call `generate_image` to remove the occluding object when the path is blocked.It instructs the agent to answer directly when the path is clear and to return a structured JSON response.Source: Qwen-discovered collision-prediction… · p. 37

Qwen-discovered occlusion-counting strategy

  • The discovered occlusion-counting strategy requires generating a complete view of objects hidden by a black box, then counting systematically from that image.It recommends identifying the object pattern first and counting row by row or column by column, including partial objects at boundaries.Source: Qwen-discovered occlusion-counting… · pp. 38–39

M Qualitative Discovered Strategies

  • Discovered strategies often resemble handcrafted ones and improve over text-only reasoning, but examples show cases where handcrafted strategies are more reliable.The examples attribute failures to panoramic views that obscure orientation, arrows that require the generator to infer direction, unhelpful grids for irregular occlusions, and inconsistent colors on distant paths.Source: Figure 22 · p. 42, Figure 23 · p. 43

N.1 REIMAGIN Prompt

  • `generate_image` is described as a generalist tool that creates images from prompts or modifies existing images for tasks such as depth estimation and segmentation.It can take an optional image or list of images as a base or reference.Source: §N.1 · p. 44
  • The depth-map overlay indicates that point B is closer to the camera than point A because B’s color is warmer.The example interprets warmer colors in the generated depth map as indicating closer objects.Source: §N.1 · p. 47
  • The generated segmentation overlay labels the right laptop and shows a lamp to its left, supporting the answer that the lamp is on the laptop’s left.The example uses a segmentation mask overlaid on the original image to inspect object relationships.Source: §N.1 · p. 48
  • A whitespace-removal generation leaves a jagged pattern on the first attempt, so the example retries, noting that outputs can be stochastic.After the retry, the example compares the resulting missing piece with the answer options.Source: §N.1 · p. 48
  • REIMAGIN prompts tailor transformations to each task, from solidifying dashed paths to building a unified 3D view and labeled top-down blueprint with accurate object layout and orientation.For the blueprint workflow, the prompt calls out relevant landmarks and warns against hallucinating details.Source: §N.1 · p. 49

N.2 Test-Time Scaling Selector Prompt

  • The selector judges candidate images by prompt alignment, prioritizing layout, object orientation, and prompt-mentioned placements over labels, style, colors, or professionalism.Source: §N.2 · p. 50

Related papers

Loading 2609.16409v1…