PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluation

Shangzhe Di, Zhaokai Wang, Weidi Xie

arXiv:2609.07884v1cs.CV

TL;DR

Whether mainstream image generators can perform visual perception on public benchmarks without task-specific adaptation remains an open question. This paper introduces PROBEGEN, evaluating prompt-conditioned generation for depth, segmentation, and counting against specialist models. Generators show a trade-off: specialists retain in-distribution accuracy and efficiency advantages, while generators are more robust under distribution shift and compositional reasoning.

  • Problem

    The zero-shot perceptual ability of off-the-shelf image generators and their competitiveness against specialist models on public benchmarks remain unclear.

    Source: §1 · pp. 1–2

  • Method

    PROBEGEN evaluates prompt-conditioned image generation for depth, referring and reasoning segmentation, and counting, using deterministic extractors to produce benchmark-compatible predictions.

    Source: §1 · p. 3, §3 · p. 4, §3.1 · p. 4

  • Results

    Specialists remain stronger in in-distribution accuracy and efficiency, while generators show greater robustness under distribution shift and compositional complexity.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    General image generators can support multiple structured perception tasks through a single interface, with potential for open-world concepts, long-tail queries, and natural-language interaction.

    Source: §4.7 · p. 14, §5 · p. 14

    The prompt-only protocol cannot rule out prior task exposure: proprietary training data are undisclosed, and Qwen-Edit-2511’s depth scores reflect known depth-task examples in its instruction-tuning data.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

Recent work, such as Vision Banana, shows that lightweight instruction tuning can enable an image generator to achieve state-of-the-art performance across multiple visual perception tasks. Motivated by this perspective, we ask how far image generators can go on public visual perception benchmarks in a zero-shot setting. We introduce ProbeGen, a benchmark for zero-shot generative perception that casts monocular depth estimation, referring/reasoning segmentation, and object counting as conditional generation tasks specified through text prompts, and compares 20 models in total---including proprietary and open-weight image generators, specialist perception models, and MLLMs---across 11 published benchmarks. We observe that pretrained image generators show measurable zero-shot perceptual competence, but with a clear trade-off: specialist models remain stronger for in-distribution accuracy and efficiency, while generative models are often more robust under distribution shift and better at compositional semantic reasoning. We hope this study helps establish zero-shot generative perception as a meaningful research direction and provides a useful foundation for future work at the intersection of visual generation and understanding.

1 Introduction

This study asks whether general-purpose image generators can serve as zero-shot perceivers on public benchmarks, rather than only after task-specific adaptation. PROBEGEN compares generators with specialist models and finds a trade-off between in-distribution performance and robustness under distribution shift and compositional complexity.

1 Introduction

  • Whether mainstream image generators can compete with state-of-the-art specialists as zero-shot perceivers on public benchmarks remains an open question.Prior work showed that instruction-tuned generators can serve as generalist vision learners, while diffusion dense predictors often require task-specific fine-tuning or architectural changes.Source: §1 · pp. 1–2
  • PROBEGEN evaluates mainstream image generators against specialist models across depth estimation, referring and reasoning segmentation, and object counting using a shared prompt-based protocol.Deterministic extractors convert generator outputs into benchmark-comparable predictions.Source: §1 · pp. 2–3
  • Specialists retain advantages in in-distribution accuracy and efficiency, while generators show greater robustness under distribution shift and stronger compositional reasoning on complex queries.The authors identify input-preservation and instruction-following failures as characteristic generative error modes.Source: §1 · pp. 2–3
  • The study characterizes input-preservation and instruction-following failures as systematic error modes that distinguish generative perception from conventional discriminative models.The authors present these diagnoses as concrete avenues for improvement.Source: §1 · p. 3

2 Related Work

Prior perception research centers on specialized discriminative models and on generative models adapted into dense predictors. Evidence remains limited on whether off-the-shelf image generators can perform competitively without adaptation.

Discriminative specialists for visual perception

  • Specialist architectures define strong in-distribution accuracy and efficiency baselines across depth, segmentation, localization, and counting.Examples include Depth Anything, Segment Anything, and specialist counting models.Source: §2 · p. 3

Adapting image generators into perception specialists

  • Marigold and Lotus-2 demonstrate that diffusion models can become strong dense predictors after task-specific training, architectural modification, or specialized inference design.This work studies adapting generators rather than their pretrained, out-of-the-box perception ability.Source: §2 · p. 3

Evaluation gaps in zero-shot generative perception

  • Prior zero-shot studies do not establish how competitive image generators are against specialists on public benchmarks.Video-generator work omits specialist comparisons, while Vision Banana reports instruction-tuned—not zero-shot—performance.Source: §2 · pp. 3–4

3 Zero-Shot Generative Perception

PROBEGEN evaluates pretrained image generators as zero-shot perceivers through text-conditioned image generation followed by deterministic extraction into benchmark-compatible predictions. Its protocol specifies prompts and task-specific extraction for segmentation, depth, and counting.

3 Zero-Shot Generative Perception

  • The protocol operationally defines zero-shot as prompt-only use without task-specific fine-tuning, adaptation, or architectural change, while acknowledging possible pretraining exposure.A pretrained generator produces an RGB image encoding the requested perceptual quantity, which a fixed deterministic extractor maps into benchmark-compatible output.Source: §3 · p. 4

3.1 Prompt Protocol

  • Prompts request black-background, white-target masks for segmentation, brighter-is-closer grayscale depth maps, or red dots on each counted object.Prompt specificity varies by task, and the mid-level prompts are the default except in ablations.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.2 Segmentation Output Extraction

  • Otsu thresholding is the most robust of four tested segmentation binarization strategies and is used as the default across the main segmentation experiments.It consistently suppresses soft-boundary artifacts across six segmentation benchmarks.Source: §3.2 · pp. 5–6

3.3 Monocular Depth Extraction

  • Depth maps are evaluated after least-squares scale-and-shift alignment, which removes arbitrary linear ambiguity while preserving generated structural and ordinal information.The protocol uses affine-invariant evaluation because generators typically output relative rather than metrically calibrated depth.Source: §3.3 · p. 6

3.4 Object Counting Extraction

  • Counting prompts ask generators to mark each target with a red dot, then a fixed decoder isolates changed red pixels and counts shape-filtered blobs.The decoder applies color isolation, an input-difference gate, and area, circularity, and inertia-ratio constraints.Source: §3.4 · p. 6

Experimental Results

Across segmentation, depth, and counting, zero-shot generators show useful perception but do not displace specialists’ accuracy and efficiency advantages. Their strongest results appear in reasoning and out-of-distribution segmentation, while failures often reflect weak input preservation or output-format discipline.

4.1 Setup

  • The evaluation covers 11 benchmarks across depth, referring/reasoning segmentation, and counting, comparing specialists, general image generators, and MLLMs under distinct inference protocols.The setup samples 500 examples per benchmark; API-hosted model efficiency is excluded from the main tables, and Vision Banana numbers use a different protocol.Source: §4.1 · p. 7

4.2 Referring and Reasoning Segmentation

  • Nano Banana 2 reaches 70.3 gIoU on ReasonSeg, above the SAM3 + Qwen3-VL-8B-Thinking pipeline’s 60.7 gIoU; Nano Banana models also outperform specialists on DRAM and Manga109.These are reasoning and out-of-distribution segmentation results; Table 1 reports gIoU and cIoU.Source: §4.2 · pp. 7–8, Table 1 · p. 8
  • On standard referring segmentation, Nano Banana Pro reaches 70.7 gIoU on RefCOCO and 72.0 gIoU on RefCOCOg, while specialist systems retain the peak overall performance.Across general generators, Nano Banana models are consistently strong; Seedream-4.5 fails to produce usable masks, and Flux.2-dev and Qwen-Edit-2511 are weaker, especially on reasoning segmentation.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Monocular Depth Estimation

  • Depth specialists remain stronger in accuracy and efficiency: Depth Anything V2-Large runs at 828 images per minute, while Qwen-Edit-2511’s strong scores reflect known depth-task exposure.Qwen-Edit-2511 records 5.6 AbsRel on NYUv2, 17.1 on DIODE, 6.5 on ScanNet, and 8.9 on KITTI; its technical report says instruction-tuning data includes depth examples, so these scores are a near-supervised reference rather than evidence of emergent geometry.Source: §4.3 · pp. 9–10, Table 3 · p. 9

4.4 Object Counting

  • On PixMo-Points, Gemini-3.5-Flash leads MLLMs at MAE 4.4, while Nano Banana 2 leads generators at MAE 7.3 / RMSE 15.5; generators have the best RMSE range at 15.5–15.6.Integer-emitting MLLMs lead in MAE, but their best RMSE of 19.9 is worse than the best generators; marking is more visually interpretable and more robust in worst-case error, but red clutter and merged dots limit it.Source: §4.4 · pp. 10–11, Table 4 · p. 10

4.5 Ablation Studies

  • Otsu thresholding gives 68.8 average gIoU across six segmentation benchmarks, while prompt ablations show weaker generators gain most when the mid prompt specifies output format.For Nano Banana 2, RefCOCOg scores are 77.0/76.7/75.7 gIoU across prompt variants; weaker models’ short-to-mid improvements often outweigh mid-to-long changes.Source: §4.5 · p. 11, Table 5 · p. 12, Table 6 · p. 12

4.6 Failure Modes

  • Systematic failures arise when generators regenerate rather than preserve the input or ignore the requested format; evaluated unified understanding-and-generation models also fail across tasks and prompt variants.None of BAGEL, OmniGen2, Lance, or UNO produced a usable mask, depth map, or dot annotation; zero-shot surface-normal outputs could look plausible but were inaccurate and unreliable.Source: §4.6 · pp. 12–13, Figure 3 · p. 13, Figure 4 · p. 14

4.7 Discussion

  • The discussion identifies semantic grounding, spatial faithfulness, and output-format discipline as necessary ingredients; generators remain one to three orders of magnitude slower than specialists and use more GPU memory.Spatial errors may be absorbed by global scale-and-shift alignment for relative depth, but local misalignment in masks, normals, and counting cannot be undone by deterministic extraction; the authors suggest spatial-consistency objectives and reward signals.Source: §4.7 · pp. 13–14

5 Conclusion

The paper evaluates general-purpose generative models as zero-shot visual perceivers and finds a trade-off between specialist strength in in-distribution accuracy and efficiency and generative-model competitiveness on OOD and reasoning-heavy tasks.

5 Conclusion

  • Specialized models remain stronger in in-distribution accuracy and efficiency, while zero-shot generative prompting is more competitive on OOD scenarios and reasoning-heavy tasks.The paper also notes persistent challenges in alignment, output fidelity, and speed.Source: §5 · p. 14
  • Text prompting repurposes image synthesis for dense perception, and the evaluated models show non-trivial geometric and semantic understanding without task-specific fine-tuning.The paper will release PROBEGEN and its prompt templates.Source: §5 · p. 14
  • The authors present these findings as a path toward unified architectures that bridge synthesis and perception.Source: §5 · p. 14

A Counting Decoder Details

The supplement expands on the deterministic decoder that maps a generator’s marked image to an integer count, covering decoder selection, leakage-safe hyperparameter setting, and residual errors.

A Counting Decoder Details

A.1 Decoder Selection

The decoder comparison selects the shape-filtered blob detector with a diff gate, which achieves the lowest reported average MAE across the evaluated generators.

A.1 Decoder Selection

  • 5.07 average MAE: the shape-filtered SimpleBlobDetector combined with the diff gate dominates the alternatives and is adopted as the deployed decoder.The detector constrains candidates by area, circularity, and inertia ratio.Source: §A.1 · p. 18
  • MAE in the 30–60 range: connected-components decoders without a shape model systematically overcount elongated red smears and thin red contours as dots.Source: §A.1 · p. 18
  • 5.50 MAE for Nano Banana 2 versus 10.90 MAE for GPT-Image-2: the diff-vs-input gate alone reduces overcounting more for the former, while scene re-rendering weakens its signal for the latter.Source: §A.1 · p. 18

A.2 Leakage-Safe Hyperparameter Sweep

The leakage-safe sweep uses source-image grouping to keep correlated generations together, then selects robust directional parameter changes rather than the literal validation argmin.

A.2 Leakage-Safe Hyperparameter Sweep

  • 300 samples from three generators are pooled for the blob-filter grid search, with five hyperparameters considered.The parameters are an HSV saturation floor, minimum and maximum blob areas, minimum circularity, and minimum inertia ratio.Source: §A.2 · p. 18
  • The 100 source images are split 50/50 by image id, keeping all three model-versions of each image in the same fold to prevent scene overlap between train and validation.Hyperparameters are selected on train MAE, and pooled validation MAE is reported.Source: §A.2 · p. 19, Table 8 · p. 19
  • The sweep raises the saturation floor from 150 to 160 and maximum blob area from 800 to 1500, while leaving the remaining shape floors at conservative values.The stated rationale is that the literal argmin drives several parameters toward extreme values likely to overfit small validation sets.Source: §A.2 · p. 19

A.3 Residual Failure Modes

The mark-then-count probe has residual errors that no decoder can fully recover: red scene clutter can be mistaken for markers, while nearby markers can merge. These are generation-side limitations, motivating different marking strategies rather than more post-processing.

A.3 Residual Failure Modes

  • Red structures already present in a scene can be confused with genuine markers, and the shape filter suppresses but cannot eliminate this background clutter.Examples include red packaging and poppies; the authors describe the residual as inherent to the mark-then-count probe.Source: §A.3 · pp. 19–20
  • In densely packed scenes, touching markers can be recovered as one blob and cause undercounting; peak_dt splits such blobs but over-splits isolated dots, worsening results on this distribution.The authors identify this as a generation-side error and suggest changing the marking strategy rather than improving post-processing.Source: §A.3 · p. 20
  • The authors propose changing the marking strategy, such as alternating colours or using a separate channel, rather than relying on a better post-processing pipeline.Source: §A.3 · p. 20

B Efficiency Measurements

Efficiency measurements distinguish controlled local-model benchmarks from API response-time estimates. The API figures include external latency and queueing, so the authors report them for reference rather than direct comparison with local throughput.

Local models

  • Local models were timed on one NVIDIA H800 at batch size 1 after warm-up, with full per-image inference measured on fixed task-specific benchmark subsets.The reported measures are mean throughput in images per minute and peak GPU memory, using inputs resized to a long side of 1024.Source: B Efficiency Measurements · p. 20

API-hosted models

  • API-hosted models were timed one request at a time, and their response-time-derived throughput is not comparable with local throughput.These times include network latency, server-side queueing, and provider load balancing; GPU memory is unavailable.Source: B Efficiency Measurements · p. 20
  • The reported API measurements nevertheless support the qualitative conclusion that image generators deliver roughly one to a few images per minute, one to three orders of magnitude below specialists.Source: B Efficiency Measurements · p. 20

C Additional Qualitative Results

The additional galleries use logged selection criteria rather than best-case examples. They illustrate softer zero-shot generator depth, competitive referring and reasoning segmentation, generator advantages on stylized OOD images, and the mark-then-count interface.

Additional qualitative results

Depth and segmentation galleries

  • On NYUv2, zero-shot depth comparisons show Depth Anything V2-L and Lotus-2 recover sharper, better-calibrated geometry, while general generators produce coherent but softer relative depth without any depth-specific adaptation.Source: C Additional Qualitative Results · pp. 20–21
  • Figure 6 presents referring and reasoning segmentation: Nano Banana 2 grounds both direct references and multi-step reasoning queries zero-shot, and remains competitive with the agentic SAM3 + Qwen3-VL pipeline.Source: C Additional Qualitative Results · p. 21

Out-of-distribution robustness

  • On Manga109 and DRAM, Nano Banana Pro/2 preserve semantic binding and produce clean masks where specialists over- or under-segment, qualitatively corroborating the reported OOD gap.Figure 7 samples above-median generator-advantage cases across object classes.Source: C Additional Qualitative Results · p. 21

Counting pipeline

  • Figure 8 illustrates the counting interface: generators paint one red marker per instance, and deterministic Ecount recovers counts against ground-truth points in moderate and densely packed scenes.The displayed examples are accurate-decode cases with count error at most 2.Source: C Additional Qualitative Results · p. 21

D Limitations and Future Work

The authors present PROBEGEN as an initial measurement framework whose estimates are constrained by evaluation scale, generation-side artifacts, and current task and model coverage. They outline extensions to broader benchmarks, models, output interfaces, and inference settings.

Additional qualitative results

  • The qualitative figures use bounded selections: median-AbsRel depth scenes, high-IoU segmentation examples, above-median generator-advantage OOD cases, and accurate counting decodes.These selection criteria frame the galleries as illustrative rather than exhaustive benchmark summaries.Source: Figure 5 · p. 22, Figure 6 · p. 24, Figure 7 · p. 25

Evaluation scale and protocol

  • Each benchmark uses a 500-image subset and one deterministic sample per image, widening confidence intervals and making Vision Banana results on full benchmarks not directly comparable.Scaling to full benchmarks and multi-sample inference is proposed to tighten estimates.Source: D Limitations and Future Work · pp. 21–22

Generation-side bottlenecks

  • Imperfect pixel alignment, output-format non-compliance, and residual counting artifacts are generation-side bottlenecks, so reported metrics are lower bounds on what better-aligned generators could achieve.Source: D Limitations and Future Work · p. 22

Broader model and task coverage

  • PROBEGEN covers three tasks and a fixed generator roster; four additional open-weight models produced no valid mask, depth map, or dot annotation under the prompt-only protocol.Their outputs were global image changes or near-copies, and are omitted from the benchmark tables because they cannot be scored.Source: D Limitations and Future Work · p. 22
  • The authors propose extending the protocol to more open-weight generators and structured tasks, including surface normals, which they found unsolved in zero-shot settings.Source: D Limitations and Future Work · p. 23

Beyond RGB output spaces

  • Potential extensions include bounding boxes from masks or drawn rectangles, color-coded keypoint markers, and text answers decoded with OCR.The authors caution that converting structured annotations to RGB introduces residual errors and recommend keeping the extractor deterministic and simple.Source: D Limitations and Future Work · p. 23
  • Comparing alternative task interfaces could help separate a model's perceptual competence from the difficulty of its rendering format.Examples include counting with boxes instead of dots and segmentation with a colored overlay instead of a separate binary mask.Source: D Limitations and Future Work · p. 23

Richer inference settings

  • The effect of explicit thinking or reasoning modes remains untested; the authors identify varying these controls alongside prompts and sampling strategies as a low-cost extension.Source: D Limitations and Future Work · p. 23
Loading 2609.07884v1…