PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Generators are Generalist Vision Learners

Valentin Gabeur, Shangbang Long, Songyou Peng, Paul Voigtlaender, Shuyang Sun, Yanan Bao, Karen Truong, Zhicheng Wang, Wenlei Zhou, Jonathan T. Barron, Kyle Genova, Nithish Kannen, Sherry Ben, Yandong Li, Mandy Guo, Suhas Yogin, Yiming Gu, Huizhong Chen, Oliver Wang, Saining Xie, Howard Zhou, Kaiming He, Thomas Funkhouser, Jean-Baptiste Alayrac, Radu Soricut

arXiv:2604.20329v3cs.CVcs.AI

TL;DR

The paper addresses limited evidence that generative vision models learn strong, general visual understanding. It instruction-tunes Nano Banana Pro to express vision-task outputs as RGB images, achieving state-of-the-art or competitive 2D and 3D performance while retaining image-generation capabilities. The results support image-generation pretraining as a foundation for generalist vision models and image generation as a unified interface for vision tasks.

  • Problem

    Evidence remains limited that image and video generators develop strong, general visual understanding despite observed zero-shot visual behaviors.

  • Method

    Vision Banana instruction-tunes Nano Banana Pro on its original image-generation data plus a small amount of vision-task data, parameterizing task outputs as RGB images.

  • Results

    Vision Banana achieves state-of-the-art or competitive results across 2D and 3D understanding, including surpassing specialized segmentation and metric-depth models, while retaining image-generation performance.

  • Takeaways & Limitations

    The findings suggest that image-generation pretraining can produce generalist visual representations and that image generation can serve as a unified interface for vision tasks.

  • Takeaways & Limitations

    The current evaluation focuses on monocular image inputs; extending the framework to multi-view and video inputs remains future work.

Abstract

from arXiv · show

Recent works show that image and video generators exhibit zero-shot visual understanding behaviors, in a way reminiscent of how LLMs develop emergent capabilities of language understanding and reasoning from generative pretraining. While it has long been conjectured that the ability to create visual content implies an ability to understand it, there has been limited evidence that generative vision models have developed strong understanding capabilities. In this work, we demonstrate that image generation training serves a role similar to LLM pretraining, and lets models learn powerful and general visual representations that enable SOTA performance on various vision tasks. We introduce Vision Banana, a generalist model built by instruction-tuning Nano Banana Pro (NBP) on a mixture of its original training data alongside a small amount of vision task data. By parameterizing the output space of vision tasks as RGB images, we seamlessly reframe perception as image generation. Our generalist model, Vision Banana, achieves SOTA results on a variety of vision tasks involving both 2D and 3D understanding, beating or rivaling zero-shot domain-specialists, including Segment Anything Model 3 on segmentation tasks, and the Depth Anything series on metric depth estimation. We show that these results can be achieved with lightweight instruction-tuning without sacrificing the base model's image generation capabilities. The superior results suggest that image generation pretraining is a generalist vision learner. It also shows that image generation serves as a unified and universal interface for vision tasks, similar to text generation's role in language understanding and reasoning. We could be witnessing a major paradigm shift for computer vision, where generative vision pretraining takes a central role in building Foundational Vision Models for both generation and understanding.

1. Introduction

The paper asks whether image generators develop general visual understanding, and tests this by instruction-tuning Nano Banana Pro to produce benchmarkable visual outputs. Vision Banana matches or surpasses specialized models across 2D and 3D understanding while retaining strong image-generation performance.

  • The paper investigates whether image-generation training produces internal representations suitable for general visual understanding tasks.
  • Vision Banana instruction-tunes Nano Banana Pro to generate RGB visualizations that can be decoded into computer-vision outputs for quantitative evaluation.
  • Vision Banana surpasses or matches state-of-the-art results across 2D and 3D vision benchmarks, including segmentation and metric depth estimation.
  • 53.5% win rate against Nano Banana Pro on GenAI-Bench and 47.8% on ImgEdit show that instruction-tuning preserved competitive image generation and editing.
  • The findings suggest that image generators are generalist vision learners and that image generation can provide a unified interface for visual understanding tasks.

2. Method

The method aligns Nano Banana Pro with vision tasks through lightweight instruction-tuning while preserving its original generative priors. Vision task outputs are represented as invertible RGB images and evaluated across 2D and 3D understanding.

  • Vision Banana is created by mixing a small amount of vision-task data into Nano Banana Pro’s original training mixture.
  • Vision tasks are formatted as invertible visualizations, allowing generated outputs such as depth heatmaps to be decoded for quantitative assessment.
  • Low-ratio vision-data mixing is designed to align vision tasks without degrading Nano Banana Pro’s original image-generation capabilities.
  • The evaluation covers 2D referring-expression, semantic, and instance segmentation alongside 3D monocular metric depth and surface-normal estimation.

3. Vision Banana - Generalist Vision Model from Image Generator

Vision Banana reframes diverse 2D and 3D vision tasks as image generation, achieving SOTA-level performance across segmentation, depth, and surface-normal estimation without specialized architectures or custom losses.

  • 2D Semantic Understanding: Vision Banana reframes semantic segmentation as generating RGB visualizations whose pixel colors encode prompted categories and can be decoded into masks.The approach supports open-vocabulary categories, natural-language or structured color mappings, and RGB-space post-processing.
  • 2D Semantic Understanding: 4.7 points: Vision Banana outperforms SAM 3 in Cityscapes mIoU and leads open-vocabulary models.It narrows the gap with closed-set, non-zero-shot specialists such as SegMan.
  • 2D Semantic Understanding: 73.8% cIoU on RefCOCOg UMD and 79.3% gIoU on ReasonSeg establish state-of-the-art zero-shot referring-expression segmentation.The model consistently outperforms SAM 3 Agent and other recent zero-shot methods, though trained-on-training-split methods retain an advantage on RefCOCOg.
  • 2D Semantic Understanding: Vision Banana transfers across segmentation paradigms, handling nuanced language cues and referring expressions despite not being explicitly trained on free-form queries for standard segmentation.Examples include identifying “patterns on the wall” and distinguishing crescent-shaped croissants from other croissants.
  • 3D Understanding from Monocular Images: Vision Banana achieves SOTA performance on monocular metric depth and surface-normal estimation, producing globally consistent geometry and finer-grained normal maps.It has the lowest mean and median angular errors across three indoor surface-normal datasets, while remaining competitive outdoors; on Virtual KITTI 2, Lotus-2 has slightly lower quantitative errors despite lower visual fidelity.
  • 3D Understanding from Monocular Images: 0.929 versus 0.918 average δ1: Vision Banana outperforms Depth Anything V3 across their four shared evaluated datasets.The model uses no real-world depth data or camera parameters during training or inference, and achieves an average δ1 of 0.882 across six benchmarks.

4. Discussion

The paper argues that generative vision pretraining can produce models capable of both image generation and broad visual understanding. Encoding task outputs as RGB images provides a unified interface, while generative modeling can represent ambiguous output distributions without bespoke architectures; computational overhead remains a deployment challenge.

  • Large-scale image-generation pretraining naturally acquires robust visual understanding capabilities that can support both generation and understanding.The authors position Vision Banana's results as evidence that generative vision pretraining can underpin foundational vision models.
  • Representing vision-task outputs as RGB images lets natural-language prompts instruct a pretrained visual generator through a unified interface.The paper reports that this simple design can outperform modern domain-specific specialist models when combined with powerful pretrained visual generators.
  • Generative modeling can handle inherently ambiguous vision outputs by learning the full data distribution rather than collapsing predictions to a blurry mean.This may reduce reliance on custom architectures and training losses designed for multimodal outputs.
  • Current evaluation covers fundamental 2D semantic and monocular 3D understanding tasks, leaving multi-view, video, broader task diversity, and cross-modality reasoning for future work.The authors also identify computational cost as a significant hurdle for deploying generative vision frameworks.

A. Multi-stage clustering algorithm for parsing instance masks

Vision Banana converts generated RGB images into discrete instance masks through a multi-stage algorithm that handles unknown instance counts, color variation, noise, boundaries, and fragmented objects.

  • Overview: The algorithm dynamically assigns unique colors to instances, then robustly maps generated RGB outputs back to discrete masks.This is necessary because instance counts are unknown a priori and generated images contain noise, color drift, and boundary artifacts.
  • Background Initialization: Background pixels are initialized using proximity to a predefined background color and excluded from later processing.The method uses a color tolerance threshold of τ=14.
  • Color Similarity Grouping: Remaining pixels are grouped with seed-based flood fill using 16-connectivity and RGB color similarity.Adjacent pixels are merged according to their color distance from the component seed.
  • Boundary Artifact Elimination: Boundary halos are removed by 3 × 3 binary erosion when the preserved area ratio falls below θ_erosion=0.1.This prevents thin colored boundaries from becoming spurious components.
  • Spatially Constrained Merging: Spatially disjoint components are merged when their average colors are similar and their combined bounding box does not expand excessively.This recovers fragmented parts of the same object, such as regions separated by occlusion.

B. Detailed SA-Co/Gold Quantitative Evaluation

Vision Banana achieves state-of-the-art zero-shot performance on SA-Co/Gold, including the primary open-vocabulary segmentation metric, while evaluation combines segmentation quality with query-presence classification.

  • Benchmark: SA-Co/Gold evaluates noun-phrase instance segmentation across seven diverse subsets, including Metaclip, SA-1B, Crowded, and Wiki-Common.The benchmark also includes Food&Drink, Sports Equip., and Attributes.
  • Metrics: The primary classification-gated F1 metric combines Positive Micro-F1 with Image-Level Matthews Correlation Coefficient.Positive Micro-F1 measures mask precision and recall across ten IoU thresholds, while Image-Level MCC measures query-presence classification.
  • Results: Vision Banana demonstrates state-of-the-art zero-shot performance, outperforming specialized models including OWLv2, DINO-X, and APE-D.Positive/negative query filtering uses Gemini 3.1 Flash-Lite, which may contribute to the strong Image-Level MCC score.
  • Results: Vision Banana also achieves state-of-the-art Positive Micro-F1, the metric directly measuring open-vocabulary segmentation quality.This result remains central despite the model’s use of Gemini for query filtering in the unified metric.

C. Qualitative Instance Segmentation Analysis

Qualitative examples show Vision Banana producing coherent, non-overlapping masks and resolving spatial ambiguity, while failures remain in cluttered scenes and can be amplified by annotation mismatch.

  • Evaluation Caveats: Quantitative F1 can remain substantially below SAM 3 despite visually accurate masks because smaller or finer predictions misalign with SAM-annotated ground truth at high IoU thresholds.The discrepancy reflects sensitivity to annotation geometry rather than only visible mask quality.
  • Success Cases: Vision Banana commits to a single coherent mask mode, avoiding SAM 3’s mode-averaging artifacts under spatial ambiguity.It also consistently produces non-overlapping masks without unnatural gaps between adjacent boundaries.
  • Failure Cases: Observed failures include merging multiple instances, splitting group instances, and occasionally missing target objects entirely.Examples include merged walls or furniture, crowds split into individuals, and missed background walls or persons.
  • Evaluation Caveats: The model has clear room for improvement in highly cluttered or crowded scenes, while annotation biases and zero-shot evaluation explain part of the gap with SAM 3.These factors constrain interpretation of the quantitative comparison.

D. Image generation and editing capabilities

Instruction-tuning preserves Vision Banana’s generative abilities: compared with Nano Banana Pro, it maintains detailed text-to-image quality and robust instruction adherence for image editing.

  • Evaluation Setup: Figures 11 and 12 compare Vision Banana with Nano Banana Pro on text-to-image generation and image editing, respectively.The figures assess whether instruction-tuning caused forgetting of core generative features.
  • Text-to-Image: Vision Banana continues producing detailed, contextually accurate text-to-image outputs with quality similar to Nano Banana Pro.The comparison uses prompts sampled from GenAI-Bench.
  • Image Editing: Vision Banana shows robust adherence to detailed instructions on image-editing tasks.The comparison uses instruction-based prompts from ImgEdit.
  • Segmentation Context: The qualitative segmentation figures report best F1 scores with associated ground truth annotations and predicted mask counts for SAM 3 and Vision Banana.Figures 9 and 10 present success and failure cases on SA-Co/Gold.
Loading 2604.20329v3…