PaperlayerView exact revision on arXiv

Source-linked AI summary

Let RGB Be the Language of Vision

Timing Yang, Jinrui Yang, Xinlong Li, Yuhan Wang, Haoran Li, Yanqing Liu, Guoyizhe Wei, Jixuan Ying, Chen Wei, Rama Chellappa, Yuyin Zhou, Cihang Xie, Alan Yuille, Feng Wang

arXiv:2607.12450v1cs.CV

TL;DR

Vision tasks commonly use separate representations and dedicated components, and lack a universal interface. RINO casts visual inputs and outputs as RGB for a shared image editor, reporting zero-shot results comparable to in-domain experts across tested understanding and generation tasks.

  • Problem

    Vision lacks a universal interface, with different visual information types typically requiring distinct representations and dedicated model components.

    Source: §1 · p. 1

  • Method

    RINO represents visual inputs and outputs as RGB and processes them with a shared image encoder and decoder, without task-specific heads, encoders, decoders, or adapters.

    Source: §2 · p. 3

  • Results

    Across tested understanding and generation tasks, RINO reports robust zero-shot performance comparable to in-domain experts and a new zero-shot state of the art for conditioned image generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results provide evidence that RGB can serve as a shared interface for vision perception and generation.

    Source: §1 · p. 2

    RINO's transfer ability relies on generic image-editing backbones whose web-scale pretraining may already contain diverse visual signals such as masks and depth maps.

    Source: §1 · p. 2

Abstract

from arXiv · show

This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to transfer across tasks through a unified visual interface, in a way analogous to how language models operate over text. We refer to this formulation as RGB In and RGB Out (RINO). Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance on both dense understanding tasks such as segmentation and depth estimation (where we unify outputs as RGB), and dense-conditioned generation tasks such as pose-to-image generation (where we unify inputs as RGB). We hope this study provides useful insights toward general unified vision-language systems, where diverse visual tasks can be expressed, interpreted, and solved through a shared visual language. Code is available at https://github.com/yangtiming/RINO.

1 Introduction

RINO represents diverse visual inputs and outputs as RGB, letting a shared image-editing model address understanding and conditioned-generation tasks through one visual interface. The reported zero-shot results are promising, though transfer depends on generic image-editing backbones and may leave gaps against specialist models.

Motivation

  • Vision’s varied representations—such as RGB images, one-hot masks, and continuous depth maps—typically require dedicated encoders, decoders, and adapters, hindering free-form interaction and generalization.The introduction contrasts this with language models’ use of text as a common input and output format.Source: §1 · p. 1
  • RINO proposes a unified visual representation expressing all visual inputs and outputs in RGB, motivated by Vision Banana’s generative approach to segmentation, depth, and surface-normal estimation that renders segmentation masks in RGB.Source: §1 · p. 1

Approach

  • RINO expresses visual inputs and outputs in RGB across over 20 tasks and benchmarks, using a pretrained image editor without auxiliary parameters and only lightweight, parameter-free format converters.All visual information shares the model’s encoding and decoding system with natural images.Source: §1 · pp. 1–2

Results

  • RINO handles tested understanding and generation tasks zero-shot; its depth δ1 score of 0.938 approaches Depth Anything (Yang et al., 2024a), and it establishes a new zero-shot state of the art for conditioned image generation.Its understanding and generation results also receive high scores in human preference studies.Source: §1 · p. 2

Limitations and future work

  • RINO’s transfer relies on generic image-editing backbones whose pretraining may include masks and depth maps; their limited representation in that data may leave performance gaps versus in-domain experts.The paper calls the fully zero-shot version RINO-Zero and proposes instruction-tuning with standard editing data mixed with RGB-formatted non-natural visual data.Source: §1 · p. 2

2 Method: Unified RGB Interface for Vision

RINO casts vision understanding and generation as RGB/text-to-RGB tasks handled by a generic pretrained image editor, with diverse visual signals sharing one interface. Task-specific format conversion occurs only outside model inference.

Unified RGB interface

  • RINO uses a generic pretrained image editor to map an input image and text prompt to an output image and, optionally, a text response, without task-specific heads, encoders, decoders, or adapters.Natural images and structured signals—including masks, depth, pose, and edges—share the model’s image encoder and decoder.Source: §2 · p. 3

Visual understanding

  • In visual understanding, a natural image and task prompt produce an RGB-formatted structured prediction; parameter-free converters map it to task-specific formats only before or after inference.For example, segmentation colors map to class labels and RGB depth visualizations decode to depth maps; visual reasoning remains inside the base model.Source: §2 · p. 3

Visual generation

  • In conditioned generation, an RGB visual condition and text prompt produce a natural image, making generation the dual of mapping a natural image to an RGB structured prediction.Conditions can include semantic layouts, pose skeletons, depth maps, edge maps, or masks.Source: §2 · p. 3

Base models

  • RINO-Zero uses released weights from Qwen-Image-Edit, LongCat-Image-Edit, and FireRed-Image-Edit as black-box RGB-to-RGB editors, without adding, removing, or fine-tuning layers.The supplied passage describes Qwen-Image-Edit as built on a 20B-parameter model and LongCat-Image-Edit as a compact 6B editor.Source: §2 · p. 3

3 Experiments

The experiments evaluate RINO across zero-shot RGB-to-RGB understanding and generation tasks using generic image editors, including spatial generation with ordinary RGB conditioning images. They also examine task-specific performance and, through ablations, the spatial information conveyed by edge images.

3.1 Understanding Tasks: Unifying Outputs as RGB

  • Qwen-Image-Edit reaches δ1 0.938 on DIODE-indoor versus 0.952 for a specialist, while FireRed is the strongest editor for surface normals across three datasets.For normals, FireRed only slightly trails DSINE in mean angular error, though DSINE has lower median error; depth performance drops on wide-range scenes under the linear depth-space protocol.Source: §3.1 · pp. 4–5, Table 1 · p. 4, Table 2 · p. 4
  • On ADE20K, RINO can outperform semantic-segmentation specialist MaskCLIP in the coarse 10-class setting, but may need more domain-specific knowledge to align predictions with fine-grained names such as the 150 classes.The ADE20K strategy predicts masks for 10 super-classes, then generates fine-grained categories within their regions; parameter-free local majority filtering and small connected-component removal mitigate color jitter and noisy pixels.Source: §3.1 · pp. 5, 7
  • On COCO, RINO reaches 16.3% box AP50 and 13.9% mask AP50; for panoptic segmentation, Qwen’s stronger spatial masks are limited by recognition and instance association.Panoptic results include Qwen SQ 65–74 and RQ 15–17; on Cityscapes, Qwen reaches 22.4 PQ on stuff but 1.5 PQ on things. Detection uses oracle classes and instance counts, while panoptic prompts specify only present classes.Source: §3.1 · pp. 7–8, Table 6 · p. 7, Table 7 · p. 8
  • Zero-shot editors produce coherent pose layouts decodable into COCO-17 keypoints without pose-specific training or heads, though precise localization remains challenging.Similarity-aligned PCK improves over raw PCK, suggesting the editors capture coarse pose structure despite imperfect absolute localization.Source: §3.1 · pp. 9–10, Table 8 · p. 9
  • Generic editors can ground referring expressions and segment free-form concepts without task-specific heads, but open-set absent-case rejection remains a weakness.RefCOCOg grounding requires parsing an expression and identifying its target without boxes, classes, or counts; SA-Co editors produce coherent masks, but predictions for absent phrases keep IL_MCC near zero and suppress cgF1.Source: §3.1 · pp. 10–12, Table 9 · p. 10, Table 10 · p. 11

3.2 Generation Tasks: Unifying Inputs as RGB

  • Qwen’s zero-shot depth-conditioned generation scores RMSE-255 33.83, FID 19.44, and CLIPScore 30.15, while FireRed has CLIPScore 30.34 but worse depth fidelity.Qwen is behind the strongest trained controls but competitive with several trained adapter/control baselines in depth fidelity, using only an RGB condition image and instruction.Source: §3.2 · pp. 12–13, Table 11 · p. 12
  • Zero-shot editors preserve segmentation-map layouts, with ADE20K mIoU around 46 and COCOStuff mIoU around 37, while image quality varies by editor backbone.ADE20K mIoU is Qwen 46.24, FireRed 45.82, and LongCat 46.37; COCOStuff mIoU is FireRed 37.55, Qwen 37.40, and LongCat 36.38. Qwen and FireRed have similar FID, while LongCat trails.Source: §3.2 · pp. 13–14, Table 12 · p. 14
  • The supplied pose-conditioned generation setup feeds an RGB OpenPose-style skeleton and caption to the editor, without a pose encoder, ControlNet branch, or trainable adapter.The skeleton is rendered from ground-truth COCO-17 keypoints, and the editor is instructed to align body parts with colored bones while omitting the overlay; no result is supplied in this part.Source: §3.2 · p. 14
  • Zero-shot editors achieve highly competitive pose-conditioned generation, and RINO often follows poses better than ControlNet-based baselines for unusual body configurations and diverse styles.The editors were not trained on this task; their outputs also maintain strong text-image alignment and visual realism.Source: Table 13 · p. 15, Figure 12 · p. 15, §3.2 · pp. 15–16
  • LongCat achieves 28.0 AP and 44.2 AR, exceeding InstanceDiffusion on both; editors’ FID is 15.3–18.5 vs. 25.5, while the reference leads at AP50, 50.0 vs. 42–46.The zero-shot editors consume per-instance RGB maps and VLM-recaptioned prompts without an instance encoder, grounding module, detector head, or trainable adapter.Source: Table 14 · p. 16, §3.2 · pp. 16–17
  • Zero-shot Qwen-Image-Edit reaches Canny F1 14.90 versus 37.04 for ControlNet++, while FID 59.4 and CLIP-Score 27.3 trail trained controllers.RINO treats the edge map as an ordinary RGB condition without an edge encoder, ControlNet branch, adapter, or task-specific decoder.Source: Table 15 · p. 17, §3.2 · pp. 17–18
  • Canny-image conditioning yields higher controllability than caption-only or VLM-described edges, indicating that the RGB map carries pixel-level spatial information.Canny F1 falls from 14.90 with the edge image to 6.75 with caption-only input and 7.79 with caption plus a Canny description; structure improves FID from 71.6 to 58–59, while the description lowers CLIP-Score from 27.8 to 24.0.Source: §3.2 · p. 18

4 Related Work

Related work treats visual estimation and controllable generation as separate directions, typically relying on task-specific architectures or condition modules. RINO builds on RGB as a shared interface and applies it to frozen, open-source image editors across 25 tasks.

Visual estimation models

  • Estimation spans specialist approaches to depth and segmentation, including mask classification, but tasks still use their own encoders, decoders, and losses (Ranftl et al., 2022; Cheng et al., 2021, 2022).Some segmentation methods extend to open-vocabulary categories via language embeddings (Xu et al., 2022a; Dong et al., 2023).Source: §4 · pp. 18–19

Controllable image generation

Toward a unified model for estimation and generation

  • RINO uses frozen open-source image editors and parameter-free RGB conversions to unify estimation and conditioned generation across 25 tasks in a fully zero-shot RGB-to-RGB formulation.The paper measures how closely this system approaches specialist models.Source: §4 · p. 19
  • Vision Banana (Gabeur et al., 2026) reframes perception as RGB image generation and instruction-tunes one generator, but relies on a proprietary backbone.The passage says it is competitive with zero-shot specialists and notes similar behavior in open-source editors (Liu et al., 2026).Source: §4 · p. 19

5 Conclusion

RINO uses RGB as a shared interface for perception and generation, allowing one frozen image editor to handle diverse visual tasks without task-specific components. Fully zero-shot RINO-Zero approaches specialist models across 25 tasks, while its accuracy remains limited by the backbone’s visual priors and parameter-free RGB read-out.

5 Conclusion

  • Fully zero-shot RINO-Zero approaches in-domain specialist models across 25 tasks spanning dense estimation, 3D geometry, and conditioned generation.It also offers a zero-shot alternative to adapter-based controllable generation.Source: §5 · p. 19
  • Expressing visual inputs and outputs as RGB lets a single frozen image editor handle broad tasks without task-specific encoders, decoders, adapters, or auxiliary parameters.The conclusion presents RGB as a common interface for both perception and generation.Source: §5 · p. 19
  • RINO accuracy remains bounded by the editing backbone’s visual priors and parameter-free RGB read-out, leaving gaps to specialists on some tasks.Lightly instruction-tuning editors on RGB-formatted visual signals is proposed as a natural way to close the gap.Source: §5 · p. 19
  • The findings invite extending the shared RGB interface to 3D vision, video, and world models.Source: §5 · p. 19

Related papers

Loading 2607.12450v1…