Source-linked AI summary
Let RGB Be the Language of Vision
Timing Yang, Jinrui Yang, Xinlong Li, Yuhan Wang, Haoran Li, Yanqing Liu, Guoyizhe Wei, Jixuan Ying, Chen Wei, Rama Chellappa, Yuyin Zhou, Cihang Xie, Alan Yuille, Feng Wang
TL;DR
Vision lacks a universal interface because different visual signals require separate representations and task-specific components. RINO represents visual inputs and outputs as RGB and enables a single image editor to handle understanding and generation tasks zero-shot, with competitive performance across applications and a δ1 score of 0.938 for depth estimation.
Problem
Vision lacks a universal interface because different visual signals require distinct representations, encoders, decoders, and adapters.
Method
RINO represents visual inputs and outputs as RGB, processing natural images and structured signals through the same image encoder and decoder without task-specific components.
Results
RINO handles tested understanding and generation tasks zero-shot, with competitive cross-application performance and a depth-estimation δ1 score of 0.938.
Takeaways & Limitations
RGB can serve as a common interface for perception and generation, supporting broad task transfer through a single frozen image editor.
Takeaways & Limitations
RINO’s zero-shot performance remains limited by task-specific recognition and instance-matching ambiguities, leaving gaps to specialist models on some tasks.
Abstract
from arXiv · showhide
This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to transfer across tasks through a unified visual interface, in a way analogous to how language models operate over text. We refer to this formulation as RGB In and RGB Out (RINO). Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance on both dense understanding tasks such as segmentation and depth estimation (where we unify outputs as RGB), and dense-conditioned generation tasks such as pose-to-image generation (where we unify inputs as RGB). We hope this study provides useful insights toward general unified vision-language systems, where diverse visual tasks can be expressed, interpreted, and solved through a shared visual language. Code is available at https://github.com/yangtiming/RINO.
1 Introduction
RINO proposes RGB In and RGB Out as a unified visual interface, representing diverse visual inputs and outputs in RGB and converting vision tasks into a shared image-editing formulation. Built on pretrained image-editing models without task-specific modules, it achieves broad zero-shot transfer across understanding and generation tasks while retaining limitations tied to its backbone dependence.
- Unified RGB interface: RINO represents all visual input and output signals in RGB format, providing a shared interface for diverse vision tasks.The paradigm uses the same encoding and decoding system for structured visual signals and natural images.
- Unified RGB interface: RINO evaluates a single RGB-based formulation across over 20 vision tasks and benchmarks spanning dense understanding, 3D estimation, and conditioned generation.All visual inputs and outputs share the same encoding and decoding system as natural images.
- Zero-shot transfer: RINO broadly handles tested understanding and generation tasks zero-shot, producing visually meaningful results with quantitative performance comparable to in-domain expert models.The model is built upon pretrained image-editing models without introducing auxiliary parameters.
- Zero-shot transfer: 0.938 δ1 is RINO’s depth-estimation score, approaching Depth Anything, a specialist model trained specifically for the task.The cited metric is δ1 score.
- Limitations: RINO’s strong transfer ability relies on generic image-editing backbones such as Qwen-Image-Edit and FireRed, revealing a limitation of the current models.The introduction presents the work as an initial attempt toward a large-scale, multi-task unified vision-language system.
2 Method: Unified RGB Interface for Vision
RINO unifies vision understanding and generation by representing all visual information as RGB and expressing tasks through a shared RGB/text interface. A generic image editor processes these signals without task-specific architectural components or fine-tuning.
- Unified interface: RINO uses a multimodal image editor F to map an input image and text prompt to an output image and optional text response.The formulation is described as an RGB/text paradigm for general vision-language understanding and generation.
- Unified interface: Natural images, masks, depth maps, pose maps, and edge maps share the same RGB encoding, image encoder, and decoder without task-specific heads, encoders, decoders, or adapters.This shared interface uses a generic pretrained image editing model such as Qwen-Image-Edit.
- Visual understanding: For visual understanding, a natural image and task prompt produce an RGB representation of structured information, such as a segmentation mask or depth map.An optional text output can explain the prediction or describe the generated visual result.
- Visual generation: For visual generation, an RGB condition such as a layout, pose, depth map, edge map, or mask guides production of a natural image under a text instruction.The paper treats conditioned generation as the dual problem of visual understanding.
- Base models: RINO-Zero runs three released image editors as black-box RGB-to-RGB models without adding, removing, or fine-tuning layers.The base models are Qwen-Image-Edit, LongCat-Image-Edit, and FireRed-Image-Edit.
3 Experiments
Across dense understanding and conditioned generation, RINO’s zero-shot RGB-to-RGB interface achieves meaningful, sometimes competitive performance without task-specific heads or training. Its main weaknesses are recognition, instance association, precise localization, and open-set presence detection, while generation results remain strong despite trailing trained control models in some settings.
- Dense understanding: RINO achieves semantic segmentation comparable to zero-shot specialists, surpassing segmentation experts on PASCAL VOC mAcc and aAcc and outperforming MaskCLIP in ADE20K’s 10-class coarse setting.These results indicate strong dense-prediction potential, although additional domain-specific knowledge may improve alignment.
- Dense understanding: 16.3% box AP50 and 13.9% mask AP50 show non-trivial zero-shot COCO localization and instance prediction, although Mask R-CNN reaches 63.5% box AP50.Qualitative failures include over-detection and over-segmentation, which the authors expect lightweight task-specific fine-tuning could mitigate.
- Dense understanding: Zero-shot panoptic masks are structurally meaningful and often align with boundaries, but recognition and instance association remain limiting, with Qwen reaching 22.4 PQ on stuff versus 1.5 PQ on things.Thing classes are especially prone to merged or missed adjacent instances and small objects.
- Other zero-shot tasks: RINO supports meaningful zero-shot pose estimation and free-form grounding, but precise keypoint localization and grounding accuracy remain challenging; open-vocabulary masks are coherent but struggle with absent-query detection.Qwen leads the editors on open-vocabulary segmentation, while grounding remains below Vision Banana and open-vocabulary pmF1 trails dedicated models.
- Conditioned generation: For controllable generation, Qwen’s zero-shot result is RMSE-255 33.83, FID 19.44, and CLIPScore 30.15, while pose-conditioned generation is comparable to in-domain models.On segmentation-conditioned generation, editors cluster around 46 mIoU on ADE20K and 37 mIoU on COCOStuff; trained control models remain stronger in some settings.
4 Related Work
Prior work largely uses dedicated architectures for visual estimation and controllable generation, while recent unified approaches reframe vision-task outputs as RGB images. Vision Banana bridges estimation and generation with one instruction-tuned generator competitive with zero-shot specialists.
- Visual estimation models: Visual estimation maps natural images to structured signals, with monocular depth progressing from mixed-dataset relative-depth models to large-scale, metric-aware systems.These specialist models define the performance ceiling and serve as references for the studied tasks.
- Visual estimation models: Open-vocabulary estimation names novel categories through language embeddings, while pose estimation, detection, and instance segmentation follow dedicated-architecture traditions.The passage also identifies text-grounded approaches for detection and instance segmentation, though the description is truncated.
- Controllable image generation: Controllable generation synthesizes natural images from structured conditions including edges, depth, semantic or instance maps, boxes, and poses.ControlNet attaches a trainable encoder copy to a text-to-image backbone, while later methods improve fidelity, extend autoregressive backbones, or learn per-condition adapter networks.
- Toward a unified model for estimation and generation: Vision Banana bridges estimation and generation by parameterizing vision-task outputs as RGB images and instruction-tuning a single generator.It is reported as competitive with zero-shot specialists including Segment Anything and the Depth Anything series.
5 Conclusion
RINO establishes RGB as a shared interface for perception and generation, allowing one frozen image editor to perform diverse visual tasks without task-specific components. RINO-Zero operates fully zero-shot across 25 tasks and approaches in-domain specialist models.
- Unified interface: RGB serves as a common interface for visual inputs and outputs across perception and generation.This formulation enables a single frozen image editor to handle both task types.
- Unified interface: A single frozen image editor handles broad task coverage without task-specific encoders, decoders, adapters, or auxiliary parameters.The shared setup uses no task-specific architectural additions.
- Zero-shot performance: 25 tasks are covered by RINO-Zero fully zero-shot, spanning dense estimation, 3D geometry, and conditioned generation.RINO-Zero requires no training and approaches in-domain specialist models across these tasks.