PaperlayerView exact revision on arXiv

Source-linked AI summary

Region-based Cluster Discrimination for Visual Representation Learning

Yin Xie, Kaicheng Yang, Xiang An, Kun Wu, Yongle Zhao, Weimo Deng, Zimin Ran, Yumeng Wang, Ziyong Feng, Roy Miles, Ismail Elezi, Jiankang Deng

arXiv:2507.20025v1cs.CV

TL;DR

Global vision-language representations have limited effectiveness for dense prediction, and instance-wise learning can miss semantic structure. RICE combines region-specific data, Region Transformer layers, and unified object/OCR cluster discrimination; it is reported to outperform prior methods across segmentation, dense detection, and MLLM perception.

  • Problem

    Global vision-language representations constrain effectiveness on dense prediction tasks, while instance-wise contrastive learning can treat semantically similar samples as negatives and weaken object- and scene-centric understanding.

    Source: §1 · p. 1

  • Method

    RICE builds a billion-scale candidate-region dataset and combines Region Transformer layers with a unified cluster-discrimination loss for object and OCR learning.

    Source: §1 · pp. 1–2, §5 · p. 8

  • Results

    RICE consistently outperforms prior methods across segmentation, dense detection, and visual perception for MLLMs.

    Source: §5 · p. 8

  • Takeaways & Limitations

    RICE is presented as a region-aware approach for strengthening object recognition and OCR capabilities in visual representations.

    Source: §5 · p. 8

Abstract

from arXiv · show

Learning visual representations is foundational for a broad spectrum of downstream tasks. Although recent vision-language contrastive models, such as CLIP and SigLIP, have achieved impressive zero-shot performance via large-scale vision-language alignment, their reliance on global representations constrains their effectiveness for dense prediction tasks, such as grounding, OCR, and segmentation. To address this gap, we introduce Region-Aware Cluster Discrimination (RICE), a novel method that enhances region-level visual and OCR capabilities. We first construct a billion-scale candidate region dataset and propose a Region Transformer layer to extract rich regional semantics. We further design a unified region cluster discrimination loss that jointly supports object and OCR learning within a single classification framework, enabling efficient and scalable distributed training on large-scale data. Extensive experiments show that RICE consistently outperforms previous methods on tasks, including segmentation, dense detection, and visual perception for Multimodal Large Language Models (MLLMs). The pre-trained models have been released at https://github.com/deepglint/MVT.

1. Introduction

The introduction identifies limits in global, instance-wise vision-language learning and image-level cluster supervision for region understanding. RICE combines region-specific data, Region Transformer layers, and unified object/OCR cluster discrimination, and is reported to outperform prior methods across several downstream tasks.

1. Introduction

  • Instance-wise contrastive learning treats semantically similar instances as negatives, while OCR pairs can shift focus toward text and away from high-level visual semantics.The introduction links these issues to weaker zero-shot object- and scene-centric understanding.Source: §1 · p. 1
  • RICE constructs region-specific object and OCR datasets and uses a unified cluster-discrimination framework for object classification and multi-label OCR learning.Object regions receive k-means semantic labels, while OCR regions receive tokenized text labels.Source: §1 · pp. 1–2
  • RICE consistently outperforms previous methods on segmentation, dense detection, and visual perception for MLLMs.The introduction presents this as the outcome of its region-aware representation-learning approach.Source: §1 · p. 2
  • The model combines standard Transformer layers with Region Transformer layers to strengthen regional representations while retaining global feature learning.Its efficient region-cluster classification loss jointly supports object and OCR learning and scalable distributed training.Source: §1 · p. 2

2. Related Work

Related work spans image-level instance and cluster discrimination, localized region representation learning, and multimodal language models. The paper positions region cluster centers as supervision that can support scalable region learning without descriptive text annotations.

Image Representation Learning

  • Instance-based methods such as CLIP [23] [50] [74] [75] and SigLIP contrast images, while UNICOM [3] and MLCD [4] group instances but generally assign pseudo-labels at image level.The section notes that instance-wise negatives restrict semantic relationships and image-level pseudo-labeling can hinder region representations.Source: §2 · p. 2

Region Representation Learning

  • Region methods range from overlap-based self-supervision in PreDet to text-aligned RegionCLIP [83] and self-training in GLIP [36]; RICE instead uses region cluster centers as supervision.The section says this removes reliance on descriptive textual annotations.Source: §2 · pp. 2–3

Multimodal Large Language Model

  • Multimodal LLM work includes vision-language integration and region-aware systems such as MiniGPT-4 [84] and LLaVA-G [81], while complex region understanding remains limited by image-encoder representations.The section also describes LISA [31] for reasoning-based segmentation and GLaMM [52] for region-level captioning and segmentation.Source: §2 · p. 3

3. Methodology

The methodology builds region-aware object and OCR datasets, then combines global and regional visual processing with a unified cluster-discrimination framework. Its data curation, Region Transformer layers, and specialized losses target region-level semantics while supporting efficient training.

3.1. Region Data Curation

  • The object dataset contains 400 million images and 2 billion candidate regions; OCR curation yields 50 million images and 400 million candidate regions.Object regions are clustered into one million semantic centers; OCR entries are retained when PaddleOCR confidence exceeds 0.7.Source: §3.1 · p. 3

3.2. Region-aware Representation

  • Standard Transformer layers preserve global context, while Region Transformer layers extract regional semantics using balanced region sampling and mask-guided attention.The visibility mask supports batching regions of different sizes, and the attention layer returns fixed-length region embeddings in one forward pass.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Region Cluster Discrimination

  • Object regions use a single positive cluster center, whereas OCR regions use multiple positive token embeddings; sampled negatives reduce computation and conflicting gradients.The negative sampling ratio controls sampling density, and the text reports that an appropriate ratio enhances performance.Source: §3.3 · pp. 4–5

4. Experiments

Across MLLM, segmentation, detection, and tracking evaluations, RICE shows strong performance, with reported gains over competing encoders and methods. Ablations identify useful hyperparameter settings under a ViT-B/16 setup using 10% of the pretraining data.

4.1. Implementation Details

  • Pretraining uses LAION2B, SAM1B, and COYO-700M, processing 13 billion samples initially with a global batch of 32K across 64 GPUs.The evaluated backbones are ViT-L/14 and ViT-B/16; high-resolution fine-tuning uses 1 billion samples.Source: §4.1 · p. 5

4.2. RICE as a Vision Encoder for MLLMs

  • RICE outperforms CLIP and SigLIP on most multimodal benchmarks, with particularly strong perception gains and OCR improvements including +50 OCRBench points over CLIP-336px.At 560px, RICE also scores 87.38% on DocVQA, above the 85.83% score of Qwen2.5-VL’s specialized backbone.Source: §4.2 · pp. 5–6, Table 1 · p. 6
  • In LLaVA-1.5 with Vicuna-7B, RICE improves refCOCO val, testA, and testB by 1.4%, 1.2%, and 2.8%, respectively, over CLIP.In LLaVA-NeXT with Qwen2.5-7B, it gains 0.7%, 0.7%, and 1.5% over MLCD on those splits.Source: §4.2 · p. 6, Table 2 · p. 7

4.3. Detection Probe

  • RICE outperforms existing methods across COCO and LVIS detection and segmentation benchmarks and Roboflow100-VL, evaluated with Cascade Mask R-CNN and frozen-backbone features.Table 3 reports Average Precision across these benchmarks and domains.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4. Tracking Probe

  • RICE performs better than competing vision encoders across LaSOT, TrackingNet, GOT-10k, and TNL2k tracking metrics, while PCA visualizations show stable focus on tracked entities across frames.The visualizations use ViT-B/16 token features projected onto RGB channels; sequential frames are arranged vertically.Source: §4.4 · pp. 7–8, Table 4 · p. 8, Figure 5 · p. 8

4.5. Ablation Study

  • Ablations using ViT-B/16 and 10% of pretraining data favor 10 object boxes and 1–2 million cluster centers, with COCO peaking at 1M and LVIS-Det at 2M.COCO-Det performs best at negative sampling rate 0.05, while other metrics favor 0.1; the study also analyzes positive OCR labels.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusions

RICE combines region-aware representation learning with joint object and OCR training, and consistently outperforms prior methods across several downstream tasks.

5. Conclusions

  • RICE consistently outperforms prior methods on segmentation, dense detection, and visual perception for MLLMs.Source: §5 · p. 8
  • The method uses a billion-scale candidate region dataset and a Region Transformer to capture regional semantic information.Source: §5 · p. 8
  • An efficient region-based classification loss jointly supports object and OCR learning and enables scalable distributed training on large-scale data.Source: §5 · p. 8

Related papers

Loading 2507.20025v1…