PaperlayerView exact revision on arXiv

Source-linked AI summary

FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model

Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Ji Ao, Dawei Leng, Yuhui Yin

arXiv:2510.10921v3cs.CVcs.AIcs.LG

TL;DR

Current vision-language models often struggle with fine-grained visual and linguistic distinctions, especially in non-English settings. FG-CLIP 2 combines bilingual data, region-text supervision, long captions, and discriminative objectives for English and Chinese. It reports stronger performance across 29 datasets and 8 tasks in both languages.

  • Problem

    Existing vision-language models often miss fine-grained object attributes, spatial relations, and linguistic distinctions, while bilingual fine-grained alignment remains underexplored.

    Source: §1 · p. 1

  • Method

    FG-CLIP 2 uses two-stage bilingual training with short and long captions, region-text supervision, and TIC to distinguish semantically similar descriptions.

    Source: §1 · p. 2, §3.2 · pp. 3–4

  • Results

    FG-CLIP 2 outperforms existing models on 29 datasets across 8 vision-language tasks in both Chinese and English.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The paper adds Chinese long-caption retrieval and region-based classification benchmarks for evaluating fine-grained multimodal understanding.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Fine-grained vision-language understanding requires precise alignment between visual content and linguistic descriptions, a capability that remains limited in current models, particularly in non-English settings. While models like CLIP perform well on global alignment, they often struggle to capture fine-grained details in object attributes, spatial relations, and linguistic expressions, with limited support for bilingual comprehension. To address these challenges, we introduce FG-CLIP 2, a bilingual vision-language model designed to advance fine-grained alignment for both English and Chinese. Our approach leverages rich fine-grained supervision, including region-text matching and long-caption modeling, alongside multiple discriminative objectives. We further introduce the Textual Intra-modal Contrastive (TIC) loss to better distinguish semantically similar captions. Trained on a carefully curated mixture of large-scale English and Chinese data, including a newly released 12M Chinese region-text dataset, FG-CLIP 2 achieves powerful bilingual performance. To enable rigorous evaluation, we present a new benchmark for Chinese multimodal understanding, featuring long-caption retrieval and bounding box classification. Extensive experiments on 29 datasets across 8 tasks show that FG-CLIP 2 outperforms existing methods, achieving state-of-the-art results in both languages. We release the model, code, and benchmark to facilitate future research on bilingual fine-grained vision-language alignment.

1. Introduction

Existing vision-language models excel at global alignment but often miss fine-grained visual and linguistic distinctions, while fine-grained alignment remains underexplored for bilingual settings. FG-CLIP 2 combines fine-grained training objectives with bilingual data, and experiments report gains across both languages.

1. Introduction

  • Current models often struggle with object attributes, spatial configurations, and subtle semantic distinctions because coarse-grained image-text training misses fine-grained correspondences.These correspondences matter for visual grounding and attribute recognition.Source: §1 · p. 1
  • Prior fine-grained approaches are predominantly English-focused, while Chinese vision-language models primarily target coarse-grained alignment, leaving bilingual fine-grained alignment underexplored.The paper’s Chinese benchmark adds long-caption retrieval and bounding-box classification beyond conventional short-text retrieval.Source: §1 · p. 2
  • The two-stage training paradigm moves from global alignment using short and long descriptions to objectives for regional alignment, discrimination, and cross-modal ranking.TIC uses filtered hard negatives among highly similar text pairs to help distinguish similar region-level descriptions.Source: §1 · p. 2
  • FG-CLIP 2 outperforms existing models on 29 datasets across 8 vision-language tasks in both Chinese and English.The paper also introduces FineRegion-CN, a 12-million-image Chinese region-text dataset, and a Chinese benchmark for long-caption retrieval and bounding-box classification.Source: §1 · p. 2

2. Related Work

Prior work has advanced global alignment, localization, and multilingual coverage, but fine-grained and multilingual understanding have often been pursued separately. This section positions unified bilingual fine-grained alignment as the remaining gap.

2. Related Work

  • Large-scale alignment models show strong zero-shot capabilities but primarily use global semantics and often English-only corpora, limiting fine-grained and multilingual understanding.These models also underpin downstream multimodal reasoning, detection, and segmentation systems.Source: §2 · p. 2
  • Region-level supervision methods use boxes or masked regions to improve local correspondence, while other methods enhance feature generation through architectural or training changes.Examples include AlphaCLIP, UMG-CLIP, FineCLIP, FG-CLIP, CLOC, TIPS, and SigLIP 2.Source: §2 · p. 2
  • Chinese-CLIP and R2D2 target Chinese understanding, and MetaCLIP 2 broadens multilingual data coverage, but prior work does not explicitly optimize fine-grained and multilingual understanding together.The paper identifies this separation as the gap motivating its approach.Source: §2 · p. 2

3. Approach

FG-CLIP 2 uses two-stage bilingual training, moving from global alignment with short and long captions to fine-grained region-caption learning. Its approach combines discriminative objectives with English and Chinese training data and introduces Chinese benchmarks for long-caption retrieval and region-level evaluation.

3.1. Architecture

  • The two-stage framework pairs bilingual global alignment with short and long captions, then adds region-level alignment and fine-grained contrastive signals.Its architecture uses data-adaptive image resolution, a longer text input limit, pooled global features, and region embeddings.Source: §3 · p. 2, §3.1 · p. 2, Figure 1 · p. 3

3.2. Training Objectives

  • Global alignment uses sigmoid-based image-text matching with both short captions and LMM-generated long captions as complementary supervision.The short captions provide concise global semantics, while long captions add detailed context.Source: §3.2 · p. 3
  • Region-level visual learning aligns RoIAlign-derived region features with their corresponding bounding-box descriptions, while textual learning contrasts each positive pair against 10 attribute-perturbed negatives.The textual objective uses binary classification over one positive and 10 semantically similar negatives.Source: §3.2 · p. 3
  • TIC selects top-10 hard-negative texts after filtering pairs with similarity above 0.95, encouraging lower similarity for semantically close but distinct descriptions.The loss operates within the text modality to sharpen the text encoder’s representation space.Source: §3.2 · pp. 3–4
  • Cross-modal rank loss enforces a margin between positive and hard-negative pairs, with its margin synchronized across GPUs using similarities from the previous training step.Stage II combines the global objective with four auxiliary objectives, using fixed weights λ1 = 0.1, λ2 = 0.5, λ3 = 0.1, and λ4 = 0.4.Source: §3.2 · p. 4

3.3. Training Data

  • Training uses English LAION-2B captions and Chinese Wukong, Zero, and in-house image-text data, then adds FineHARD and the 12-million-image FineRegion-CN region-text datasets.FineRegion-CN contains Chinese bounding boxes and fine-grained region descriptions.Source: §3.3 · p. 4

3.4. Bilingual Evaluation Protocols

  • Because Chinese benchmarks mostly emphasize short-caption retrieval, the evaluation suite adds LIT-CN, DCI-CN, and DOCCI-CN for long-caption retrieval.LIT-CN contains 33,010 image-text pairs with descriptions averaging 131 tokens; DCI-CN and DOCCI-CN use Chinese translations of their source captions.Source: §3.4 · p. 4
  • BoxClass-CN evaluates Chinese region-text alignment with 24,221 images, 66,258 region-text pairs, and 566 semantically refined categories.Its construction pipeline generates captions, extracts referring expressions, localizes boxes, and filters them using region-text similarity.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate FG-CLIP 2 on fine-grained visual understanding, bilingual retrieval, and downstream tasks. Across these settings, the reported results include gains over prior models, while ablations assess the training losses and bilingual data.

4.2.1. FINE-GRAINED UNDERSTANDING

  • FG-CLIP 2 achieves significant gains over prior models on FG-OVD, which tests region-text matching across four difficulty levels.The subsets range from trivial to hard according to the linguistic subtlety between correct and distractor descriptions; evaluation uses Top-1 accuracy.Source: Table 1 · p. 5, §4.2.1 · p. 5

4.2.2. BOUNDING BOX CLASSIFICATION

  • FG-CLIP 2 achieves state-of-the-art bounding box classification in English and Chinese, significantly outperforming compared open-source models.The evaluation uses COCO and LVIS for English categories and BoxClass-CN for Chinese.Source: Table 2 · p. 5, §4.2.2 · p. 6

4.2.3. OPEN-VOCABULARY OBJECT DETECTION

  • Combined with LLMDet, FG-CLIP 2 achieves the best performance among open-source methods on LVIS open-vocabulary detection.The training-free evaluation combines alignment-model similarity scores with detector confidences via geometric averaging.Source: Table 3 · p. 6, §4.2.3 · p. 7

4.3.1. LONG/SHORT CAPTION IMAGE-TEXT RETRIEVAL

  • FG-CLIP 2 improves long- and short-caption retrieval across English and Chinese settings, outperforming Meta CLIP 2 despite its smaller ViT-L/16 backbone and 1.0B parameters.Meta CLIP 2 uses a 1.8B-parameter ViT-H/14 backbone.Source: Table 4 · p. 6, Table 5 · p. 7, §4.3.1 · p. 7

4.3.2. ZERO-SHOT IMAGE CLASSIFICATION

  • FG-CLIP 2 is competitive with SigLIP 2 on zero-shot image classification and outperforms the other listed comparison models.The evaluation uses standard prompts on ImageNet-1K and ImageNet-v2.Source: Table 4 · p. 6, §4.3.2 · pp. 7–8

4.4. Dense prediction tasks

  • FG-CLIP 2 achieves the best open-vocabulary segmentation performance across models of various scales, with consistent improvements over the baseline.Evaluation covers ADE20k, Pascal Context, and Pascal VOC category schemas.Source: Table 6 · p. 7, §4.4 · p. 8

4.5. Large Multimodal Model tasks

  • LMMs equipped with FG-CLIP 2 outperform those using other open-source encoders on the evaluated multimodal benchmarks.The benchmarks span visual question answering, multimodal understanding, referring expression comprehension, and fine-grained visual reasoning.Source: Table 7 · p. 8, §4.5 · p. 8

4.6. Ablation Study

  • Removing LTIC lowers COCO Top-1 by 4.8 points to 70.1% and FG-OVD Hard from 52.3% to 51.6%; removing LCMR lowers FG-OVD Hard to 50.9%.Removing both losses drops COCO Top-1 to 62.7%. The bilingual second-stage variant improves Chinese and English-only benchmarks, while a re-trained-baseline comparison attributes gains to the architecture and learning objectives.Source: Table 8 · p. 8, §4.6 · pp. 8–9

5. Conclusion and Limitations

FG-CLIP 2 combines bilingual fine-grained supervision and discriminative objectives, and the authors report strong performance across 29 datasets and 8 tasks. The paper also introduces a Chinese multimodal benchmark and identifies longer inputs and object relations as future directions.

  • FG-CLIP 2 achieves superior performance across 29 datasets and 8 tasks, demonstrating strong bilingual generalization.Source: §5 · p. 9
  • Its two-stage training uses short and long captions, region-text supervision, and discriminative objectives including TIC loss to distinguish similar descriptions.Source: §5 · p. 9
  • The new Chinese benchmark includes long-caption image-text retrieval and bounding box classification.Source: §5 · p. 9
  • The authors release the model, code, and benchmark to support research on bilingual fine-grained vision-language understanding.Source: §5 · p. 9
  • Future work will extend the model to longer textual inputs and explicitly model relations among objects.Source: §5 · p. 9

Impact Statement

The impact statement notes that the work advances machine learning and has broad potential societal implications, while identifying no specific consequences for special emphasis.

A. Training Data Details

The appendix summarizes the two-stage training datasets and provides a separate detailed Cat-Seg parameter table.

  • The first-stage dataset overview lists LAION-2B-enhanced at 1.6B English examples and three Chinese sources: Wukong at 100M, Zero at 250M, and in-house data at 500M.Source: A. Training Data Details · p. 13
  • The region-text datasets are FineHARD, with 12M English examples, and FineRegion-CN, with 12M Chinese examples.Source: A. Training Data Details · p. 13
  • The Cat-Seg configuration specifies 80,000 maximum iterations, a 0.0002 base learning rate, and 8 images per batch.Source: A. Training Data Details · p. 13

B. Open-Vocabulary Object Detection Experimental Details

The object-detection setup recalibrates LLMDet category predictions with FG-CLIP 2 alignment scores. The dense-prediction details describe Cat-Seg, its configuration, and bilingual training-data examples.

B. Open-Vocabulary Object Detection Experimental Details

C. Dense Prediction Tasks Experimental Details

D. Ablation on Bilingual Data Composition

The Stage II ablation compares English-only with bilingual training data from the same Stage I checkpoint. The bilingual variant improves results on both Chinese and English evaluations.

D. Ablation on Bilingual Data Composition

  • Both variants are initialized from the same Stage I checkpoint trained on the full bilingual dataset.Stage II compares English-only training data with bilingual English-Chinese training data.Source: D. Ablation on Bilingual Data… · p. 14
  • The authors describe the improvements across both language evaluations as a mutually promoting effect between English and Chinese.Source: D. Ablation on Bilingual Data… · p. 14
  • The bilingual Stage II variant outperforms the English-only variant on Chinese benchmarks and consistently improves English-only evaluation results.The comparison starts from the same Stage I checkpoint; Table C identifies the two data-composition settings.Source: D. Ablation on Bilingual Data… · p. 14, Table C · p. 15

E. Comparison with Re-trained Baselines

Using only Stage II data and no Stage I pretraining, FG-CLIP 2 outperforms the retrained SigLIP 2 and Chinese-CLIP baselines across the reported benchmarks. The comparison uses identical data conditions to focus on architectural and learning-objective differences.

E. Comparison with Re-trained Baselines

  • The baselines are retrained on exactly FG-CLIP 2’s Stage II training data, and all compared models omit Stage I pretraining.This setup is intended to isolate architecture and learning objectives from training-data effects.Source: E. Comparison with Re-trained Baselines · p. 15
  • The authors attribute FG-CLIP 2’s gains in this controlled comparison to its architecture and tailored learning objectives beyond bilingual training data.Source: E. Comparison with Re-trained Baselines · p. 15
  • FG-CLIP 2 outperforms SigLIP 2 and Chinese-CLIP across all six reported benchmarks, including DCI, MSCOCO, DOCCI-CN, Flickr30k-CNA, COCO80, and Fine-Grained Understanding.The comparison uses only Stage II data, without Stage I pretraining.Source: Table D · p. 15, E. Comparison with Re-trained Baselines · p. 15

F. Hyperparameter Ablation Study

The hyperparameter study tunes four auxiliary losses sequentially against validation performance averaged over representative tasks. A separate visualization reports precise bilingual alignment between dense visual features and text.

F. Hyperparameter Ablation Study

  • Four auxiliary losses are added and tuned one at a time, with earlier weights fixed and future losses excluded, selecting weights by average validation performance.The losses are FGV, FGT, TIC, and CMR, alongside the global alignment objective.Source: F. Hyperparameter Ablation Study · p. 15
  • The hyperparameter analysis averages performance across DCI, MSCOCO, DOCCI-CN, Flickr30k-CNA, Bbox Classification (COCO80), and Fine-Grained Understanding.Figure B shows average performance over different values of each loss hyperparameter.Source: F. Hyperparameter Ablation Study · p. 15

G. Visualization of Semantic Alignment Capability for Dense Visual Features

  • In Chinese and English contexts, high-similarity regions are precisely localized to matched text in the dense-feature visualization.Warmer colors indicate higher similarity between image regions and matched text.Source: G. Visualization of Semantic Alignment… · p. 15

H. Visual Analysis of the Textual Intra-modal Contrastive Loss

The visual analysis reports improved separation of similar textual phrases with TIC, illustrates the descriptive range of LIT-CN captions, and presents BoxClass-CN categories and examples.

H. Visual Analysis of the Textual Intra-modal Contrastive Loss

I. Examples of LIT-CN

J. BoxClass-CN Category Schema and Example Explanation

Loading 2510.10921v3…