PaperlayerView exact revision on arXiv

Source-linked AI summary

FG-CLIP: Fine-Grained Visual and Textual Alignment

Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, Yuhui Yin

arXiv:2505.05071v3cs.CVcs.AI

TL;DR

CLIP’s short-caption and whole-image alignment design limits fine-grained image-text understanding and regional representation learning. FG-CLIP combines billion-scale long captions, region-specific grounding, and hard negatives through FineHARD, and reports superior performance across diverse downstream tasks. The results support its effectiveness in capturing nuanced visual details and improving multimodal model performance.

  • Problem

    CLIP’s 77-token text limit, whole-image alignment, and scarcity of hard negatives constrain fine-grained visual-textual alignment and subtle distinction learning.

  • Method

    FG-CLIP uses 1.6 billion long caption-image pairs, region-specific captions covering 40 million bounding boxes across 12 million images, and 10 million hard negatives integrated as FineHARD.

  • Results

    FG-CLIP reports significant improvements over previous methods across fine-grained understanding, retrieval, bounding box classification, open-vocabulary detection, and multimodal benchmarks.

  • Takeaways & Limitations

    FG-CLIP captures global- and region-level semantic details and distinguishes subtle differences more effectively across diverse downstream tasks.

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks such as image-text retrieval and zero-shot classification but struggles with fine-grained understanding due to its focus on coarse-grained short captions. To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances fine-grained understanding through three key innovations. First, we leverage large multimodal models to generate 1.6 billion long caption-image pairs for capturing global-level semantic details. Second, a high-quality dataset is constructed with 12 million images and 40 million region-specific bounding boxes aligned with detailed captions to ensure precise, context-rich representations. Third, 10 million hard fine-grained negative samples are incorporated to improve the model's ability to distinguish subtle semantic differences. We construct a comprehensive dataset, termed FineHARD, by integrating high-quality region-specific annotations with hard fine-grained negative samples. Corresponding training methods are meticulously designed for these data. Extensive experiments demonstrate that FG-CLIP outperforms the original CLIP and other state-of-the-art methods across various downstream tasks, including fine-grained understanding, open-vocabulary object detection, image-text retrieval, and general multimodal benchmarks. These results highlight FG-CLIP's effectiveness in capturing fine-grained image details and improving overall model performance. The data, code, and models are available at https://github.com/360CVGroup/FG-CLIP.

1. Introduction

FG-CLIP addresses CLIP’s fine-grained limitations by combining large-scale long captions, region-specific grounding, and hard negatives. It reports improvements across fine-grained understanding, retrieval, detection, and multimodal benchmarks.

  • Motivation: CLIP’s 77-token text limit and image-level alignment hinder detailed descriptions and region-specific representations.These constraints make fine-grained alignment between visual regions and textual attributes difficult.
  • Contributions: 1.6 billion long caption-image pairs provide global-level semantic details for fine-grained alignment.Large multimodal models generate the long captions used for this training signal.
  • Contributions: 40 million bounding boxes across 12 million images supply region-specific, context-rich captions for precise visual grounding.The dataset is designed to improve representations needed for fine-grained understanding.
  • Contributions: 10 million hard fine-grained negative samples train FG-CLIP to distinguish subtle differences between semantically similar pairs.These samples are integrated with visual grounding data into FineHARD.
  • Results: FG-CLIP achieves state-of-the-art results in fine-grained understanding, bounding box classification, long-caption retrieval, and open-vocabulary object detection.As an LMM backbone, it also improves attribute analysis, object localization, and hallucination reduction.

2. Related Work

Related work identifies limitations in CLIP and existing image-text datasets, especially for detailed descriptions, regional representations, and hard negative discrimination. FG-CLIP responds with two-stage global and regional training using richer captions and challenging negatives.

  • CLIP limitations: CLIP uses a dual-encoder shared embedding space but struggles with fine-grained visual details.Its text encoder accepts only 77 tokens, and whole-image alignment limits region-specific feature extraction.
  • FG-CLIP: FG-CLIP’s first training stage aligns global image and text representations using global contrastive learning.The second stage adds regional contrastive learning and hard-negative learning with region-text data.
  • Image-text datasets: Existing datasets such as LAION, COCO, Flickr30K, and Conceptual Captions often emphasize general scene descriptions over fine-grained details.This limits their direct usefulness for advanced fine-grained applications.
  • Image-text datasets: Hard fine-grained negatives are scarce because existing datasets predominantly contain positive examples that are relatively easy to distinguish.Their absence limits learning of subtle visual and textual variations.

3. Approach

FG-CLIP uses staged global, regional, and hard-negative contrastive learning, supported by detailed recaptioning and the FineHARD dataset, to improve fine-grained image-text alignment.

  • Training framework: FG-CLIP uses two-stage training: global contrastive learning first, followed by regional contrastive learning and hard-negative learning.The second stage integrates all three loss components.
  • Global contrastive learning: Large multimodal models generate long, context-rich captions alongside short captions to capture global semantic details.Both caption types are aligned with image representations during training.
  • Regional contrastive learning: Regional contrastive learning aligns RoIAlign-derived region embeddings with text phrases or sentences corresponding to individual bounding boxes.This differs from global learning, which uses image-level [CLS] features.
  • Hard-negative learning: Hard negatives are created by modifying attributes while preserving object names, encouraging discrimination between semantically close but distinct region-text pairs.The hard-negative loss compares each regional feature with its positive and corresponding negative descriptions; M captions are used per region, with j = 1 positive and j > 1 negative.
  • FineHARD dataset: FineHARD combines 12 million images, 40 million detailed bounding-box descriptions, and 10 million hard-negative samples for second-stage training.The dataset is built from GRIT images with generated captions, parsed referring expressions, and region-specific annotations.

4. Experiments

FG-CLIP is evaluated across fine-grained understanding, regional classification, open-vocabulary detection, retrieval, zero-shot classification, and multimodal benchmarks. Ablations show that regional contrastive learning and hard negative learning improve complementary fine-grained capabilities.

  • Fine-Grained Understanding: FG-CLIP evaluates local understanding using FG-OVD, where each region is matched against one positive and ten negative descriptions across four difficulty levels.The hard, medium, easy, and trivial subsets replace one, two, three, or all attribute words to vary text similarity.
  • Fine-Grained Understanding: FG-CLIP achieves significant improvements particularly on the hard and medium FG-OVD subsets.The reported gains are attributed to its hard fine-grained negative samples learning strategy.
  • Bounding Box Classification: FG-CLIP achieves leading performance in bounding box classification across COCO-val2017, LVIS, and Open Images.Regional contrastive learning supports localized dense representations, while combining regional and global information outperforms FineCLIP.
  • Open-Vocabulary Object Detection: FG-CLIP achieves leading performance in open-vocabulary detection, with AP novel_50 emphasized because it measures recognition of novel objects.The evaluation reports box AP at IoU 0.5 for base, novel, and all categories.
  • Image-level and General Multimodal Tasks: FG-CLIP improves performance across long- and short-caption retrieval, zero-shot classification, and general multimodal benchmarks.It remains marginally behind EVA-CLIP on zero-shot classification, while improving over CLIP, Long-CLIP, and FineCLIP in the reported comparisons.
  • Ablation Study: 46.1% versus 24.5% is the FG-OVD hard-dataset accuracy with hard-negative learning, while medium accuracy rises from 47.1% to 66.6% and easy accuracy from 49.5% to 68.7%.Adding regional contrastive learning separately increases bounding box classification accuracy from 46.8% to 53.7% and FG-OVD easy accuracy from 42.9% to 49.5%.

5. Conclusion

FG-CLIP advances fine-grained understanding by combining advanced alignment techniques with large-scale, high-quality datasets and hard negative samples. Extensive experiments validate its superior performance across diverse downstream tasks, while FineHARD provides a unified resource for multimodal research.

  • FG-CLIP combines advanced alignment techniques, large-scale high-quality datasets, and hard negative samples to capture global- and region-level semantic details.The model is designed to distinguish subtle differences more effectively.
  • Extensive experiments across diverse downstream tasks validate FG-CLIP’s superior performance.
  • FineHARD unifies high-quality region-specific annotations with challenging fine-grained negative samples as a resource for multimodal research.
  • Expanding dataset diversity and integrating more sophisticated multimodal models are identified as future directions for fine-grained understanding.

Impact Statement

The paper states that its work has broad societal implications but does not identify any specific consequences that require highlighting.

  • The work is presented as advancing machine learning with broad implications for society, without specific consequences requiring emphasis.

A. Examples of Curated Visual Grounding Data

The curated visual grounding examples pair images with long and short captions plus detailed region-specific annotations. Hard negatives alter attributes while preserving object identities to create subtle semantic contrasts.

  • Curated examples combine an image, long and short captions, and multiple region-specific annotations with detailed descriptions.
  • The examples include global descriptions of scenes such as sewing projects alongside localized descriptions of objects and regions.The supplied examples describe sewing equipment, fabric, a cutting mat, a plant pot, and a baby hat.
  • Hard fine-grained negatives modify bounding-box attributes while keeping object names unchanged.Figure 3 presents positive and corresponding negative descriptions for image regions.
  • Negative descriptions create subtle contrasts by changing attributes such as material, color, or configuration.

C. Visualization Comparison

The visualization comparisons examine similarity patterns and localization across challenging images and text inputs. FG-CLIP more precisely identifies fine-grained targets, while hard-negative learning further improves region capture.

  • Figure 4 compares similarity matrix visualizations for different methods on challenging sample images.Warmer colors indicate higher similarity, while cooler colors indicate lower relevance.
  • FG-CLIP identifies more relevant tokens for “Black dog” than CLIP, EVA-CLIP, and FineCLIP in a multi-dog image.
  • For the small “Black nose” target, FG-CLIP localizes the target accurately without responding broadly to other black regions.
  • FG-CLIP more accurately distinguishes the red gemstone from differently colored gemstones than the compared methods.
  • Correlation analysis across different input texts indicates precise positional understanding and stable visual localization within the same image.
  • Hard-negative learning improves region capture, including localization of “Man in red clothes” compared with training without hard-negative loss.

D.1. Comparison of Different Methods on Fine-Grained Benchmark

Table 7 visualizes region-text similarity comparisons on selected FG-OVD test samples, showing that FG-CLIP identifies extremely difficult examples more effectively than other methods.

  • The comparison matches text with localized dense features using the testing strategy detailed in Section 4.2.
  • Similarity scores are normalized from 0.0 for the lowest-scoring sentence to 1.0 for the highest-scoring sentence.
  • FG-CLIP identifies extremely difficult FG-OVD test samples, while other methods struggle to achieve comparable performance.

D.2. Performance Comparison on Identical Datasets

Table 8 compares methods on the same dataset, with results indicating that both the proposed dataset and model architecture contribute substantially to FG-CLIP's improvements.

  • Table 8 reports comparisons of different methods on the same dataset.
  • Experiments compare FineCLIP and FG-CLIP on the 12M dataset to ensure a fair comparison under time constraints.The larger 1.6B+12M setup is not used in this comparison.
  • Substantial improvements from Row 1 to Row 2 and Row 2 to Row 3 highlight the importance of both the proposed dataset and model architecture.

D.3. Performance on General Multimodal Benchmarks

FG-CLIP is evaluated on general multimodal benchmarks beyond GQA, POPE, and RefCOCO, where LLaVA with FG-CLIP achieves better performance.

  • Table 9 presents comparisons on general multimodal benchmarks.
  • Experiments extend beyond GQA, POPE, and RefCOCO to other general multimodal benchmarks.
  • LLaVA with FG-CLIP achieves better performance on the general multimodal benchmarks reported in Table 9.
Loading 2505.05071v3…