Source-linked AI summary
Long-CLIP: Unlocking the Long-Text Capability of CLIP
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Jiaqi Wang
TL;DR
CLIP’s short effective text context limits detailed image-text understanding and retrieval, motivating a longer-context alternative. Long-CLIP efficiently fine-tunes CLIP with representation-preserving strategies, improving retrieval while retaining zero-shot classification and plug-and-play compatibility.
Problem
CLIP is restricted to 77 input tokens and has an empirical effective length of only 20 tokens, limiting detailed descriptions and attribute-relationship modeling.
Method
Long-CLIP stretches positional embeddings while preserving trained positions and matches primary coarse- and fine-grained CLIP feature components during efficient fine-tuning.
Results
Long-CLIP surpasses CLIP on long- and short-text retrieval, shows no zero-shot classification decay, and supports up to 248-token inputs.
Takeaways & Limitations
Long-CLIP can replace CLIP in downstream image-generation frameworks in a plug-and-play manner without additional training.
Takeaways & Limitations
Long-CLIP still has an upper bound on input token length, despite substantially increasing it.
Abstract
from arXiv · showhide
Contrastive Language-Image Pre-training (CLIP) has been the cornerstone for zero-shot classification, text-image retrieval, and text-image generation by aligning image and text modalities. Despite its widespread adoption, a significant limitation of CLIP lies in the inadequate length of text input. The length of the text token is restricted to 77, and an empirical study shows the actual effective length is even less than 20. This prevents CLIP from handling detailed descriptions, limiting its applications for image retrieval and text-to-image generation with extensive prerequisites. To this end, we propose Long-CLIP as a plug-and-play alternative to CLIP that supports long-text input, retains or even surpasses its zero-shot generalizability, and aligns the CLIP latent space, making it readily replace CLIP without any further adaptation in downstream frameworks. Nevertheless, achieving this goal is far from straightforward, as simplistic fine-tuning can result in a significant degradation of CLIP's performance. Moreover, substituting the text encoder with a language model supporting longer contexts necessitates pretraining with vast amounts of data, incurring significant expenses. Accordingly, Long-CLIP introduces an efficient fine-tuning solution on CLIP with two novel strategies designed to maintain the original capabilities, including (1) a knowledge-preserved stretching of positional embedding and (2) a primary component matching of CLIP features. With leveraging just one million extra long text-image pairs, Long-CLIP has shown the superiority to CLIP for about 20% in long caption text-image retrieval and 6% in traditional text-image retrieval tasks, e.g., COCO and Flickr30k. Furthermore, Long-CLIP offers enhanced capabilities for generating images from detailed text descriptions by replacing CLIP in a plug-and-play manner.
1 Introduction
CLIP’s text input is constrained by a 77-token positional embedding, with an effective length of only 20 tokens, limiting detailed text processing. Long-CLIP addresses this through efficient fine-tuning that preserves CLIP representations while extending text capacity and improving retrieval performance.
- Motivation: CLIP’s positional embedding imposes a 77-token ceiling, while its effective text length is only 20 tokens.Brief-text training leaves higher token positions inadequately trained.
- Motivation: Limited long-text capability hampers CLIP’s handling of fine-grained attributes and relationships in images.CLIP’s summary-text training can cause the image encoder to disregard details and model attributes as a ‘bag of concepts’.
- Challenges: Naive positional interpolation and fine-tuning degrade CLIP’s short-text performance and shift its feature space.ImageNet zero-shot classification accuracy decreases by 13.1%, while COCO T2I R@1 decreases by 14.4%.
- Approach: Long-CLIP uses knowledge-preserved positional-embedding stretching and primary component matching to support long text while retaining CLIP capabilities.The method keeps the first 20 positional embeddings and aligns coarse-grained information with short captions alongside fine-grained information with detailed captions.
- Results: 25% higher recall on long-text image retrieval and 6% higher recall on short-text image retrieval are reported for Long-CLIP versus original CLIP.The model also shows no decay on zero-shot classification and can replace CLIP in image-generation models without additional training.
2 Related Works
CLIP aligns image and text modalities and supports broad zero-shot applications, but its effective text length is limited and it struggles with fine-grained attributes and their relationships. Long-CLIP addresses these limitations through knowledge-preserved positional stretching and primary component matching.
- CLIP and its limitations: CLIP is a contrastive vision-language model used for zero-shot classification, retrieval, generation, detection, segmentation, and video understanding.It learns matching relationships from large-scale text-image pairs.
- CLIP and its limitations: CLIP’s R@1 grows slowly beyond 20 input tokens, indicating an effective text length of no more than 20 tokens.The experiment uses the urban-200 long-caption image-text evaluation dataset.
- CLIP and its limitations: CLIP’s learned absolute positional embedding imposes a 77-token ceiling, while higher positions are inadequately trained because its data mostly contains short texts.These conditions make detailed descriptions difficult to process.
- Knowledge-preserved stretching: Long-CLIP retains the first 20 positional embeddings and interpolates the remaining 57 positions with a larger ratio to extend context while preserving trained short-text positions.The method targets the trade-off between longer input support and preserving established positional representations.
- Primary Component matching: Primary Component matching aligns fine-grained image features with long captions and coarse-grained features with short summaries.Its decomposition, filtration, and reconstruction functions select attributes and their importance for coarse-grained image features.
4 Experiments
Long-CLIP is evaluated on zero-shot classification, short- and long-caption retrieval, ablations, and plug-and-play image generation. Across these experiments, it improves fine-grained retrieval, preserves short-text and classification capabilities, and captures details from longer prompts.
- Comparing with CLIP Model: Long-CLIP uses knowledge-preserved positional stretching and primary component matching, with a maximum input length extended to 248 tokens.The experiments also compare against fixed-ratio positional interpolation and direct fine-tuning using only long text-image pairs.
- Comparing with CLIP Model: Long-CLIP outperforms its baseline in most fine-grained retrieval aspects by representing detailed information across multiple attributes.The method aligns fine-grained image features with long captions while preserving coarse-grained information for short captions.
- Ablation Study: Removing either knowledge-preserved stretching or primary component matching causes a significant loss of short-text capabilities.The ablation compares each strategy with fixed-ratio interpolation or simple long-caption/image fine-tuning.
- Ablation Study: Alternative short-text-preservation strategies are less effective because they do not require identifying the differing importance of image attributes.Undistinguished Image Feature and Mixed-length Text produce a significant performance drop on long-text retrieval when the same image feature is aligned with both captions.
- Plug-and-Play in Image Generation using CLIP: Replacing Stable Diffusion V1-5’s original CLIP text encoder with Long-CLIP requires no additional training and captures details beyond CLIP’s 77-token limit.For short prompts, generated images closely resemble those from the original model; for detailed or long prompts, Long-CLIP captures previously overlooked information.
5 Conclusion
Long-CLIP supports substantially longer text while preserving zero-shot classification and plug-and-play image-generation use. Its remaining limitation is a finite input-length bound, and its scaling potential is constrained by scarce long text-image data.
- Long-CLIP supports text inputs of up to 248 tokens and captures detailed attributes more effectively on retrieval tasks.
- Long-CLIP preserves zero-shot classification performance and can replace CLIP in image-generation systems through a plug-and-play interface.
- Long-CLIP still has an upper bound on input token length, although its limit is substantially improved.
- Only 1M long text-image pairs were used because such data are scarce, leaving larger-scale training as a potential avenue for improvement.
1 Urban-1k Dataset
Urban-1k is a scaled-up urban image-caption dataset with GPT-4V-generated descriptions. Its table reports results on the dataset, with the best result marked in bold.
- Urban-1k is a scaling-up version of the Urban-200 dataset.
- The dataset contains 1k urban images paired with captions generated by GPT-4V.
- Each Urban-1k caption contains about 107 words on average.
- Table 1 reports results on Urban-1k, with the best result shown in bold.
2 Insufficient Long-text Ability for CLIP-based Models
The paper evaluates phrase-region alignment methods for long-caption understanding on an urban dataset. These methods use contrastive learning to associate caption phrases with corresponding image patches.
- Recent models such as PTP-BLIP and X-VLM align text phrases with corresponding image regions through contrastive learning.
- The evaluated approaches aim to improve the ability to capture fine-grained information by matching caption phrases with image patches.
- The paper evaluates these models on an urban dataset containing long captions.
3 Long-CLIP with SDXL
Long-CLIP is integrated into Stable-Diffusion-XL by replacing its CLIP-L text encoder, while applying knowledge-preserved stretching to the Open-CLIP bigG encoder.
- Long-CLIP replaces the CLIP-L text encoder in Stable-Diffusion-XL with Long-CLIP-L.
- Knowledge-preserved stretching is applied to the Open-CLIP bigG text encoder because its training cost is high.
- Stable-Diffusion-XL can exceed the 77-token limit with little reduction in image quality using this integration.
4 Generalizability of Proposed Strategy
Applying knowledge-preserved stretching and primary component matching to DeCLIP ViT-B/32 produces a consistent improvement.
- DeCLIP ViT-B/32 shows a consistent improvement after fine-tuning with knowledge-preserved stretching and primary component matching.
5 Detailed Experiment Setting
The experiments use ShareGPT4V for long-text fine-tuning and evaluate classification and retrieval with standard CLIP procedures and metrics.
- ShareGPT4V provides about 1M image–long caption pairs for fine-tuning the CLIP model.
- Fig. 1 reports SDXL breaking the 77-token limit with little reduction in image quality.
- Zero-shot classification averages embeddings from 80 predefined CLIP prompts for each class before L2 normalization and cosine-similarity classification.
- Table 4 lists the detailed hyper-parameters used for long-text fine-tuning.
- Text-image retrieval ranks images or captions by cosine similarity between L2-normalized embeddings and evaluates Recall@K for K=1, 5, and 10.
6 More Examples on Retrieval and Image Generation
Examples show Long-CLIP capturing detailed information in images and text to improve retrieval and cover more prompt details during image generation.
- Long-CLIP captures detailed information in both modalities, helping distinguish similar images and texts and improve retrieval accuracy.
- Fig. 2 presents retrieval examples where CLIP fails but Long-CLIP successfully retrieves the relevant image or text.
- Fig. 3 shows Long-CLIP covering more details from text prompts during image generation than the CLIP baseline.