Source-linked AI summary
FILIP: Fine-grained Interactive Language-Image Pre-Training
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, Chunjing Xu
TL;DR
Existing vision-language pre-training either relies on efficient but coarse global alignment or finer-grained attention mechanisms with lower efficiency. FILIP introduces token-wise maximum-similarity late interaction in the contrastive loss, retaining offline modality precomputation, and reports state-of-the-art results across downstream tasks. Its architecture and training procedure remain open to further optimization.
Problem
Existing methods use either global feature similarity that misses finer-grained information or cross/self-attention with inferior training and inference efficiency.
Method
FILIP is a dual-stream model that modifies contrastive learning with cross-modal late interaction using token-wise maximum similarity between image patches and textual words.
Results
FILIP achieves state-of-the-art performance across multiple downstream vision-language tasks, including zero-shot image classification and image-text retrieval.
Takeaways & Limitations
FILIP learns fine-grained patch-word representations while preserving offline image and text precomputation for efficient inference.
Takeaways & Limitations
The architecture and training procedure can still be optimized, and future work targets improved encoders, interaction layers, and additional masked losses for generation tasks.
Abstract
from arXiv · showhide
Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. FILIP successfully leverages the finer-grained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability to pre-compute image and text representations offline at inference, keeping both large-scale training and inference efficient. Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks including zero-shot image classification and image-text retrieval. The visualization on word-patch alignment further shows that FILIP can learn meaningful fine-grained features with promising localization ability.
1 INTRODUCTION
FILIP addresses the tension between fine-grained vision-language alignment and efficient training and inference by replacing attention-based interaction with cross-modal late interaction. It achieves strong downstream performance while preserving offline precomputation of image and text representations.
- Motivation: Global-feature contrastive alignment enables efficient dual-stream inference but misses finer-grained cross-modal information.Image and text encoders can be decoupled and representations pre-computed offline, but interaction is based only on global features.
- Motivation: Region-feature methods complicate pre-training through ROI extraction and storage, while their zero-shot ability is limited by predefined object regions.
- Motivation: Cross- and self-attention capture finer-grained interactions but are less efficient during training and inference.Cross-attention requires an encoder-decoder structure, while self-attention complexity grows with concatenated sequence length.
- FILIP: FILIP uses token-wise maximum similarity between visual and textual tokens in the contrastive loss instead of cross- or self-attention.The mechanism aligns image patches and textual words while retaining offline computation of image and text representations.
- Results: 77.1% top-1 accuracy on zero-shot ImageNet classification accompanies state-of-the-art results across multiple vision-language tasks.The reported tasks include zero-shot image classification and image-text retrieval, while word-patch visualizations show promising localization ability.
2 RELATED WORK
Vision-language pre-training uses distinct interaction architectures, with dual-stream models supporting flexible and efficient inference through separate modality encoders. FILIP retains this dual-stream structure while adding fine-grained cross-modal interaction.
- Vision-Language Pre-training: Vision-language pre-training extends the pre-train-and-fine-tune paradigm to joint cross-modal learning using datasets ranging from public collections to more than 100M samples.
- Multi-Modality Interaction Mechanism: Single-stream models concatenate visual and textual features before transformer processing, whereas dual-stream models use separate encoders for each modality.
- FILIP: Figure 1 depicts FILIP as a dual-stream architecture with Transformer-based image and text encoders and token representations projected into a shared multimodal space.
- FILIP: FILIP preserves dual-stream flexibility and efficient retrieval inference while introducing a new interaction mechanism for fine-grained representations.Separate encoders allow image and text features to be pre-computed offline.
3 METHOD
FILIP is a dual-stream vision-language model that captures fine-grained image-text alignment through cross-modal late interaction while retaining efficient inference. Its method combines token-wise similarity, prompt-based evaluation, data augmentation, and the FILIP300M pre-training dataset.
- Model Architecture: FILIP uses separate Transformer-based image and text encoders whose token representations are projected into a shared, L2-normalized multimodal space.The image encoder processes a [CLS] token and projected image patches, while the text encoder uses BPE tokenization and a decoder-only Transformer.
- Fine-Grained Contrastive Learning: The resulting image-to-text and text-to-image similarities guide the contrastive loss, allowing the dual-stream model to learn fine-grained patch-word alignment.The two directional similarities need not be equal because each modality independently selects its closest tokens in the other modality.
- Efficiency: FILIP preserves offline feature pre-computation and efficient inference while modeling finer-grained interactions than global-feature methods.Its dual-stream design retains modality-specific encoders, while late interaction is applied through the contrastive similarity calculation rather than cross- or self-attention.
- Cross-Modal Late Interaction: Cross-modal late interaction replaces global similarity with token-wise maximum similarities between visual patches and textual tokens.For each image patch or text token, FILIP finds the closest token in the other modality and averages these maxima over non-padded tokens.
- Prompt Ensemble and Templates: Prompt templates augment downstream classification labels with prefixes, category descriptions, and optional suffixes to support fine-grained alignment.The paper reports that adding a suffix containing the reference word “it” empirically improves zero-shot classification performance.
- Pre-Training Dataset: FILIP300M supplies about 340M filtered image-text pairs, combining Internet-collected data with CC3M, CC12M, and YFCC100M.Filtering removes low-resolution or extreme-aspect-ratio images, non-English or meaningless text, and pairs with excessively repeated captions; image and text augmentation further expands training pairs.
4 EXPERIMENTS
FILIP is evaluated across zero-shot classification, image-text retrieval, component ablations, efficiency, and fine-grained alignment visualizations. It improves performance over CLIP and preserves practical efficiency through optimized late interaction.
- Zero-shot image classification: FILIPbase and FILIPlarge outperform CLIP on average top-1 accuracy across 12 zero-shot classification datasets despite using less training data.The absolute improvements are 5.6% and 3.0%, respectively.
- Zero-shot image classification: 30% improvement over CLIP is achieved on Aircrafts on average by the two FILIP models.
- Image-text retrieval: FILIP achieves state-of-the-art performance under all retrieval metrics on Flickr30K and MSCOCO except zero-shot text-to-image retrieval on Flickr30K.For zero-shot image-to-text retrieval on MSCOCO, FILIP’s absolute R@1 is 2.7% higher than ALIGN.
- Ablation study: Cross-modal late interaction adds 5.5% image-to-text R@1, 3.8% text-to-image R@1, and 3.9% ImageNet top-1 accuracy over vanilla CLIP ViT-B/32.The ablation evaluates MSCOCO zero-shot retrieval and ImageNet zero-shot classification.
- Efficiency study: Combining all three efficiency attempts makes late interaction only slightly slower and more memory-intensive than CLIP’s original loss, without an accuracy drop.The efficiency study targets the communication, memory, and computation costs of token-wise similarity.
- Fine-grained alignment visualization: FILIP aligns patches with textual tokens for single objects, differently shaped instances, and key components more accurately than CLIP in ImageNet visualizations.For electric locomotive, patches matching “electric” and “locomotive” are aligned to their respective token indices.
5 CONCLUSION AND FUTURE WORK
FILIP introduces token-wise maximum similarity for fine-grained vision-language pre-training and learns representations for image patches and sentence words. The authors report competitive downstream results while identifying architecture, interaction, and generative-task extensions for future work.
- FILIP uses token-wise maximum similarity to learn fine-grained representations for image patches and words in sentences.
- FILIP achieves competitive results against several large-scale multimodal pre-training methods across various downstream tasks.
- Future work includes improving the image encoder and interaction layer and adding masked language/image losses for more generation tasks.
A.1 DATASETS SUMMARY
Table 6 reports the numbers of image-text pairs used to pre-train FILIP, CLIP, and ALIGN across their respective datasets.
- Table 6 compares the image-text pair counts used for pre-training FILIP, CLIP, and ALIGN.
A.2 DETAILED EXPERIMENTAL SETTINGS
FILIP follows CLIP-like architectures with reduced embedding dimensions and uses specified optimization, scheduling, regularization, retrieval, and model-specific settings.
- Model Architectures: FILIPbase and FILIPlarge follow CLIP’s architecture design while reducing embedding dimensions from 512/768 to 256 for efficient loss computation.
- Details for Pre-training and Hyperparameters: The contrastive-loss temperature is learnable and initialized to 0.07, while pre-training uses the LAMB optimizer.
- Details for Pre-training and Hyperparameters: The learning-rate schedule assigns a base learning rate and scales the peak rate using a square-root strategy based on effective total batch size.
- Details for Pre-training and Hyperparameters: Large weight decay is used to stabilize mixed-precision training and improve generalization, with specific parameters exempted from weight decay.
- Details for Pre-training and Hyperparameters: Table 9 provides model- and dataset-specific pre-training hyperparameters, including total batch sizes across workers.
- Details for Image-text Retrieval: Image-text retrieval is evaluated on 1K Flickr30K and 5K MSCOCO test sets, with optional fine-tuning on their training sets.
A.3 MORE VISUALIZATIONS OF WORD-PATCH ALIGNMENT AND GRAD-CAM HEATMAPS
Figure 3 visualizes FILIP’s cross-modal alignment through word-patch alignment and Grad-CAM heatmaps, showing meaningful relationships between image patches and textual tokens.
- Figure 3 combines word-patch alignment with Grad-CAM heatmaps to visualize FILIP’s cross-modal alignment.
- The heatmaps average self-attention maps over image patches associated with targeted class-label tokens and across all attention heads.
- The visualizations show meaningful alignment between image patches and textual tokens.
A.4 PROMPT TEMPLATES FOR DOWNSTREAM TASKS
Downstream image classification and image-text retrieval use prompt templates, with classification templates combining prefixes, labels, category descriptions, and suffixes.
- Image Classification: Image classification prompts combine a prefix, class label, category description, and suffix, with full combinations ensembled across selected components.
- Image-text Retrieval: Zero-shot image-text retrieval uses prompts for Flickr30K and MSCOCO without a category-description component.
- Image Classification: Figure 3 visualizations cover different ImageNet classes, whose label-token positions are indicated in parentheses after each class label.