PaperlayerView exact revision on arXiv

Source-linked AI summary

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, Pascale Fung

arXiv:2203.06386v2cs.CLcs.AIcs.CV

TL;DR

Dual-stream VLP models such as CLIP excel at multimodal alignment, but their text encoders are limited on generative multimodal tasks. The paper uses VLKD to align CLIP with BART, reporting zero-shot multimodal generation performance while retaining BART’s language abilities.

  • Problem

    Dual-stream VLP models learn strong multimodal representations but their relatively weak text encoders limit their use for generative tasks such as image captioning and open-ended VQA.

    Source: §1 · p. 1

  • Method

    VLKD aligns BART’s encoder with CLIP’s joint multimodal embedding space and uses image-conditioned language modeling to connect BART’s encoder and decoder.

    Source: §1 · p. 2

  • Results

    44.5% zero-shot accuracy on VQAv2 accompanies strong zero-shot performance on multimodal generation tasks, while BART’s original NLU and NLG abilities are maintained.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model retains BART’s language abilities, making it versatile for multimodal and unimodal tasks.

    Source: §1 · p. 2

Abstract

from arXiv · show

The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with $7\times$ fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.

1 Introduction

Dual-stream VLP models learn strong multimodal representations but their text encoders are weak for generative tasks. The paper introduces VLKD to pair CLIP’s multimodal space with BART’s language capabilities, reporting zero-shot generation results and retained BART language abilities.

1 Introduction

  • Dual-stream VLP models learn strong multimodal representations through contrastive pre-training, but their text encoders are weak for generative tasks such as captioning and open-ended VQA.The passage attributes this limitation to their discriminative multimodal pre-training objective.Source: §1 · p. 1
  • VLKD aligns BART’s encoder with CLIP’s joint multimodal embedding space and uses image-conditioned language modeling to connect BART’s encoder and decoder.CLIP’s weights are frozen during training, and its text encoder is discarded for downstream finetuning and inference.Source: §1 · p. 2
  • 44.5% zero-shot accuracy on VQAv2 and 84.6 CIDEr on COCO show VLKD’s performance on multimodal generation tasks.The model also retains BART’s original NLU and NLG abilities.Source: §1 · p. 2
  • The paper reports an efficient CLIP-to-BART distillation approach, evaluation across six benchmarks, and analyses and ablations intended to inform future work.The introduction also illustrates masked-word image description and question answering using visual information and PLM knowledge.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Related work covers vision-language pre-training architectures and knowledge-distillation approaches. The paper positions VLKD as a way to align CLIP’s multimodal space with BART for multimodal generation.

2.1 Vision-language Pre-training

  • Dual-stream VLP models use separate image and text encoders for efficient downstream alignment tasks, but cannot be directly used for multimodal generation.Prior generative VLP approaches use image-conditioned autoregressive language modeling, while some other approaches rely on labeled-region object detectors and are described as less scalable.Source: §2.1 · p. 2

2.2 Knowledge Distillation

  • Knowledge distillation transfers teacher-model knowledge through logits, intermediate representations, attention, or contrastive representations.Prior work also extended contrastive representation distillation to transfer knowledge from CLIP for zero-shot detection and multilingual settings.Source: §2.2 · pp. 2–3
  • This paper applies conventional and contrastive distillation to transfer knowledge from CLIP to BART and uses cross-attention to transfer CLIP image-encoder knowledge to the BART decoder.Source: §2.2 · p. 3

3 Proposed Method

VLKD transfers CLIP’s multimodal representations to BART, combining encoder alignment with image-conditioned text infilling while keeping CLIP frozen. The method uses three objectives and trains on image-text data substantially smaller than prior VLP work.

3 Proposed Method

  • VLKD transfers multimodal knowledge from CLIP to BART, combining CLIP’s multimodal representations with BART’s text-generation ability.The architecture uses three objectives to align and coordinate the models.Source: §3 · p. 3, §3.2 · p. 3

3.1 Model Architecture

  • CLIP provides aligned image and text encoders, while BART pairs a bidirectional encoder with a left-to-right decoder.BART is pretrained with text span infilling and supports both natural-language generation and understanding.Source: §3.1 · p. 3

3.2 Training Objectives

  • Training uses TTDM, ITCL, and ICTI while keeping CLIP frozen to preserve the alignment of its encoders.The losses align BART’s encoder with CLIP and train BART’s encoder-decoder through image-conditioned language modeling.Source: §3.2 · p. 3, Figure 2 · p. 4

3.2.1 Text-Text Distance Minimization

  • TTDM minimizes the L2 distance between CLIP’s text representation and BART’s projected, normalized sequence-level representation.The projection maps BART encoder outputs into CLIP’s multimodal space.Source: §3.2.1 · pp. 3–4

3.2.2 Image-Text Contrastive Learning

  • ITCL uses a symmetric InfoNCE loss to align BART encoder representations with CLIP image representations.The method avoids adding projections to CLIP features and instead projects BART outputs into CLIP’s representation space.Source: §3.2.2 · p. 4

3.2.3 Image-Conditioned Text Infilling

  • ICTI trains BART to infill masked text conditioned on projected CLIP image and text features.It masks 15% of whole-word tokens, with span lengths drawn from a Poisson distribution with λ = 3; the decoder attends to the concatenated features and predicts tokens autoregressively.Source: §3.2.3 · pp. 4–5

3.3 Datasets for VLKD

  • VLKD trains on CC3M’s 3 million image-text pairs, adding about 700K Visual Genome Caption pairs for larger model variants.The authors also test subsets of 1M and 100K CC3M pairs; downstream datasets do not overlap with pretraining images.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate multimodal generation and assess whether VLKD affects BART’s text-processing abilities. The finetuning datasets cover image captioning, open-ended VQA, and NLU and NLG tasks.

4.1 Finetuning Datasets

  • Image captioning is tested on COCO with the Karpathy split and on NoCaps to assess performance on out-of-domain objects.Image captioning generates a relevant description given an image.Source: §4.1 · p. 5
  • Open-ended VQA is evaluated on VQAv2 and OK-VQA, with the latter requiring knowledge to answer questions correctly.The evaluation lets the model generate answers freely rather than treating VQA as a discriminative task.Source: §4.1 · p. 5
  • NLU is evaluated on GLUE, excluding WNLI, while NLG is evaluated on abstractive summarization using XSUM.GLUE comprises nine text-classification tasks; XSUM requires long-text comprehension and short summaries containing key information.Source: §4.1 · p. 5

4.2 Implementation Details

The zero-shot evaluations test image captioning and open-ended VQA, alongside implementation details for the VLKD model. VLKD performs strongly on these generation tasks, including a reported VQAv2 test-dev result of 44.5%.

4.2 Implementation Details

  • Implementation uses BART-large with a CLIP visual backbone; VLKD pretraining uses 64 V100 GPUs for 10 epochs, taking about five hours.The reported CLIP variants use ViT-B/16, ViT-L/14, RN50×16, and RN50×64 backbones.Source: §4.2 · p. 5

4.3.1 Zero-Shot Image Captioning

  • Zero-shot image captions are reported as plausible, with correct objects, relationships, and actions, and overall performance comparable to finetuned UpDown on NoCaps.The authors note that caption details such as colors may sometimes be omitted.Source: §4.3.1 · p. 6
  • For zero-shot captioning, the model receives a text prompt and mask tokens; experiments use six masks for COCO and eight for NoCaps.The authors report negligible performance impact from these mask counts and say they can control generated-text length.Source: §4.3.1 · p. 6

4.3.2 Zero-Shot VQA

  • 44.5% zero-shot accuracy on VQAv2 test-dev is reported as a new state of the art; versus Frozen, accuracy improves by 13.1% on VQAv2 validation and 7.4% on OK-VQA test with 7× fewer parameters.The comparison attributes the results to the VLKD model.Source: §4.3.2 · p. 6

4.4 Multimodal Finetuning Evaluation

Finetuning evaluation applies VLKD to image captioning and generative VQA while keeping CLIP frozen. The reported findings include competitive captioning performance and an advantage on out-of-domain VQA questions with rare answers.

4.4 Multimodal Finetuning Evaluation

  • Finetuning keeps the zero-shot generative input format and freezes CLIP parameters; VQAv2 training uses answer-generation cross-entropy with label smoothing and VG-QA augmentation.The VQAv2 loss is weighted by each answer candidate’s weight.Source: §4.4 · p. 6, §4.4.2 · p. 7

4.4.1 Finetuning Image Captioning

  • After COCO finetuning, the RN50×64 model is comparable to state-of-the-art models on NoCaps; zero-shot NoCaps CIDEr is higher on the out-of-domain set than on in- and near-domain sets.On COCO, the model outperforms VL-T5/BART without an extra object detector, though a small gap to state-of-the-art remains.Source: §4.4.1 · p. 6

4.4.2 Finetuning VQA

  • On out-of-domain questions with rare answers, VLKD shows a salient advantage, with generative VQA requiring no predefined answer list.The evaluation uses the Karpathy test split.Source: §4.4.2 · p. 7

4.5 Evaluation of NLU and NLG

The evaluation examines whether VLKD preserves BART’s NLU and NLG abilities. Results indicate that performance remains close to the original BART on both types of text task.

4.5 Evaluation of NLU and NLG

  • The authors contrast these findings with prior VLP models that generally suffer weakened NLU performance, and speculate that catastrophic forgetting may contribute.They also suggest simultaneous multimodal and text-only objectives may relegate NLP ability to an auxiliary role.Source: §4.5 · p. 7
  • VLKD performs only slightly worse than original BART and significantly outperforms BERT on NLU, with the authors attributing this to BART’s knowledge being well maintained.Source: §4.5 · p. 7
  • On abstractive summarization, the performance gap between VLKD and its BART backbone is reported as negligible.Source: §4.5 · p. 7

5 Ablation Study

The ablations examine distillation objectives, masking, dataset size, and whether CLIP weights are frozen. They show task-specific objective effects and that reducing distillation data can sharply lower performance at the smallest size.

Knowledge Distillation Objectives

  • Removing TTDM or ITCL degrades zero-shot VQAv2 and COCO captioning; ITCL contributes more to captioning, while TTDM helps more on VQA.Removing both causes a large performance drop, underscoring the importance of aligning CLIP and BART embedding spaces.Source: §5 · p. 8, Table 6 · p. 9

Number of Masks

  • Mask count has a trivial influence on zero-shot captioning; the best COCO performance uses m = 6 and the best NoCaps performance uses m = 8.The authors attribute the small influence to learning to fill variable-length token sequences at masked positions.Source: §5 · p. 8, Table 7 · p. 9

Dataset Size of Distillation

  • Reducing distillation data from 3M to 1M causes only a slight performance drop, while reducing it further to 100K causes a sharp drop.Source: §5 · p. 9, Table 8 · p. 9

Unfreeze CLIP Weights

  • With CLIP unfrozen, the ViT-B/16 variant scores 31.7 zero-shot accuracy on VQAv2 and 44.8 CIDEr on COCO Caption.The authors speculate that unfreezing harms CLIP’s pretrained multimodal space and thereby downgrades VLKD performance.Source: §5 · p. 9

6 Conclusion

The paper distills CLIP’s multimodal knowledge into BART by aligning their encoders and adding a cross-modal language-modeling loss. It reports zero-shot multimodal generation and strong performance on VQA and NLP tasks, including after finetuning.

6 Conclusion

  • VLKD aligns CLIP’s multimodal encoders and BART’s textual encoder in a shared multimodal space and adds a cross-modal language-modeling loss.Source: §6 · p. 9
  • The method supports multimodal generation in zero-shot and fully finetuned settings without losing BART’s original NLP ability.Source: §6 · p. 9
  • The authors report new state-of-the-art zero-shot VQA performance and excellent finetuned performance on NLP and multimodal tasks.Source: §6 · p. 9

A Hyper-parameters

The appendix identifies hyperparameter settings for VLKD and downstream finetuning, then gives specific optimization details for GLUE and XSUM.

A Hyper-parameters

  • The appendix presents VLKD hyperparameters for two CLIP variants and downstream-task finetuning hyperparameters, using the same setting across datasets within each task.Source: A Hyper-parameters · p. 14
  • GLUE training uses LAMB for 10 epochs, with grid searches over batch size, learning rate, and weight decay, plus first-epoch warmup and linear decay.The searched values are batch sizes 16, 32, and 64; learning rates 1e-4, 5e-4, and 1e-3; and weight decay 1e-4 or 1e-3.Source: A Hyper-parameters · p. 14
  • For XSUM, the authors follow the hyperparameters used in Lewis et al. (2020).Source: A Hyper-parameters · p. 14

B More Examples of Zero-shot Inference

This section presents additional examples of zero-shot image captioning and open-ended VQA outputs.

B More Examples of Zero-shot Inference

Related papers

Loading 2203.06386v2…