PaperlayerView exact revision on arXiv

Source-linked AI summary

Compressing Visual-linguistic Model via Knowledge Distillation

Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, Zicheng Liu

arXiv:2104.02096v1cs.CVcs.AI

TL;DR

Large VL models are costly to deploy, and differing Teacher and Student detectors produce visual tokens that are difficult to align. This paper adapts the Teacher to Student proposals and distills attention and hidden representations. DistillVLM improves captioning and VQA results over non-distilled baselines.

  • Problem

    Large VL models have high inference latency and memory footprints, while different Teacher and Student detectors produce mismatched visual tokens that impede alignment.

    Source: §1 · pp. 1–2

  • Method

    DistillVLM adapts the Teacher to use Student detector proposals and distills attention distributions and hidden representations through token-wise contrastive learning.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    120.8 CIDEr on COCO captioning and 69.8 accuracy on VQA exceed the VL pre-training baselines by 5.1 and 0.8, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VL distillation improves performance over non-distilled pre-training and is reported to be more data efficient, with ablations supporting the proposed strategies.

    Source: §5 · p. 9

Abstract

from arXiv · show

Despite exciting progress in pre-training for visual-linguistic (VL) representations, very few aspire to a small VL model. In this paper, we study knowledge distillation (KD) to effectively compress a transformer-based large VL model into a small VL model. The major challenge arises from the inconsistent regional visual tokens extracted from different detectors of Teacher and Student, resulting in the misalignment of hidden representations and attention distributions. To address the problem, we retrain and adapt the Teacher by using the same region proposals from Student's detector while the features are from Teacher's own object detector. With aligned network inputs, the adapted Teacher is capable of transferring the knowledge through the intermediate representations. Specifically, we use the mean square error loss to mimic the attention distribution inside the transformer block and present a token-wise noise contrastive loss to align the hidden state by contrasting with negative representations stored in a sample queue. To this end, we show that our proposed distillation significantly improves the performance of small VL models on image captioning and visual question answering tasks. It reaches 120.8 in CIDEr score on COCO captioning, an improvement of 5.1 over its non-distilled counterpart; and an accuracy of 69.8 on VQA 2.0, a 0.8 gain from the baseline. Our extensive experiments and ablations confirm the effectiveness of VL distillation in both pre-training and fine-tuning stages.

1. Introduction

Large VL models are difficult to deploy on resource-constrained devices, while differing Teacher and Student detectors produce visual tokens that cannot be readily aligned. DistillVLM addresses this mismatch and reports gains over non-distilled baselines on captioning and VQA.

1. Introduction

  • VL distillation is applied during both pre-training and fine-tuning, using a compact transformer and lightweight detector for the Student.The paper presents this as a strategy for training smaller VL models with knowledge distillation.Source: §1 · pp. 1–2
  • Knowledge distillation transfers a strong Teacher’s knowledge to a smaller Student, including through output probabilities, feature maps, or intermediate representations.The paper positions VL distillation as adapting these approaches to visual-linguistic models.Source: §1 · pp. 1–2
  • Different Teacher and Student detectors produce semantically mismatched object proposals, preventing direct alignment of their attention distributions and hidden embeddings.This matters because large VL models have high inference latency and memory footprints that limit deployment on resource-constrained edge devices.Source: §1 · pp. 1–2
  • DistillVLM uses Student detector proposals for both networks, then transfers Teacher knowledge through attention matching and token-wise contrastive alignment of hidden embeddings.The contrastive loss compares token embeddings with randomly sampled negatives held in a sample queue.Source: §1 · p. 2
  • 120.8 CIDEr on COCO captioning and 69.8 accuracy on VQA exceed the VL pre-training baselines by 5.1 and 0.8, respectively.The paper also reports extensive ablations of its distillation strategies.Source: §1 · p. 2

2. Related Work

Prior VL pre-training commonly encodes images as regional features before training a multimodal model, while smaller models trade model size for training efficiency challenges. Knowledge distillation transfers Teacher knowledge through outputs, intermediate features, or transformer attention and representations.

2. Related Work

  • Small VL models can reduce model size while still requiring substantial pre-training data; MiniVLM is reported to need 14M examples for a good representation.Prior work also explores grid features and visual representations learned from scratch.Source: §2 · p. 2
  • Many VL models use a detector to produce regional visual tokens, then pre-train on visual-linguistic data with masked language, image-text matching, or masked region objectives.Other approaches use convolutional grid features or learn visual representations from scratch.Source: §2 · p. 2
  • Knowledge distillation transfers a strong Teacher’s knowledge by minimizing divergence, with losses based on outputs, intermediate features, or attention distributions.Transformer language-model work also uses sample queues and contrastive distillation of representations.Source: §2 · pp. 2–3

3. Visual-linguistic Knowledge Distillation

DistillVLM transfers visual-linguistic knowledge across modalities using aligned inputs and intermediate representations. Its training combines attention and hidden-state distillation with downstream classification distillation and the original pretraining objectives.

3.1. Visual Token Alignment

  • Teacher and Student use the same lightweight-detector proposals, token order, and object tags; the Teacher is retrained with those proposals to address alignment.Using the Student’s visual tokens for the Teacher may cause a small performance drop, which Teacher adaptation is intended to address.Source: §3.1 · p. 4

3.2. Attention Distribution Distillation

  • The attention loss minimizes divergence between Teacher and Student self-attention matrices at the last transformer layer.The paper notes that VL attention can encode intra-modal and cross-modal co-reference relations.Source: §3.2 · p. 4

3.3. Hidden Representation Distillation

  • Hidden-state distillation contrasts Student representations with positive Teacher representations and random negatives in a queue, using token-wise embeddings for better results.The queue stores frozen, pre-trained Teacher representations and permits comparison with more samples independently of batch size.Source: §3.3 · pp. 4–5

3.4. Classification Distillation

  • For downstream fine-tuning, classification distillation minimizes cross-entropy between the Student and Teacher softmax predictions.The temperature is fixed at 1, and the loss is intended to alleviate the domain gap from different pretraining corpora.Source: §3.4 · p. 5

3.5. Training

  • Pretraining retains the original masked-language-modeling and image-text-matching losses alongside weighted attention and hidden-state distillation losses.Classification distillation does not obviously contribute during pretraining, so it is applied at fine-tuning instead.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate DistillVLM on COCO captioning and VQA after pre-training and fine-tuning, with ablations of distillation strategies and tests across training durations and data amounts. Distillation improves results over non-distilled pre-training, including with reduced data, and the combined pre-training and fine-tuning approach yields the strongest reported DistillVLM results.

4.1. Datasets

  • The experiments pre-train on VL-7M, a combination of six datasets yielding 4 million unique images and 7 million image-text pairs, before COCO captioning and VQA evaluation.COCO uses Karpathy’s split; VQA results are reported on test-std and test-dev.Source: §4.1 · p. 5

4.2. Implementation Details

  • The 34.5M-parameter compact model uses aligned TEE proposals and object tags for Teacher and Student; the Teacher is retrained on the new visual tokens, then frozen during distillation.The Teacher’s region features still come from its own detector, while proposals come from the Student’s lightweight detector.Source: §4.2 · pp. 5–6

4.3. Results and Analysis

  • 120.8 vs. 123.7 CIDEr, DistillVLM’s best combined pre-training and fine-tuning result is comparable to Oscarb with 70% fewer parameters.The captioning comparisons include different visual feature extractors, and the reported improvements occur in both distillation stages.Source: Table 1 · p. 6, §4.3 · p. 7
  • Pre-training distillation improves captioning and VQA over the non-distilled model, with gains of 4.4 CIDEr and 0.7 on VQA test-std.The reported captioning gain also includes 1.2% at BLEU@4.Source: §4.3 · p. 7
  • Combining attention and hidden-embedding distillation gives the best result among the tested loss configurations, and the two losses are reported as complementary.Attention-only distillation gives minor gains; hidden-embedding distillation improves performance over the VL pre-training baseline across criteria.Source: §4.3 · p. 7, Table 2 · p. 7
  • NCE with token-wise embeddings performs better than MSE, while last-layer distillation outperforms layer-to-layer distillation in the reported strategy comparisons.Mean-pooled embeddings with NCE are reported on par with token-wise NCE; layer-to-layer distillation may also be constrained by differences in layer and head counts.Source: §4.3 · pp. 7–8, Table 3 · p. 7
  • Increasing the NCE sample queue improves performance, with a queue of 4,096 yielding the best results across metrics in the tested settings.The queue stores fixed random Teacher representations, and its design permits more negatives without requiring a larger batch size.Source: §4.3 · p. 8, Table 4 · p. 8
  • VL distillation improves CIDEr across training durations and data amounts; with 1% of VL-7M, it scores 4.1 higher than pre-training alone.The captioning gains are shown across different epochs and percentages of VL-7M image-text pairs.Source: §4.3 · p. 8, Figure 3 · p. 8

5. Conclusion

The conclusion presents VL distillation as a way to compress large visual-linguistic models. Experiments report better performance and greater data efficiency than non-distilled VL pre-training.

5. Conclusion

  • VL distillation is proposed as a technique for compressing large visual-linguistic models.The conclusion reports better performance and greater data efficiency than non-distilled VL pre-training, with ablations supporting the effectiveness of the strategies.Source: §5 · pp. 8–9

Appendix

The appendix provides supplementary implementation details, downstream ablations, image-retrieval results, and qualitative analysis of attention distribution distillation.

Appendix

  • Supplementary material covers a PyTorch-style pseudo-implementation, downstream ablations, image-retrieval results, and qualitative attention-distillation analysis.It also includes an analysis of VL distillation and its broader impact for future studies.Source: Appendix · p. 12

Pseudo-implementation for VL Distillation

This section introduces a PyTorch-style pseudo-implementation of the proposed VL distillation scheme.

Pseudo-implementation for VL Distillation

VL Distillation in Image Captioning Task

Fine-tuning distillation with the classification-logit loss improves captioning, while attention and hidden-state losses do not contribute in these experiments.

VL Distillation in Image Captioning Task

Transferring to Image-text Retrieval Task

The paper evaluates whether pre-training-stage VL distillation transfers to COCO image-text retrieval, reporting R@1 results on 1K and 5K test splits. The reported distilled model reaches higher R@1 scores than its non-distilled counterpart.

Transferring to Image-text Retrieval Task

  • The retrieval evaluation uses COCO top-K accuracy on 1K and 5K test splits, with the first transformer [CLS] token feeding a binary image-text match classifier.Irrelevant texts are sampled as non-matching cases; fine-tuning uses a 2e-5 learning rate for 30 epochs.Source: Transferring to Image-text Retrieval… · p. 12
  • The passage reports 81.1 versus 77.3 R@1 accuracy for DistillVLM without distillation and MiniVLM, respectively, and notes that less training data is used.Source: Transferring to Image-text Retrieval… · p. 12
  • 80.0 and 58.3 R@1 on the 1K and 5K test sets, respectively, are reported for DistillVLM with pre-training-stage distillation.The passage reports these as gains of 2.7 and 4.0, respectively.Source: Transferring to Image-text Retrieval… · p. 12

Discussion on Teacher VLM Adaptation

Teacher adaptation uses Student detector proposals to provide new visual tokens while retaining Teacher features. Results compare adapted and unadapted Teachers in pre-training and fine-tuning.

Discussion on Teacher VLM Adaptation

  • The Teacher is adapted using Student detector region proposals while retaining features from the Teacher’s own object detector.The comparison covers Teacher VLM with and without adaptation in both pre-training and fine-tuning.Source: Discussion on Teacher VLM Adaptation · p. 12
  • With aligned visual tokens and the proposed attention and hidden-state losses, pre-training distillation raises CIDEr from 110.6 to 115.0.The Table 7 comparison concerns pre-training, with its first row giving captioning performance without VL distillation.Source: Discussion on Teacher VLM Adaptation · p. 12, Table 7 · p. 13
  • Without Teacher adaptation, pre-training distillation using old visual tokens yields a slight captioning improvement, 112.3 versus 110.6.Without semantic correspondence between tokens, the experiment uses classification distillation on masked-token logits.Source: Discussion on Teacher VLM Adaptation · p. 12, Table 7 · p. 13
  • 115.6 versus 115.0 CIDEr is reported when the Teacher is retrained with new visual tokens versus using aligned tokens without retraining.The passage says distillation also gives satisfactory results without Teacher retraining.Source: Discussion on Teacher VLM Adaptation · p. 13, Table 7 · p. 13
  • In downstream fine-tuning, unadapted-Teacher distillation adds 0.2 BLEU@4 and 0.1 CIDEr, while adapted-Teacher distillation adds 0.5 and 1.0, respectively.The experiments use only the classification distillation loss; Table 8’s first row is the result without downstream distillation.Source: Discussion on Teacher VLM Adaptation · p. 13, Table 8 · p. 13

Detailed Results for Data Efficient VL Distillation

On image captioning, pre-training-stage VL distillation improves results across the reported data fractions and training durations.

Detailed Results for Data Efficient VL Distillation

  • VL distillation improves image-captioning results across all reported metrics, data fractions, and training durations.The experiments compare data fractions from 1% to 100% and converging epochs from 1 to 100.Source: Detailed Results for Data Efficient VL… · p. 13
  • Across image-captioning tasks, VL distillation during pre-training improved all metrics across data fractions from 1% to 100% and converging epochs of 1, 5, 10, 20, 50, and 100.Source: Detailed Results for Data Efficient VL… · p. 13
  • For image-captioning tasks, the reported comparisons show that VL distillation comprehensively improves all metrics, regardless of the available data fraction or which pre-training epoch setting is used.Source: Detailed Results for Data Efficient VL… · p. 13

Effect of Attention Distribution Distillation

Attention distribution distillation makes the DistillVLM’s last-block attention maps more similar to the Teacher’s in the sampled examples. The paper visualizes this comparison alongside the distillation setup and data-efficient captioning results.

Effect of Attention Distribution Distillation

  • The sampled DistillVLM attention maps are more similar to the Teacher VLM’s maps than those of the non-distilled counterpart.The authors say this indicates that attention distribution distillation mimics the Teacher’s distribution to some extent.Source: Effect of Attention Distribution… · p. 13, Figure 5 · p. 16
  • Table 9 presents data-efficient image-captioning results, with gray rows for non-distilled pre-training and following rows for VL distillation.Source: Table 9 · p. 14
  • Figure 4 depicts attention-distribution and hidden-embedding mimicking for a batch of one with N textual and visual tokens, excluding [PAD] tokens.Source: Figure 4 · p. 15
  • Figure 5 compares averaged last-block attention maps from the Teacher VLM, DistillVLM, and the non-distilled counterpart, with l2 distances to the Teacher shown below.The maps average 12 self-attention heads; the examples are random samples from the VL corpus.Source: Effect of Attention Distribution… · p. 13, Figure 5 · p. 16

Related papers

Loading 2104.02096v1…